1 Oils Source Code
2 ================
3
4 [![Build
5 Status](https://github.com/oilshell/oil/actions/workflows/all-builds.yml/badge.svg)](https://github.com/oilshell/oil/actions/workflows/all-builds.yml) <a href="https://gitpod.io/from-referrer/">
6 <img src="https://img.shields.io/badge/Contribute%20with-Gitpod-908a85?logo=gitpod" alt="Contribute with Gitpod" />
7 </a>
8
9 [Oils][] is our upgrade path from bash to a better language and runtime!
10
11 - [OSH][] runs your existing shell scripts.
12 - [YSH][] is for Python and JavaScript users who avoid shell.
13
14 (The project was [slightly renamed][rename] in March 2023, so there are still
15 old references to "Oil". Feel free to send pull requests with corrections!)
16
17 [Oils]: https://www.oilshell.org/
18
19 [OSH]: https://www.oilshell.org/cross-ref.html#OSH
20 [YSH]: https://www.oilshell.org/cross-ref.html#YSH
21
22 [rename]: https://www.oilshell.org/blog/2023/03/rename.html
23
24 [Oils 2023 FAQ][faq-2023] / [Why Create a New Unix Shell?][why]
25
26 [faq-2023]: https://www.oilshell.org/blog/2023/03/faq.html
27 [why]: https://www.oilshell.org/blog/2021/01/why-a-new-shell.html
28
29 It's written in Python, so the code is short and easy to change. But we
30 automatically translate it to C++ with custom tools, to make it fast and small.
31 The deployed executable doesn't depend on Python.
32
33 This README is at the root of the [git repo][git-repo].
34
35 [git-repo]: https://github.com/oilshell/oil
36
37 <div id="toc">
38 </div>
39
40 ## Contributing
41
42 * Try making the **dev build** of Oils with the instructions on the
43 [Contributing][] page. This should take 1 to 5 minutes if you have a Linux
44 machine.
45 * If it doesn't, let us know. You can post on the `#oil-dev` channel of
46 [oilshell.zulipchat.com][], or file an issue on Github.
47 * Feel free to grab an [issue from
48 Github](https://github.com/oilshell/oil/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22).
49 Let us know what you're thinking before you get too far.
50
51 [Contributing]: https://github.com/oilshell/oil/wiki/Contributing
52 [oilshell.zulipchat.com]: https://oilshell.zulipchat.com/
53 [blog]: https://www.oilshell.org/blog/
54
55 ### Quick Start on Linux
56
57 After following the instructions on the [Contributing][] page, you'll have a
58 Python program that you can quickly run and change! Try it interactively:
59
60 bash$ bin/osh
61
62 osh$ name=world
63 osh$ echo "hello $name"
64 hello world
65
66 - Try running a shell script you wrote with `bin/osh myscript.sh`.
67 - Try [YSH][] with `bin/ysh`.
68
69 Let us know if any of these things don't work! [The continuous
70 build](http://travis-ci.oilshell.org/) tests them at every commit.
71
72 ### Dev Build vs. Release Build
73
74 Again, note that the **developer build** is **very different** from the release
75 tarball. The [Contributing][] page describes this difference in detail.
76
77 The release tarballs are linked from the [home
78 page](https://www.oilshell.org/). (Developer builds don't work on OS X, so use
79 the release tarballs on OS X.)
80
81 ### Important: We Accept Small Contributions!
82
83 Oils is full of [many ideas](https://www.oilshell.org/blog/), which may be
84 intimidating at first.
85
86 But the bar to contribution is very low. It's basically a medium size Python
87 program with many tests, and many programmers know how to change such programs.
88 It's great for prototyping.
89
90 - For OSH compatibility, I often merge **failing [spec
91 tests](https://www.oilshell.org/cross-ref.html#spec-test)**. You don't even
92 have to write code! The tests alone help. I search for related tests with
93 `grep xtrace spec/*.test.sh`, where `xtrace` is a shell feature.
94 - You only have to make your code work **in Python**. Plain Python programs
95 are easy to modify. The semi-automated translation to C++ is a separate
96 step, although it often just works.
97 - You can **influence the design** of [YSH][]. If you have an itch to
98 scratch, be ambitious. For example, you might want to show us how to
99 implement [nonlinear pipelines](https://github.com/oilshell/oil/issues/843).
100
101 ### Docs
102
103 The [Wiki](https://github.com/oilshell/oil/wiki) has many developer docs. Feel
104 free to edit them. If you make a major change, let us know on Zulip!
105
106 There are also READMEs in some subdirectories, like `opy/` and `mycpp/`.
107
108 If you're confused, the best thing to do is to ask on Zulip and someone should
109 produce a pointer and/or improve the docs.
110
111 Docs for **end users** are linked from each [release
112 page](https://www.oilshell.org/releases.html).
113
114 ## Repository Structure
115
116 Try this to show a summary of what's in the repo and their line counts:
117
118 $ metrics/source-code.sh overview
119
120 (Other functions in this file may be useful as well.)
121
122 ### A Collection of Interpreters
123
124 Oils is naturally structured as a set of mutually recursive parsers and
125 evaluators. These interpreters are specified at a high-level: with regular
126 languages, Zephyr ASDL, and a statically-typed subset of Python.
127
128 bin/ # Main entry points like bin/osh (source in bin/oils_for_unix.py)
129 frontend/ # Input and lexing common to OSH and YSH
130 osh/ # OSH parsers and evaluators (cmd, word, sh_expr)
131 ysh/ # YSH parser and evaluator
132 data_lang/ # Languages based on JSON
133 library/ # Builtin commands and functions
134 core/ # Other code shared between OSH and YSH
135 pyext/ # Python extension modules, e.g. libc.c
136 pylib/ # Borrowed from the Python standard library.
137 tools/ # User-facing tools, e.g. the osh2oil translator
138
139 ### DSLs / Code Generators
140
141 Here are the tools that transform that high-level code to efficient code:
142
143 asdl/ # ASDL implementation, derived from CPython
144 pgen2/ # Parser Generator, borrowed from CPython
145 mycpp/ # Experimental translator from typed Python to C++.
146 # Depends on MyPy. See mycpp/README.md
147 pea/ # Perhaps a cleaner version of mycpp
148 opy/ # Python compiler in Python (mycpp/ will replace it)
149
150 ### Native Code and Build System
151
152 We have native code to support both the dev build (running under CPython) and
153 the `oils-for-unix` build (pure C++):
154
155 NINJA-config.sh # Generates build.ninja
156
157 build/ # High level build
158 NINJA-steps.sh
159 NINJA_main.py # invoked by NINJA-config.sh
160 NINJA_subgraph.py
161 oil-defs/ # Files that define our slice of CPython.
162 py.sh # For development builds, running CPython
163 cpp/ # C++ code which complements the mycpp translation
164 NINJA-steps.sh
165 NINJA_subgraph.py
166 mycpp/ # Runtime for the translator
167 NINJA-steps.sh
168 NINJA_subgraph.py
169
170 prebuilt/ # Prebuilt files committed to git, instead of in _gen/
171
172 Python-2.7.13/ # For the slow Python build
173
174 # Temp dirs (see below)
175 _bin/
176 _build/
177 _gen/
178 _test/
179
180 ### Several Kinds of Tests
181
182 Unit tests are named `foo_test.py` and live next to `foo.py`.
183
184 test/ # Test automation
185 gold/ # Gold Test cases
186 gold.sh
187 sh_spec.py # shell spec test framework
188 spec.sh # Types of test runner: spec, unit, gold, wild
189 unit.sh
190 wild.sh
191 testdata/
192 spec/ # Spec test cases
193 bin/ # tools used in many spec tests
194 testdata/ # scripts for specific test cases
195 stateful/ # Tests that use pexpect
196
197 ### Dev Tools and Scripts
198
199 We use a lot of automation to improve the dev process. It's largely written in
200 shell, of course!
201
202 benchmarks/ # Benchmarks should be run on multiple machines.
203 metrics/ # Metrics don't change between machines (e.g. code size)
204 client/ # Demonstration of OSH as a headless server.
205 deps/ # Dev dependencies and Docker images
206 devtools/ # For Oils developers (not end users)
207 release.sh # The (large) release process.
208 services/ # talk to cloud services
209 demo/ # Demonstrations of bash/shell features. Could be
210 # moved to tests/ if automated.
211 old/ # A junk drawer.
212 web/ # HTML/JS/CSS for tests and tools
213 soil/ # Multi-cloud continuous build (e.g. sourcehut, Github)
214
215 ### Temp Dirs
216
217 Directories that begin with `_` are **not** stored in `git`. The dev tools
218 above create and use these dirs.
219
220 _bin/ # Native executables are put here
221 cxx-dbg/
222 _build/ # Temporary build files
223 _cache/ # Dev dependency tarballs
224 _devbuild/ # Generated Python code, etc.
225 _gen/ # Generated C++ code that mirrors the repo
226 frontend/
227 _release/ # Source release tarballs are put here
228 VERSION/ # Published at oilshell.org/release/$VERSION/
229 benchmarks/
230 doc/
231 metrics/
232 test/
233 spec.wwz
234 wild.wwz
235 ...
236 web/ # Static files, copy of $REPO_ROOT/web
237 table/
238 _test/ # Unit tests, mycpp examples
239 tasks/
240 _tmp/ # Output of other test suites; temp files
241 spec/
242 wild/
243 raw/
244 www/
245 osh-parser/
246 osh-runtime/
247 vm-baseline/
248 oheap/
249 startup/
250 ...
251
252 ### Build Dependencies in `../oil_DEPS`
253
254 These tools are built from shell scripts in `soil/`. The `oil_DEPS` dir is
255 "parallel" to Oils because it works better with container bind mounds.
256
257 ../oil_DEPS/
258 re2c/ # to build the lexer
259 cmark/ # for building docs
260 spec-bin/ # shells to run spec tests against
261 mypy/ # MyPy repo
262 mycpp-venv/ # MyPy binaries deps in a VirtualEnv
263
264 py3/ # for mycpp and pea/
265 cpython-full/ # for bootstrapping Oils-CPython
266
267
268 ### Build System for End Users version.
269
270 These files make the slow "Oils Python" build, which is very different than the
271 **developer build** of Oils.
272
273 Makefile
274 configure
275 install
276
277 These files are for the C++ `oils-for-unix` tarball (in progress):
278
279 _build/
280 oils.sh
281
282 ### Doc Sources
283
284 doc/ # A mix of docs
285 doctools/ # Tools that use lazylex/ to transform Markdown/HTML
286 lazylex/ # An HTML lexer which doctools/ builds upon.
287 README.md # This page, which is For Oils developers
288
289 LICENSE.txt # For end users
290 INSTALL.txt
291
292 ## More info
293
294 There are README files in many subdirectories, like
295 [mycpp/README.md](mycpp/README.md).
296
297 * [The blog][blog] has updates on the project status.
298 * [Oils Home Page](https://www.oilshell.org/)
299 * [oilshell.zulipchat.com][] is for any kind of discussion
300 * Subscribe for updates:
301 * [/r/oilshell on Reddit](https://www.reddit.com/r/oilshell/)
302 * [@oilshellblog on Twitter](https://twitter.com/oilshellblog)
303
304