Why Sponsor Oils? | blog | oilshell.org

A Plan for Oil 0.8 and 0.9

I published the Summer Blog Roadmap a few weeks ago, and it repeated what I wrote in January: I want to cut features to get the project "done".

Done really means "self-sustaining". In the best case, Oil won't ever be done, but it will be compelling enough that I'll have help from users.

This post reviews our goals and recent progress. Then I make comments on project milestones and technical risks. It may fall a bit short of "a plan", as you'll see.

Recap of Our Goals

There are 5 major parts of the project:

1. The compatible OSH language. This took years of effort, but there's no "risk" left. Oil is the most bash-compatible shell, by a mile. It has new features, and helps you with many of shell's sharp edges (wiki).
2. The new Oil language. I just drafted a new doc on Oil language idioms, and it's given me new energy for the project. The language is looking good, although it's probably closer to 40% of the way to a solid 1.0 than 80%.
3. The interactive shell, and the ability to set OSH as the system (root) shell.
4. Improving performance. This is its own subproject because Oil is implemented in an unusual style, with DSLs and metaprogramming.
5. The documentation for everything above. This is a big effort, and it deserves its own category.

So the goal for 2020 was to "finish" OSH and performance (1 and 4), and polish a minimal portion of Oil (2).

We would cut the interactive shell, and large parts of the Oil language. Then we'd have a solid and improved shell for automation and programming, but a conservative one. After all, Oil is your upgrade path from bash (which I've emphasized on the home page).

Then we would work on docs in 2021, hopefully with some help.

I'm Excited About the Oil Language

But after going on vacation, and writing the idioms doc, I feel like working on the Oil language now. I find this list of benefits compelling, coherent, and within reach (and everything is still up for discussion):

• Eliminate "quoting hell" with Simple Word Evaluation. Done!
• Reduce ad hoc parsing and splitting with the QSN and QTSV interchange formats. Supported by the read and write builtins. Mostly designed, but not implemented.
• Blocks to save and restore context, and for DSLs. Partially done.
• Procs are shell functions with parameter signatures, and more familiar scoping. Mostly done.
• Fix errexit. Mostly designed, but not implemented.
• A brand new expression language. Partially done.
  • Including Egg Expressions. Mostly done

So Oil is taking a nice shape. However, polishing it will push the concrete milestone further out, in favor or something more open-ended. I don't like that. But on the other hand, the new benchmarks I discuss below are looking good, which makes me feel better. And the Oil project is supposed to be fun.

I don't know exactly what will happen, but this post and the next one will help me figure it out. Describing the technical risks might change my mind.

Recent Progress

Oil 0.8.pre8 - oil-native Passes More Spec Tests

I worked on C++ translation, as well as features to run the brainfuck interpreter, like mapfile -t and printf -v a[i].

Writing new features in statically-typed Python and automatically translating to C++ is working well: the code is both short and fast. Although memory management is still an open problem, which I discuss below.

The progress on translation can be seen in the spec-cpp results:

• 0.8.pre5 on May 21: 420 cases passing under oil-native
• 0.8.pre6 on June 13: 633 cases passing
• 0.8.pre7 on July 7: 704 cases passing
• 0.8.pre8 on July 18: 860 cases passing

Our goal is to pass around 1641 cases, as the Python version of Oil does. This is a good trajectory, but progress isn't linear, and there's significant work to do after all test cases pass.

This commit gave a hint about performance. I was surprised that with the rough translation, and no optimization, Oil runs the brainfuck programs as fast or faster than bash does. For comparison, Oil's parser needed to be optimized in the "C++ domain" to surpass the speed of the bash parser.

Oil 0.8.pre9 - New Benchmarks Are Looking Good

I was curious about performance, so I wrote benchmarks, not only comparing the same program across shells, but with Python equivalents. They're published under /release/$VERSION/benchmarks.wwz/compute/, and I'll continue to improve them. I tested out loops, conditionals, integers, strings, arrays, hash tables, shell code in the wild, and more.

The summary is that Oil is already faster than bash, with the big caveat that it doesn't reclaim any memory. The benchmarks clearly show this! We're running long loops that allocate at every iteration, so Oil easily uses 10x more memory than bash. It's easy to make it crash with more iterations of the loop.

Deallocating memory is future work, and it will necessarily slow the interpreter down.

Even with that caveat, I still think this result is a bit surprising. Semantically, Oil is still completely unoptimized Python, though it's automatically translated line-for-line to C++. For example, there are no objects on the stack, because Python can't express that!

I think this could be because Oil's more static nature means it avoids parsing at runtime. I speculated on this in Problems With Multi-Stage Parsing (the 4th post on the blog, nearly 4 years ago!).

Version 0.8.0 - A Pure Interpreter

Now let's talk about upcoming releases.

Oil 0.8.0 is almost there. The spec-cpp results above show that the translation progress is working, and we're running real programs as well as benchmarks.

Here a few things I want to do for this milestone, which may or may not happen:

• Translate some of core/process.py, and refactor some of it into core/pyos.py. We rewrite the latter file by hand rather than translating it. In other words, the I/O translation is done at a higher level than individual libc calls.
• This could allow us to run the mal Lisp in bash under oil-native. It requires the "impure" source builtin. (Trivia: source is impure because it has to use process.FdState to open files, which avoids conflict with user descriptors like echo 1>&3.)
• Maybe: Implement simple commands like ls. We pass 863 tests without starting any processes, and this would bring the total to well over 1000 out of ~1650.

Version 0.9.0 - Shell I/O

In addition to starting simple commands, Oil 0.9.0 should run pipelines, subshells, async processes with &, and do redirects.

This should let it run:

• The OSH runtime benchmarks, which include configure scripts from Python, OCaml, and TCC; and Alpine's abuild.
• All of Oil's own shell scripts, which we dogfood on every release.
• neofetch, one of the biggest shell programs in the world.
• Maybe: ble.sh unit tests, which I'm running in the continuous build.

Note: I/O needs error handling with exceptions, which needs testing. APIs will be refactored.

Technical Issue: Memory Management

This is the biggest open problem with oil-native. I had hoped to solve it before the end of the year, for Oil 0.9.0. But I haven't started, so that seems unlikely.

The current thinking is that I'll implement a dirt simple garbage collector, which can always be replaced later. I did way too much research on memory management, and I've come to realize I shouldn't spend any of our novelty budget here.

Notes:

• It will be a garbage collector for Oil's metalanguage, not Oil itself! The difference is that static types and GC interact, because tracing needs to know which fields are pointers. (I don't think this counts as "novelty", but I could be wrong.)
• The garbage collector will be either mark-and-sweep or copying. I like the idea of a copying GC because it could possibly be used to save and restore interpreter state. Although again, this could be unnecessary novelty.
• It only runs on allocation. There is no background thread.
• This is similar to D's garbage collector. And we will use some of the same strategies to optimize the code rather than the GC: by simply avoiding allocations.
  • The GC Series (dlang.org) has tips and wisdom.
• This means we can keep the GC simple. It doesn't need to be generational or incremental. I'm hoping that most of the complexity will be at build time in ASDL and mycpp, and there will be a small amount of runtime code.

Discarded ideas:

• Doing anything clever with #oheap. One motivation for this compressed binary format was that that creating the lossless syntax tree involves allocating many tiny objects. I wrote 2 prototypes of OHeap, but they didn't end up getting used in Oil.
• Compile Time Reference Counting in Lobster. Looks very interesting, but it's novel. I'd like to see it in another language first. Again, I think it's simpler to use a dumb deallocation mechanism, and manually optimize the code to perform fewer allocations. (But note that I encourage experiments by anyone who wants to test experimental algorithms on a mature, realistic codebase!)

Signal Handling

This is another technical issue that falls outside the "pure" interpreter. I think it's OK to do after 0.9.0, since we're cutting the interactive shell, and many shell programs don't use trap.

Summary

• I'm cutting the interactive shell for the forseeable feature. This is unfortunate, but it's good to have more focus. (Again, I of course welcome contributors who want to push Oil in this direction.)
• I'm excited by the Oil language, and want to get back to work on it. That will take away from translation and performance, but I think there's room for it in 2020.
• I want Oil 0.9.0 to happen before the end of the year. It will run more real shell programs, and pass more spec tests in C++. It should be pretty fast, but it may not reclaim memory!

So 0.9.0 will be compelling, but it isn't "done" or self-sustaining. We will likely need 0.10.0 and 0.11.0. The next post will speculate on those future releases, again describing milestones and risks.