Why Sponsor Oils? | blog | oilshell.org

Oils 0.18.0 - Progress on All Fronts


This is the latest version of Oils, a Unix shell. It's our upgrade path from bash to a better language and runtime:

Oils version 0.18.0 - Source tarballs and documentation.

We're moving toward the fast C++ implementation, so there are two tarballs:

If you're new to the project, see the Oils 2023 FAQ and posts tagged #FAQ.

Table of Contents
What's in this release?
Based on User Feedback
Interactive Shell (One Breaking Change)
Headless Shell
Under the hood: C++ Translation
YSH Language
Scrubbing the Semantics
Top-Level / Interactive Syntax
Functions and YSH "Standard Library"
What's Next? Zulip threads
Design Issues In Flight
Appendix: Closed Issues
Appendix: Metrics for the 0.18.0 Release
Spec Tests
Code Size

What's in this release?

We're moving toward retiring the Python tarball: OSH differs by 2 test cases in C++, and YSH differs by 77, down from 157 in the last release.

We're deep in the middle of implementing YSH: particularly functions, procs, and data languages. We're constantly testing and revising the language design.

This release "checkpoints" that work, as well as reflecting progress on every other part of the project.

Another highlight is the new Oils Reference, organized into two tables of contents, and 13 chapters. As YSH stabilizes, we'll document its behavior here.

Let's summarize what we've done, highlighting contributions. As always, we can use more help!

Based on User Feedback

The docs still need a lot of renaming from Oil → YSH. Feel free to open bugs on specific docs that are confusing.

Thank you for testing Oils!

Interactive Shell (One Breaking Change)

Headless Shell

We can now export shell completions as JSON, including the bash completions that OSH runs:

compexport -c 'echo $HO'  # completes shell language itself
                          # e.g. $HOME
compexport -c 'git ch'    # runs git's bash plugin for completion
                          # e.g. checkout, cherry-pick

I claim that this is a missing feature in bash. For example, the git plugin registers itself with this line:

complete -o bashdefault -o default -o nospace -F $wrapper

But programs that want to scrape bash completion can only see the logic in $wrapper, not the three -o options. See Projects Already Doing Something Like Shellac for such programs.

So OSH compexport may be the mechanism for other shells to reuse bash completion!

This work is also still in progress, and only lightly documented. Please join our #shell-gui channel, and help us test it.

It was motivated both by testing our completion logic (escaping), and the headless shell. We want to export shell completions to a GUI. But again, I think other programs can also use it, rather than scraping bash and re-implementing parts of it.


Fixed our own "dogfood" issues:

The is-main builtin returns 1 (false) if the current file was executed with the source builtin. It's designed to be used like Python:

if __name__ == '__main__':  # Python


if is-main; then
  main "$@"


if is-main {
  main @ARGV

Related: #tools-for-oils > Tree Shaking - Static/Dynamic. Our bundling / tree shaking may work by dynamically walking dependencies, and cat-ting the result. It would have to rewrite source and is-main though.


Still TODO:


Tuned the pool allocator that Chris Watkins implemented earlier this year. It was very pleasant to work with!

Instead of a single pool for objects under 32 bytes, we have two pools with thresholds of 24 and 48 bytes. In other words, these are members of our MarkSweepHeap:

Pool<682, 24> pool1_;
Pool<341, 48> pool2_;

The 24 bytes comes from the fixed-sized "head" of List<T>, which is the most common type in the program. We should also be able to fit the common Token in 24 bytes.

TODO: Tune our growth factors for Slab<T> so the first one fits exactly in 48 bytes.

Note: we tried other general purpose allocators like tcmalloc and mimalloc, but it's easy to do better on our workloads, measured with uftrace, cachegrind, perf, etc. I like that we have ~800 lines of code for our entire allocator and garbage collector, not 30,000!

Some color on how we arrived at this:

So allocation is no longer a bottleneck, and GC rooting now sticks out as the next thing to optimize. There was also an unrelated slowdown due to more entries in our slow Dict, so we need to write a real Dict.

Some parts of the runtime are still "hilariously unoptimized", but we're making progress.

Under the hood: C++ Translation

Tightening up our metalanguage:

YSH Language

Scrubbing the Semantics

I scrubbed the expression evaluator for consistency.

More details on issue #1710. There are still a few more behaviors to consider, like tightening up chained comparisons.

Top-Level / Interactive Syntax

YSH now allows shell-style myvar=x at the top level. I had disallowed it in favor of var myvar = 'x', but we now have a Language Design Principle of "not breaking the top level".

That is, these commands all work the same way in YSH, although with stricter error handling and no word splitting:


ls /tmp | wc -l 2>/dev/null

PYTHONPATH=. python3 -c 'print("hi")

What distinguishes YSH is really the compound commands: proc func, if case, and for while, not the top level.

The shell style is easier to type interactively, and is also useful for tilde expansion x=~/src. (YSH expressions don't have an equivalent of tilde expansion, since all strings are single- or double-quoted.)

On the other hand, myvar=x is disallowed in funcs and procs:

$ proc p { myvar=x; echo hi }
[ interactive ]:1: Use var/setvar to assign in YSH

Remember that the var foo = {age: 10} style lets you use typed data on the right-hand side.

This change allowed us to remove shopt -s parse_sh_assign, which is good because fewer global options is better.

Functions and YSH "Standard Library"

Thanks to Aidan Olsen and Melvin Walls, all YSH functions and methods are now either:

  1. Implemented in typed Python, and translated to C++ (no more CPython dependency, aka "metacircular" hack), or ...
  2. Implemented in YSH itself



What's Next? Zulip threads

As mentioned, we're deep in the middle of designing and implementing YSH. Here are Zulip threads that may be good starting points — they link to other threads.

It all fits, and is useful! (Paradox: shell encourages polyglot programming, but YSH could make scripting more "monoglot". Reducing language cacophony seems overdue in shell.)

Design Issues In Flight

Here's a random sampling of more detailed Zulip threads. They reflect what we're thinking about.

We're nailing down design details as we implement features. I know it's dense, but informed feedback helps!


Procs and funcs continue to be a difficult design issue. These threads cover a large chunk of our work in the next few months:

Also important:

Threads related to modules / namespaces / tree-shaking:


We made progress on all fronts!

This release was delayed by about 2 weeks because I wanted to show Screencasts of an Interactive Shell.

That's the next post. We have concrete demos of things no other shell can do.

In the meantime, feel free to ask questions in the comments!


Appendix: Closed Issues

Some of the work we've done is reflected in these issues. You can also view the full changelog for this release.

#1722 Rewrite FUNCNAME BASH_LINENO BASH_SOURCE and fix minor bugs
#1716 Allow shell assignment foo=bar at the top level in YSH
#1707 Polish location info and fix bugs
#1703 Trying to build CPP release from website, and getting errors from ninja-rules-cpp.sh
#1695 [breaking] GNU readline app name is now "oils", not "oil"
#1687 Initialization with rc files section of getting started doc references old filepath
#1521 globs not expanded in redirects, and we use this in `deps/from-binary.sh`
#1500 popd directory stack empty differs from bash
#1415 Arguments to functions like eval_hay() aren't checked
#1362 Enable try/except/else and try/finally errors in mycpp
#1163 replace try/finally with context managers through the codebase
#1093 No notification when 'sleep 1 &' finishes
#1011 Add is-main builtin
#915 oilshell escapes completion candidates
#532 Improve the help builtin and toolchain

Appendix: Metrics for the 0.18.0 Release

These metrics help me keep track of the project. Let's compare this release with the previous one, version 0.17.0.

Spec Tests

We improved OSH, and the 2 extra failures are TODOs on completion and compexport:

Translating the help builtin the C++ tarball very close to parity:

YSH got a lot of new behavior:

And the C++ tarball is catching up rapidly:

When we write our own JSON library, the delta should be almost zero.


The parser is faster due to the new pool allocator:

Parser memory usage increased slightly, which is a little surprising. My memory is that the pool allocator decreased memory usage, so this could have been something else:

A few more objects allocated at runtime, partially due to redirect args supporting globs (mentioned above):

Our fib benchmark got slower, and I tracked down the cause of it. It's due to translating more functions to YSH, and our poor Dict implementation! (Update: Melvin just brought this back down with a simple change.)

We still have to catch up with bash here:

Code Size

OSH got a little bigger (and we still need to add YSH):

Translating YSH led to more C++ code in the tarball:

And more executable code:

Optimizing the GC rooting should bring the binary size down a bit.