blog | oilshell.org

Oil 0.8.pre6 - Pure Bash and C++

2020-06-18

This is the latest version of Oil, a Unix shell that's our upgrade path from bash:

Oil version 0.8.pre6 - Source tarballs and documentation.

To build and run it, follow the instructions in INSTALL.txt. The wiki has tips on How To Test OSH.

Table of Contents
Highlights
Patch to run the mal Lisp
Closed Issues
What's Next?
Appendix: Selected Metrics
Lines of Native Code
Binary Size
Build Speed
Test Results
Benchmarks

Highlights

(1) Fixes so OSH can run two interpreters written in bash:

These are pure programs in the sense that they do little I/O. For example, they don't start external processes like typical shell scripts. Pure programs are useful to test the speed of the OSH interpreter itself, and because translating I/O to C++ is separate work.

(2) Fixes to run the ShellSpec test framework. For example, shopt -u verbose_errexit makes OSH silent on errexit, like other shells. Feedback on the shell's verbosity is still welcome.

Keep the bug reports coming! The fixes were minor, which is evidence that OSH is maturing.

(3) More progress translating Oil to C++, i.e. oil-native. The last release announcement described this work.

Note that these two strands of work have yet to converge. That is, the Python version of OSH can run Lisp, Brainfuck, and a JSON parser, but oil-native can't yet.

Achieving that may be a good milestone for version 0.8.0!

Patch to run the mal Lisp

Pull Request 518 on kanaka/mal was merged a couple days ago, and it shows the changes necessary to run mal under OSH. To justify the changes, I linked to sections in the Known Differences doc:

In my mind, each of these changes improved the program. So this is evidence that OSH is delivering on its claims to be a stricter, saner language, while still running real bash programs.

Related: Oil 0.8.pre4 - The Biggest Shell Programs in the World

Closed Issues

Thanks to Crestwave and Koichi Nakashima for testing and reporting bugs!

#774 eval 'break' doesn't break, 'source return.sh' doesn't return, etc.
#772 Support negative indices in string slices
#769 read -n1 </dev/null succeeds
#768 Unsetting the last element of an array and appending an element behaves differently than bash
#765 Incorrect handling of literal hyphen in patsub
#763 Suppress some errexit messages

In addition to those issues, see the full git log.

What's Next?

I'm very focused on translation to C++. Running these pure bash programs under the oil-native will be a great milestone, and it opens up the possibility of using existing benchmarks in the kanaka/mal repository. How fast does a Lisp run under OSH?

Such benchmarks aren't representative of all shell programs, but they are representative of programs that do a lot of string processing, like autocompletion plugins.

I'd still like more help with testing, bug fixes, and evaluating the Oil language. As I mentioned in the last post, Oil is looking more like a language/library for building shells than a shell itself.

If you want it to be more than that, please get involved! Send us a message on Github or Zulip.

Appendix: Selected Metrics

Let's compare this release with last month's 0.8.pre5 release.

Lines of Native Code

Most commits in this release were related to C++ translation. Here's evidence of that:

Binary Size

I still have to figure out why the size of osh_eval.opt.stripped differs so much between GCC and Clang.

I also think that the binary is too big, considering the source code size. For example, bash has over 140K lines of code, and it's also about 1 MB in size.

I spent some time with bloaty and it looks like some of the problems are:

However, to keep this in context, common shells in Rust or Go are often 10x bigger than Oil and bash. They're more like 10 MB than 1 MB.

But my goal is to sneak the Oil language in "for free", so to speak. Making the binary smaller is future work.

Build Speed

For osh_eval.opt.stripped:

Compared to oil.ovm:

Compared to bash make:

So oil-native compiles about as fast as our slice of CPython, and compiles a bit faster than bash.

Although again, it has fewer lines of code than bash, so I think it should be even faster. I think that getting rid of small translation units (.cc files) may improve build times. This is also future work.

Test Results

More builtins run in C++, leading to over 200 new tests passing:

(Minor harness bugs: I fixed the inconsistency between 1539 and 1560 that showed up last time, but there's still an inconsistency between 1589, and 1587 below.)

OSH spec tests:

Oil spec tests:

Significant lines of code:

Physical lines:

Benchmarks

These benchmarks are still noisy, but roughly unchanged. The parser benchmark measures C++ code:

The runtime benchmark measures Python code for now: