Why Sponsor Oils? | blog | oilshell.org

Metrics for Oil 0.8.pre2

I wrote about the 0.8.pre2 release in this month's recap, and here are some metrics and benchmarks for it.

This post is mainly for me to keep track of the project's progress. When the codebase is fully translated to C++, I may write a retrospective like this one on parsing speed.

New Metrics

The oil-native build has now existed for 3 months, since Oil 0.7.pre9, so we can review its metrics.

Translated Code

• oil-cpp for 0.7.pre9: 48,906 lines of C++ in the tarball
• oil-cpp for 0.8.pre2: 62,889 lines of C++ in the tarball

Two files accounted for most of the increase:

1. osh-lex.h contains string matching code, and it now recognizes the names of shell builtins and options. There might be a more compact way to do this, but using re2c is convenient for now.
2. More importantly, we're translating more of the Oil interpreter with mycpp:
   • osh_parse.cc as of 0.7.pre9: 9,687 lines of C++
   • osh_eval.cc as of 0.8.pre2: 16,491 lines of C++. (The name changed because we're translating the word evaluator, the arithmetic evaluator, and more.)

I haven't yet measured the relationship between lines of Python and lines of C++, but it feels like we're translating over half of the ~28K line interpreter.

This is good progress, but it's a significant effort. The code will take several more months to fully translate.

oil-native Parsing Speed

Let's measure against a faster release:

• Parser Performance for 0.7.0: 838 lines/ms and 281 lines/ms.
• Parser Performance for 0.8.pre2: 778 lines/ms and 242 lines/ms.

This variation feels like it's within the benchmark noise because the measurements for bash and other shells also dipped. But I'll keep an eye on it.

oil-native Size and Compilation Speed

• ovm-build 0.7.0: 491,680 bytes under GCC
• ovm-build 0.8.pre2: 663,792 bytes under GCC

We're translating and compiling more code, so this increase makes sense.

Note that I expect oil-native to be significantly smaller than the OVM build (measured below).

• ovm-build 0.7.0: 28.9 and 10.1 seconds under GCC (two machines)
• ovm-build 0.8.pre2: 49.4 and 16.9 seconds under GCC
• For comparison, bash compiles in 63 to 70 and 25 to 27 seconds under GCC (two machines, and two different measurements)

The compile time seems to be increasingly linearly with the lines of C++ code.

Established Metrics

Spec Tests

There are 47 new spec tests for OSH:

• OSH spec tests for 0.7.pre9: 1536 tests, 1364 passing, 62 failing
• OSH spec tests for 0.8.pre2: 1583 tests, 1411 passing, 66 failing

And almost 29 new for Oil:

• Oil spec tests for 0.7.pre9: 218 tests, 206 passing, 12 failing
• Oil spec tests for 0.8.pre2: 247 tests, 232 passing, 15 failing

Lines of Source Code

There are over 1000 new lines of significant source code:

• cloc for 0.7.pre9: 14,156 lines of Python and C, 308 lines of ASDL.
• cloc for 0.8.pre2: 15,272 lines of Python and C, 293 lines of ASDL.

And over 2000 lines of physical source code:

• src for 0.7.pre9: 26,313 lines of Python
• src for 0.8.pre2: 28,407 lines of Python

Runtime Speed (OVM only)

Important: this is OVM, the slice of the CPython interpreter, not oil-native.

• Runtime Performance for 0.7.0: CPython's configure is 5.6x to 7.3x slower.
• Runtime Performance for 0.8.pre2: CPython's configure is 6.2x to 7.5x slower.

Both of these numbers are bad. This is why we're translating Oil to C++!

It may have gotten slower: As mentioned in the parser benchmarks retrospective, a side effect of translation is that Oil gets slightly slower when it's run under CPython. But we care about the speed in C++, not in Python.

Soon-to-Be Obsolete

OVM Native Code

Again, we have more lines of native code because of the re2c "matchers" for shell builtin names and option names.

• nativedeps for 0.7.pre9: 137,145 lines
• nativedeps for 0.8.pre2: 143,336 lines

The compiled code size increased by a corresponding amount:

• ovm-build for 0.7.pre9: 1,126,968 bytes of native code (under GCC)
• ovm-build for 0.8.pre2: 1,139,088 bytes of native code (under GCC)

OVM Bytecode

This will also be obsolete, but it increased proportionally with the source code:

• src-bin-ratio-with-opy for 0.7.pre9: 1,124,345 bytes
• src-bin-ratio-with-opy for 0.8.pre2: 1,233,598 bytes

Next

I still want to write:

• Another index of posts in #shell-the-good-parts. I missed a couple in last month's sketch.
• What the Oil Project Has Accomplished
• Features Cut From Oil
• Why Use Oil? Even though it's reduced, I want to make sure the project is compelling once it's translated.