blog | oilshell.org

Metrics for Oil 0.8.pre2

2020-03-08

I wrote about the 0.8.pre2 release in this month's recap, and here are some metrics and benchmarks for it.

This post is mainly for me to keep track of the project's progress. When the codebase is fully translated to C++, I may write a retrospective like this one on parsing speed.

Table of Contents
New Metrics
Translated Code
oil-native Parsing Speed
oil-native Size and Compilation Speed
Established Metrics
Spec Tests
Lines of Source Code
Runtime Speed (OVM only)
Soon-to-Be Obsolete
OVM Native Code
OVM Bytecode
Next

New Metrics

The oil-native build has now existed for 3 months, since Oil 0.7.pre9, so we can review its metrics.

Translated Code

Two files accounted for most of the increase:

  1. osh-lex.h contains string matching code, and it now recognizes the names of shell builtins and options. There might be a more compact way to do this, but using re2c is convenient for now.
  2. More importantly, we're translating more of the Oil interpreter with mycpp:

I haven't yet measured the relationship between lines of Python and lines of C++, but it feels like we're translating over half of the ~28K line interpreter.

This is good progress, but it's a significant effort. The code will take several more months to fully translate.

oil-native Parsing Speed

Let's measure against a faster release:

This variation feels like it's within the benchmark noise because the measurements for bash and other shells also dipped. But I'll keep an eye on it.

oil-native Size and Compilation Speed

We're translating and compiling more code, so this increase makes sense.

Note that I expect oil-native to be significantly smaller than the OVM build (measured below).

The compile time seems to be increasingly linearly with the lines of C++ code.

Established Metrics

Spec Tests

There are 47 new spec tests for OSH:

And almost 29 new for Oil:

Lines of Source Code

There are over 1000 new lines of significant source code:

And over 2000 lines of physical source code:

Runtime Speed (OVM only)

Important: this is OVM, the slice of the CPython interpreter, not oil-native.

Both of these numbers are bad. This is why we're translating Oil to C++!

It may have gotten slower: As mentioned in the parser benchmarks retrospective, a side effect of translation is that Oil gets slightly slower when it's run under CPython. But we care about the speed in C++, not in Python.

Soon-to-Be Obsolete

OVM Native Code

Again, we have more lines of native code because of the re2c "matchers" for shell builtin names and option names.

The compiled code size increased by a corresponding amount:

OVM Bytecode

This will also be obsolete, but it increased proportionally with the source code:

Next

I still want to write: