OSH Parser Performance

We time $sh -n $file for various files under various shells, and repeat then run under cachegrind for stable metrics.

Summary

Instructions Per Line (via cachegrind)

Lower numbers are generally better, but each shell recognizes a different language, and Oil uses a more thorough parsing algorithm. In thousands of "I refs".

shell label total lines thousand irefs per line
bash 118,863 15.4
dash 111,674 1.6
mksh 111,674 5.7
oil-native 118,863 8.9

(zsh isn't measured because zsh -n unexpectedly forks.)

Average Parsing Rate, Measured on Two Machines (lines/ms)

Shell startup time is included in the elapsed time measurements, but long files are chosen to minimize its effect.

shell label total lines host broome host lenny
dash 111,674 1,535.1 3,819.4
mksh 111,674 571.1 1,061.3
bash 118,863 347.0 610.9
oil-native 118,863 307.1 610.0
zsh 118,863 104.2 323.8
osh-cpython 118,863 7.1 9.7
osh-ovm 118,863 6.4 8.8

Breakdown By File

Instructions Per Line (in thousands)

bash dash mksh oil-native num lines filename
15.6 1.4 4.4 4.9 1,679 t4014-format-patch.sh
16.7 2.1 8.4 13.9 1,733 functions
23.7 NA NA 12.6 2,029 configure-helper.sh
17.9 2.2 8.9 14.5 2,512 abuild
13.6 1.8 5.9 10.1 2,698 Build.sh
13.7 1.0 3.3 3.2 3,123 t9300-fast-import.sh
32.0 NA NA 15.7 5,160 test-cmd-util.sh
18.2 1.8 6.1 9.5 9,661 ltmain.sh
17.3 1.9 6.4 9.6 20,489 configure
12.9 1.5 5.4 8.0 69,779 configure-coreutils

Elasped Time (milliseconds)

host label bash dash mksh zsh osh-ovm osh-cpython oil-native osh to bash ratio num lines filename
broome 9 2 4 98 154 158 6 0.7 1,679 t4014-format-patch.sh
broome 10 3 7 13 444 421 12 1.1 1,733 functions
broome 15 NA NA 21 486 440 15 1.0 2,029 configure-helper.sh
broome 14 4 11 18 631 575 17 1.2 2,512 abuild
broome 11 3 8 78 456 433 16 1.4 2,698 Build.sh
broome 13 2 5 112 161 138 7 0.5 3,123 t9300-fast-import.sh
broome 32 NA NA 26 1,361 1,228 33 1.0 5,160 test-cmd-util.sh
broome 34 9 22 39 1,532 1,369 42 1.2 9,661 ltmain.sh
broome 60 15 40 123 3,370 3,036 59 1.0 20,489 configure
broome 143 35 99 613 9,896 8,890 180 1.3 69,779 configure-coreutils
lenny 3 1 1 28 88 90 2 0.7 1,679 t4014-format-patch.sh
lenny 4 1 2 3 295 274 5 1.1 1,733 functions
lenny 6 NA NA 6 314 288 5 0.8 2,029 configure-helper.sh
lenny 7 1 3 5 457 405 7 1.0 2,512 abuild
lenny 6 1 3 27 324 297 5 0.8 2,698 Build.sh
lenny 8 1 2 35 97 96 2 0.3 3,123 t9300-fast-import.sh
lenny 23 NA NA 9 983 866 14 0.6 5,160 test-cmd-util.sh
lenny 18 3 12 13 1,104 995 20 1.1 9,661 ltmain.sh
lenny 37 6 25 44 2,522 2,218 40 1.1 20,489 configure
lenny 82 16 56 198 7,303 6,664 94 1.2 69,779 configure-coreutils

Parsing Rate (lines/ms)

host label bash dash mksh zsh osh-ovm osh-cpython oil-native num lines filename
broome 194.8 905.1 401.9 17.1 10.9 10.6 269.2 1,679 t4014-format-patch.sh
broome 168.8 643.5 246.9 136.5 3.9 4.1 147.9 1,733 functions
broome 136.6 NA NA 98.2 4.2 4.6 135.5 2,029 configure-helper.sh
broome 175.9 705.2 231.1 136.5 4.0 4.4 149.7 2,512 abuild
broome 234.9 842.1 346.5 34.7 5.9 6.2 166.3 2,698 Build.sh
broome 244.2 1,409.9 587.6 27.9 19.4 22.6 450.2 3,123 t9300-fast-import.sh
broome 159.4 NA NA 198.4 3.8 4.2 155.5 5,160 test-cmd-util.sh
broome 282.1 1,035.0 434.3 249.6 6.3 7.1 229.7 9,661 ltmain.sh
broome 340.1 1,397.6 518.6 167.1 6.1 6.7 347.7 20,489 configure
broome 486.7 1,980.9 707.5 113.8 7.1 7.8 387.6 69,779 configure-coreutils
lenny 514.7 2,803.0 1,222.0 59.2 19.2 18.7 728.1 1,679 t4014-format-patch.sh
lenny 408.6 1,976.1 717.0 585.7 5.9 6.3 382.6 1,733 functions
lenny 327.3 NA NA 359.8 6.5 7.0 394.4 2,029 configure-helper.sh
lenny 354.9 2,178.7 722.0 532.7 5.5 6.2 359.2 2,512 abuild
lenny 423.7 2,632.2 1,046.1 99.6 8.3 9.1 524.6 2,698 Build.sh
lenny 404.7 4,579.2 1,537.7 89.3 32.0 32.6 1,310.0 3,123 t9300-fast-import.sh
lenny 227.9 NA NA 599.8 5.3 6.0 367.2 5,160 test-cmd-util.sh
lenny 526.7 3,122.5 779.9 738.9 8.7 9.7 481.3 9,661 ltmain.sh
lenny 553.2 3,258.4 822.9 470.4 8.1 9.2 509.7 20,489 configure
lenny 854.3 4,495.8 1,244.7 352.2 9.6 10.5 742.1 69,779 configure-coreutils

Memory Usage (Max Resident Set Size in MB)

Again, Oil uses a different algorithm (and language) than POSIX shells. It builds an AST in memory rather than just validating the code line-by-line.

host label bash dash mksh zsh osh-ovm osh-cpython oil-native num lines filename
broome 3.4 1.7 1.9 3.9 14.4 18.0 5.4 1,679 t4014-format-patch.sh
broome 3.3 1.7 1.9 3.9 17.2 21.0 7.9 1,733 functions
broome 3.3 NA NA 3.9 17.8 21.7 8.4 2,029 configure-helper.sh
broome 3.2 1.7 1.9 3.9 19.4 23.3 10.1 2,512 abuild
broome 3.3 1.8 1.9 4.1 17.8 22.1 8.7 2,698 Build.sh
broome 3.3 1.7 2.0 4.1 14.9 18.5 5.8 3,123 t9300-fast-import.sh
broome 3.5 NA NA 3.9 28.3 33.3 18.1 5,160 test-cmd-util.sh
broome 4.3 2.2 3.1 4.4 29.8 35.3 20.1 9,661 ltmain.sh
broome 3.9 2.5 2.5 4.4 50.0 58.5 39.5 20,489 configure
broome 3.8 1.9 2.2 4.0 112.8 130.7 106.1 69,779 configure-coreutils
lenny 3.3 1.8 1.9 4.0 14.5 18.1 5.7 1,679 t4014-format-patch.sh
lenny 3.4 1.7 1.9 4.0 17.3 21.2 7.8 1,733 functions
lenny 3.4 NA NA 4.1 17.6 21.8 8.3 2,029 configure-helper.sh
lenny 3.5 1.7 2.0 4.2 19.4 23.4 10.1 2,512 abuild
lenny 3.4 1.8 2.0 3.9 18.1 22.2 8.9 2,698 Build.sh
lenny 3.4 1.6 1.9 4.0 14.9 18.4 5.8 3,123 t9300-fast-import.sh
lenny 3.5 NA NA 4.0 28.3 33.3 18.0 5,160 test-cmd-util.sh
lenny 4.4 2.2 3.0 4.5 29.6 35.3 20.2 9,661 ltmain.sh
lenny 4.0 2.4 2.6 4.3 50.1 58.6 39.6 20,489 configure
lenny 3.8 1.8 2.3 4.1 112.8 130.8 106.1 69,779 configure-coreutils

Shell and Host Details

shell label shell id
bash bash-9af8f89f
dash dash-9ff48631
mksh mksh-e3755400
oil-native osh_eval.stripped-14996860
host label host id
broome broome-d80ab850
lenny lenny-6933449d

Raw Data

filename
no-host.2022-06-21__16-56-49.cachegrind.tsv
broome.2022-06-21__20-48-35.times.csv
lenny.2022-06-21__16-57-31.times.csv