OSH Parser Performance

We time $sh -n $file for various files under various shells, and repeat then run under cachegrind for stable metrics.

Summary

Instructions Per Line (via cachegrind)

Lower numbers are generally better, but each shell recognizes a different language, and Oil uses a more thorough parsing algorithm. In thousands of "I refs".

shell label total lines thousand irefs per line
bash 118,863 15.3
dash 111,674 1.6
mksh 111,674 5.7
oil-native 118,863 8.6

(zsh isn't measured because zsh -n unexpectedly forks.)

Average Parsing Rate, Measured on Two Machines (lines/ms)

Shell startup time is included in the elapsed time measurements, but long files are chosen to minimize its effect.

shell label total lines host flanders host lenny
dash 111,674 1,439.2 3,465.4
mksh 111,674 413.2 1,049.0
bash 118,863 248.6 671.0
oil-native 118,863 196.6 619.3
zsh 118,863 27.8 325.2
osh-cpython 118,863 2.7 9.9
osh-ovm 118,863 2.4 9.1

Breakdown By File

Instructions Per Line (in thousands)

bash dash mksh oil-native num lines filename
15.6 1.3 4.4 4.7 1,679 t4014-format-patch.sh
16.7 2.1 8.4 13.4 1,733 functions
23.7 NA NA 12.1 2,029 configure-helper.sh
17.8 2.1 8.9 14.0 2,512 abuild
13.6 1.8 5.9 9.7 2,698 Build.sh
13.7 1.0 3.3 3.1 3,123 t9300-fast-import.sh
32.0 NA NA 14.9 5,160 test-cmd-util.sh
18.2 1.8 6.1 9.2 9,661 ltmain.sh
17.3 1.9 6.4 9.2 20,489 configure
12.9 1.5 5.4 7.7 69,779 configure-coreutils

Elasped Time (milliseconds)

host label bash dash mksh zsh osh-ovm osh-cpython oil-native osh to bash ratio num lines filename
flanders 13 2 6 163 300 292 10 0.7 1,679 t4014-format-patch.sh
flanders 15 3 14 46 1,026 944 19 1.3 1,733 functions
flanders 20 NA NA 86 1,260 976 20 1.0 2,029 configure-helper.sh
flanders 20 5 12 69 1,848 1,246 26 1.3 2,512 abuild
flanders 18 3 12 151 1,239 910 23 1.3 2,698 Build.sh
flanders 18 2 5 201 344 295 9 0.5 3,123 t9300-fast-import.sh
flanders 42 NA NA 228 3,484 2,884 46 1.1 5,160 test-cmd-util.sh
flanders 46 12 30 264 4,425 3,303 50 1.1 9,661 ltmain.sh
flanders 83 16 52 631 8,964 7,205 108 1.3 20,489 configure
flanders 203 34 140 2,440 25,779 25,994 293 1.4 69,779 configure-coreutils
lenny 3 1 2 29 84 85 2 0.8 1,679 t4014-format-patch.sh
lenny 4 1 3 3 286 265 5 1.3 1,733 functions
lenny 5 NA NA 6 304 277 5 0.9 2,029 configure-helper.sh
lenny 6 1 4 5 437 395 8 1.4 2,512 abuild
lenny 4 1 3 24 325 302 6 1.4 2,698 Build.sh
lenny 5 1 2 32 95 96 3 0.6 3,123 t9300-fast-import.sh
lenny 16 NA NA 9 957 885 17 1.0 5,160 test-cmd-util.sh
lenny 18 3 12 13 1,139 972 20 1.1 9,661 ltmain.sh
lenny 35 7 24 49 2,432 2,167 32 0.9 20,489 configure
lenny 80 17 57 197 7,022 6,557 94 1.2 69,779 configure-coreutils

Parsing Rate (lines/ms)

host label bash dash mksh zsh osh-ovm osh-cpython oil-native num lines filename
flanders 126.6 865.9 277.8 10.3 5.6 5.7 175.1 1,679 t4014-format-patch.sh
flanders 114.9 506.9 124.6 38.0 1.7 1.8 90.9 1,733 functions
flanders 99.1 NA NA 23.7 1.6 2.1 100.2 2,029 configure-helper.sh
flanders 125.7 480.2 207.6 36.6 1.4 2.0 95.7 2,512 abuild
flanders 152.9 835.8 234.4 17.9 2.2 3.0 115.1 2,698 Build.sh
flanders 172.1 1,341.5 570.3 15.5 9.1 10.6 360.7 3,123 t9300-fast-import.sh
flanders 123.4 NA NA 22.6 1.5 1.8 111.1 5,160 test-cmd-util.sh
flanders 210.0 796.5 326.1 36.7 2.2 2.9 192.8 9,661 ltmain.sh
flanders 246.7 1,297.6 396.7 32.5 2.3 2.8 189.8 20,489 configure
flanders 344.2 2,081.2 498.6 28.6 2.7 2.7 238.2 69,779 configure-coreutils
lenny 580.2 2,551.7 1,074.9 58.7 20.0 19.7 728.4 1,679 t4014-format-patch.sh
lenny 459.0 1,810.9 647.1 553.5 6.1 6.5 345.4 1,733 functions
lenny 371.1 NA NA 333.0 6.7 7.3 398.4 2,029 configure-helper.sh
lenny 451.6 2,039.0 612.1 535.0 5.8 6.4 331.0 2,512 abuild
lenny 599.8 2,276.8 971.6 113.6 8.3 8.9 419.7 2,698 Build.sh
lenny 668.3 3,908.6 1,589.3 98.7 32.9 32.5 1,081.7 3,123 t9300-fast-import.sh
lenny 317.2 NA NA 588.9 5.4 5.8 308.5 5,160 test-cmd-util.sh
lenny 529.5 2,848.2 796.8 732.3 8.5 9.9 495.2 9,661 ltmain.sh
lenny 579.2 2,914.1 858.4 420.1 8.4 9.5 632.9 20,489 configure
lenny 868.2 4,111.7 1,216.2 354.4 9.9 10.6 742.2 69,779 configure-coreutils

Memory Usage (Max Resident Set Size in MB)

Again, Oil uses a different algorithm (and language) than POSIX shells. It builds an AST in memory rather than just validating the code line-by-line.

host label bash dash mksh zsh osh-ovm osh-cpython oil-native num lines filename
flanders 3.1 1.6 1.7 3.7 13.6 17.0 5.4 1,679 t4014-format-patch.sh
flanders 3.1 1.6 1.7 3.7 16.4 19.5 7.6 1,733 functions
flanders 3.1 NA NA 3.8 17.0 20.5 8.1 2,029 configure-helper.sh
flanders 3.1 1.7 1.7 3.7 18.3 21.7 9.6 2,512 abuild
flanders 3.2 1.7 1.8 3.8 17.3 20.4 8.4 2,698 Build.sh
flanders 3.2 1.6 1.7 3.8 14.1 17.2 5.7 3,123 t9300-fast-import.sh
flanders 3.2 NA NA 3.7 27.3 31.2 17.4 5,160 test-cmd-util.sh
flanders 4.3 2.2 2.7 4.1 28.9 33.0 19.5 9,661 ltmain.sh
flanders 3.9 2.3 2.3 4.2 49.2 54.7 38.0 20,489 configure
flanders 3.4 1.7 1.8 3.8 111.7 121.4 102.8 69,779 configure-coreutils
lenny 3.3 1.7 1.8 4.1 14.2 17.7 5.6 1,679 t4014-format-patch.sh
lenny 3.4 1.7 1.8 4.0 16.9 20.6 7.8 1,733 functions
lenny 3.4 NA NA 4.1 17.4 21.4 8.3 2,029 configure-helper.sh
lenny 3.5 1.8 1.9 4.0 19.0 23.0 10.0 2,512 abuild
lenny 3.5 1.7 2.0 4.1 17.9 21.7 8.5 2,698 Build.sh
lenny 3.5 1.8 1.9 4.1 14.5 18.1 5.8 3,123 t9300-fast-import.sh
lenny 3.4 NA NA 4.0 28.0 33.2 17.5 5,160 test-cmd-util.sh
lenny 4.3 2.2 3.1 4.6 29.4 35.1 19.7 9,661 ltmain.sh
lenny 4.0 2.5 2.7 4.4 49.6 58.0 38.2 20,489 configure
lenny 3.7 1.8 2.2 4.0 112.1 130.1 102.9 69,779 configure-coreutils

Shell and Host Details

shell label shell id
bash bash-9af8f89f
dash dash-9ff48631
mksh mksh-e3755400
oil-native osh_eval.opt.stripped-b0ca4fa9
host label host id
flanders flanders-63f04b74
lenny lenny-c758ab9d

Raw Data

filename
no-host.2021-09-13__22-31-18.cachegrind.tsv
flanders.2021-07-17__23-59-20.times.csv
lenny.2021-07-18__00-08-08.times.csv