OSH Parser Performance

We time $sh -n $file for various files under various shells, and repeat then run under cachegrind for stable metrics.

Summary

Instructions Per Line (via cachegrind)

Lower numbers are generally better, but each shell recognizes a different language, and Oil uses a more thorough parsing algorithm. In thousands of "I refs".

shell label total lines thousand irefs per line
bash 118,863 15.3
dash 111,674 1.6
mksh 111,674 5.7
oil-native 118,863 8.7

(zsh isn't measured because zsh -n unexpectedly forks.)

Average Parsing Rate, Measured on Two Machines (lines/ms)

Shell startup time is included in the elapsed time measurements, but long files are chosen to minimize its effect.

shell label total lines host broome host lenny
dash 111,674 1,449.1 3,309.8
mksh 111,674 601.1 1,156.8
bash 118,863 348.0 609.2
oil-native 118,863 310.5 564.3
zsh 118,863 104.8 277.3
osh-cpython 118,863 7.3 8.8
osh-ovm 118,863 6.7 8.3

Breakdown By File

Instructions Per Line (in thousands)

bash dash mksh oil-native num lines filename
15.6 1.4 4.4 4.8 1,679 t4014-format-patch.sh
16.7 2.1 8.4 13.6 1,733 functions
23.7 NA NA 12.3 2,029 configure-helper.sh
17.9 2.2 8.9 14.2 2,512 abuild
13.6 1.8 6.0 9.8 2,698 Build.sh
13.7 1.0 3.3 3.1 3,123 t9300-fast-import.sh
32.0 NA NA 15.1 5,160 test-cmd-util.sh
18.2 1.8 6.1 9.3 9,661 ltmain.sh
17.3 1.9 6.4 9.3 20,489 configure
12.9 1.5 5.4 7.8 69,779 configure-coreutils

Elasped Time (milliseconds)

host label bash dash mksh zsh osh-ovm osh-cpython oil-native osh to bash ratio num lines filename
broome 9 2 4 97 151 162 6 0.7 1,679 t4014-format-patch.sh
broome 10 3 7 13 403 404 13 1.3 1,733 functions
broome 14 NA NA 20 432 468 12 0.9 2,029 configure-helper.sh
broome 15 3 10 13 646 556 18 1.2 2,512 abuild
broome 12 3 8 77 463 434 14 1.1 2,698 Build.sh
broome 13 2 5 113 170 136 7 0.5 3,123 t9300-fast-import.sh
broome 33 NA NA 28 1,289 1,209 30 0.9 5,160 test-cmd-util.sh
broome 33 10 22 39 1,529 1,336 37 1.1 9,661 ltmain.sh
broome 59 18 37 122 3,216 2,983 67 1.1 20,489 configure
broome 144 36 92 612 9,468 8,670 178 1.2 69,779 configure-coreutils
lenny 4 1 2 33 107 103 2 0.6 1,679 t4014-format-patch.sh
lenny 4 1 3 3 321 309 6 1.5 1,733 functions
lenny 6 NA NA 7 344 308 5 0.9 2,029 configure-helper.sh
lenny 6 1 4 5 490 452 8 1.2 2,512 abuild
lenny 6 1 3 28 356 324 7 1.2 2,698 Build.sh
lenny 6 1 2 45 115 107 3 0.5 3,123 t9300-fast-import.sh
lenny 19 NA NA 11 1,023 920 17 0.9 5,160 test-cmd-util.sh
lenny 19 4 11 17 1,179 1,084 19 1.0 9,661 ltmain.sh
lenny 37 8 21 47 2,612 2,494 39 1.0 20,489 configure
lenny 88 17 51 232 7,849 7,448 105 1.2 69,779 configure-coreutils

Parsing Rate (lines/ms)

host label bash dash mksh zsh osh-ovm osh-cpython oil-native num lines filename
broome 197.3 878.1 429.0 17.3 11.1 10.3 273.9 1,679 t4014-format-patch.sh
broome 168.5 653.0 237.0 134.8 4.3 4.3 129.2 1,733 functions
broome 141.3 NA NA 103.9 4.7 4.3 163.5 2,029 configure-helper.sh
broome 166.6 724.1 259.6 196.2 3.9 4.5 135.9 2,512 abuild
broome 216.8 839.7 345.5 35.0 5.8 6.2 191.3 2,698 Build.sh
broome 242.0 1,514.5 590.0 27.6 18.3 22.9 452.8 3,123 t9300-fast-import.sh
broome 158.2 NA NA 184.3 4.0 4.3 172.7 5,160 test-cmd-util.sh
broome 294.8 1,012.7 437.2 248.9 6.3 7.2 263.9 9,661 ltmain.sh
broome 347.0 1,141.1 549.7 167.3 6.4 6.9 305.6 20,489 configure
broome 486.1 1,924.4 755.2 114.0 7.4 8.0 392.3 69,779 configure-coreutils
lenny 442.5 2,528.6 1,105.3 50.4 15.7 16.3 742.6 1,679 t4014-format-patch.sh
lenny 411.4 1,592.8 574.8 510.5 5.4 5.6 276.1 1,733 functions
lenny 328.6 NA NA 289.9 5.9 6.6 385.5 2,029 configure-helper.sh
lenny 394.6 1,840.3 648.9 466.5 5.1 5.6 331.2 2,512 abuild
lenny 437.5 2,188.2 977.9 97.0 7.6 8.3 379.1 2,698 Build.sh
lenny 561.3 3,602.1 1,636.8 69.0 27.3 29.1 1,210.5 3,123 t9300-fast-import.sh
lenny 275.1 NA NA 490.6 5.0 5.6 302.4 5,160 test-cmd-util.sh
lenny 501.6 2,727.6 912.9 558.3 8.2 8.9 500.1 9,661 ltmain.sh
lenny 549.1 2,711.6 955.8 437.8 7.8 8.2 531.0 20,489 configure
lenny 797.5 4,004.5 1,356.4 301.0 8.9 9.4 667.2 69,779 configure-coreutils

Memory Usage (Max Resident Set Size in MB)

Again, Oil uses a different algorithm (and language) than POSIX shells. It builds an AST in memory rather than just validating the code line-by-line.

host label bash dash mksh zsh osh-ovm osh-cpython oil-native num lines filename
broome 3.3 1.7 1.9 3.9 14.4 17.7 5.5 1,679 t4014-format-patch.sh
broome 3.2 1.8 2.0 3.9 17.0 21.0 7.9 1,733 functions
broome 3.4 NA NA 4.0 17.7 21.5 8.3 2,029 configure-helper.sh
broome 3.3 1.7 1.8 4.0 19.2 23.0 9.9 2,512 abuild
broome 3.2 1.8 2.0 4.0 17.8 22.0 8.7 2,698 Build.sh
broome 3.4 1.7 1.9 4.1 14.6 18.2 5.8 3,123 t9300-fast-import.sh
broome 3.5 NA NA 4.1 28.0 33.3 17.7 5,160 test-cmd-util.sh
broome 4.4 2.2 3.0 4.4 29.4 35.3 19.9 9,661 ltmain.sh
broome 3.9 2.5 2.6 4.5 49.7 58.4 38.2 20,489 configure
broome 3.8 1.8 2.3 3.9 112.4 130.5 103.1 69,779 configure-coreutils
lenny 3.3 1.7 1.8 4.0 14.4 17.6 5.5 1,679 t4014-format-patch.sh
lenny 3.5 1.7 1.8 4.1 17.2 20.8 7.9 1,733 functions
lenny 3.4 NA NA 3.9 17.8 21.5 8.2 2,029 configure-helper.sh
lenny 3.4 1.7 2.0 4.1 19.2 23.4 10.0 2,512 abuild
lenny 3.4 1.8 2.0 3.9 17.7 21.9 8.7 2,698 Build.sh
lenny 3.4 1.7 1.9 4.1 14.6 18.2 5.9 3,123 t9300-fast-import.sh
lenny 3.5 NA NA 4.0 28.0 33.3 17.9 5,160 test-cmd-util.sh
lenny 4.4 2.1 3.1 4.5 29.5 35.0 19.9 9,661 ltmain.sh
lenny 3.9 2.5 2.7 4.3 49.9 58.4 38.5 20,489 configure
lenny 3.8 1.9 2.3 4.0 112.4 130.4 103.0 69,779 configure-coreutils

Shell and Host Details

shell label shell id
bash bash-9af8f89f
dash dash-9ff48631
mksh mksh-e3755400
oil-native osh_eval.opt.stripped-34b24f00
host label host id
broome broome-d80ab850
lenny lenny-761a4bee

Raw Data

filename
no-host.2022-04-30__21-35-19.cachegrind.tsv
broome.2022-05-01__01-12-52.times.csv
lenny.2022-04-30__21-36-04.times.csv