OSH Parser Performance

We time $sh -n $file for various files under various shells, and repeat then run under cachegrind for stable metrics.

Summary

Instructions Per Line (via cachegrind)

Lower numbers are generally better, but each shell recognizes a different language, and Oil uses a more thorough parsing algorithm. In thousands of "I refs".

shell label total lines thousand irefs per line
bash 118,863 15.3
dash 111,674 1.6
mksh 111,674 5.7
oil-native 118,863 8.2

(zsh isn't measured because zsh -n unexpectedly forks.)

Average Parsing Rate, Measured on Two Machines (lines/ms)

Shell startup time is included in the elapsed time measurements, but long files are chosen to minimize its effect.

shell label total lines host flanders host lenny
dash 111,674 1,125.9 3,129.9
mksh 111,674 346.0 1,162.3
bash 118,863 194.8 653.0
oil-native 118,863 141.3 643.9
zsh 118,863 27.4 304.3
osh-cpython 118,863 3.0 10.4
osh-ovm 118,863 2.4 9.3

Breakdown By File

Instructions Per Line (in thousands)

bash dash mksh oil-native num lines filename
15.7 1.4 4.4 4.6 1,679 t4014-format-patch.sh
16.7 2.1 8.4 12.6 1,733 functions
23.7 NA NA 11.4 2,029 configure-helper.sh
17.9 2.2 8.8 13.2 2,512 abuild
13.6 1.8 5.9 9.2 2,698 Build.sh
13.7 1.0 3.3 3.0 3,123 t9300-fast-import.sh
32.0 NA NA 14.1 5,160 test-cmd-util.sh
18.2 1.8 6.1 8.7 9,661 ltmain.sh
17.3 1.9 6.4 8.8 20,489 configure
12.9 1.5 5.4 7.3 69,779 configure-coreutils

Elasped Time (milliseconds)

host label bash dash mksh zsh osh-ovm osh-cpython oil-native osh to bash ratio num lines filename
flanders 14 3 6 169 317 262 11 0.8 1,679 t4014-format-patch.sh
flanders 16 4 10 56 1,021 909 22 1.4 1,733 functions
flanders 17 NA NA 106 1,084 973 25 1.4 2,029 configure-helper.sh
flanders 21 4 14 83 1,541 1,363 32 1.5 2,512 abuild
flanders 15 4 12 164 982 1,218 26 1.8 2,698 Build.sh
flanders 16 3 8 217 299 443 12 0.8 3,123 t9300-fast-import.sh
flanders 52 NA NA 253 3,053 3,781 62 1.2 5,160 test-cmd-util.sh
flanders 61 11 36 294 3,818 3,758 73 1.2 9,661 ltmain.sh
flanders 116 24 64 705 7,994 7,773 151 1.3 20,489 configure
flanders 283 47 173 2,300 29,362 18,802 428 1.5 69,779 configure-coreutils
lenny 3 1 2 31 90 94 3 0.8 1,679 t4014-format-patch.sh
lenny 4 1 4 3 289 256 5 1.2 1,733 functions
lenny 5 NA NA 7 290 272 5 0.9 2,029 configure-helper.sh
lenny 7 2 4 6 431 383 7 1.0 2,512 abuild
lenny 5 1 3 26 305 285 5 1.1 2,698 Build.sh
lenny 5 1 2 36 102 94 2 0.5 3,123 t9300-fast-import.sh
lenny 17 NA NA 10 902 803 15 0.9 5,160 test-cmd-util.sh
lenny 19 4 10 15 1,076 930 18 1.0 9,661 ltmain.sh
lenny 37 8 21 47 2,299 2,058 34 0.9 20,489 configure
lenny 81 18 51 210 6,940 6,227 91 1.1 69,779 configure-coreutils

Parsing Rate (lines/ms)

host label bash dash mksh zsh osh-ovm osh-cpython oil-native num lines filename
flanders 121.4 533.2 295.0 10.0 5.3 6.4 155.5 1,679 t4014-format-patch.sh
flanders 109.8 450.7 181.8 30.8 1.7 1.9 80.4 1,733 functions
flanders 119.1 NA NA 19.2 1.9 2.1 82.7 2,029 configure-helper.sh
flanders 117.5 593.0 183.4 30.3 1.6 1.8 78.4 2,512 abuild
flanders 182.0 644.2 218.5 16.5 2.7 2.2 103.6 2,698 Build.sh
flanders 199.0 1,183.0 381.4 14.4 10.5 7.1 257.2 3,123 t9300-fast-import.sh
flanders 100.1 NA NA 20.4 1.7 1.4 83.0 5,160 test-cmd-util.sh
flanders 159.0 881.6 269.9 32.9 2.5 2.6 131.5 9,661 ltmain.sh
flanders 176.7 871.5 318.3 29.1 2.6 2.6 135.6 20,489 configure
flanders 246.2 1,495.5 403.0 30.3 2.4 3.7 163.2 69,779 configure-coreutils
lenny 503.4 2,368.1 748.2 54.5 18.7 17.8 616.4 1,679 t4014-format-patch.sh
lenny 451.2 1,779.3 453.7 503.2 6.0 6.8 365.5 1,733 functions
lenny 381.0 NA NA 310.3 7.0 7.5 410.3 2,029 configure-helper.sh
lenny 381.9 1,467.3 701.3 456.7 5.8 6.6 368.4 2,512 abuild
lenny 565.5 2,327.9 1,058.0 105.5 8.8 9.5 509.8 2,698 Build.sh
lenny 648.7 2,929.6 1,585.3 86.0 30.7 33.3 1,252.2 3,123 t9300-fast-import.sh
lenny 297.4 NA NA 542.9 5.7 6.4 340.2 5,160 test-cmd-util.sh
lenny 519.6 2,451.4 971.4 637.7 9.0 10.4 545.2 9,661 ltmain.sh
lenny 559.9 2,637.3 959.7 431.5 8.9 10.0 610.7 20,489 configure
lenny 863.2 3,802.7 1,378.4 331.8 10.1 11.2 765.5 69,779 configure-coreutils

Memory Usage (Max Resident Set Size in MB)

Again, Oil uses a different algorithm (and language) than POSIX shells. It builds an AST in memory rather than just validating the code line-by-line.

host label bash dash mksh zsh osh-ovm osh-cpython oil-native num lines filename
flanders 3.1 1.8 1.6 3.7 13.6 17.0 5.2 1,679 t4014-format-patch.sh
flanders 3.1 1.7 1.7 3.6 16.2 19.7 7.3 1,733 functions
flanders 3.2 NA NA 3.7 16.7 20.2 7.8 2,029 configure-helper.sh
flanders 3.1 1.7 1.7 3.8 18.3 21.7 9.1 2,512 abuild
flanders 3.2 1.7 1.8 3.7 17.1 20.4 8.0 2,698 Build.sh
flanders 3.1 1.7 1.7 3.7 14.2 17.0 5.5 3,123 t9300-fast-import.sh
flanders 3.3 NA NA 3.7 27.0 31.0 16.5 5,160 test-cmd-util.sh
flanders 4.2 2.1 2.8 4.2 28.7 32.7 18.6 9,661 ltmain.sh
flanders 3.9 2.2 2.3 4.1 48.9 54.4 36.0 20,489 configure
flanders 3.5 1.7 1.9 3.7 111.3 121.0 97.1 69,779 configure-coreutils
lenny 3.4 1.8 1.9 4.1 13.9 17.6 5.3 1,679 t4014-format-patch.sh
lenny 3.5 1.7 1.9 4.0 16.8 20.5 7.8 1,733 functions
lenny 3.5 NA NA 4.0 17.4 21.0 8.0 2,029 configure-helper.sh
lenny 3.5 1.8 2.0 4.2 18.9 22.9 9.4 2,512 abuild
lenny 3.4 1.8 2.0 4.0 17.6 21.4 8.6 2,698 Build.sh
lenny 3.5 1.7 1.9 4.1 14.4 18.0 5.8 3,123 t9300-fast-import.sh
lenny 3.6 NA NA 4.0 27.6 32.9 16.8 5,160 test-cmd-util.sh
lenny 4.4 2.2 3.0 4.4 29.3 35.0 18.8 9,661 ltmain.sh
lenny 4.0 2.5 2.6 4.3 49.3 57.8 36.3 20,489 configure
lenny 3.8 1.8 2.3 4.1 111.7 129.5 97.4 69,779 configure-coreutils

Shell and Host Details

shell label shell id
bash bash-9af8f89f
dash dash-9ff48631
mksh mksh-e3755400
oil-native osh_eval.opt.stripped-f952c68a
host label host id
flanders flanders-63f04b74
lenny lenny-013e16cc

Raw Data

filename
no-host.2021-05-10__13-55-48.cachegrind.tsv
flanders.2021-05-10__12-20-50.times.csv
lenny.2021-05-10__13-56-31.times.csv