OSH Parser Performance

We time $sh -n $file for various files under various shells, and repeat then run under cachegrind for stable metrics.

Summary

Instructions Per Line (via cachegrind)

Lower numbers are generally better, but each shell recognizes a different language, and Oil uses a more thorough parsing algorithm. In thousands of "I refs".

shell label total lines thousand irefs per line
bash 118,863 15.3
dash 111,674 1.6
mksh 111,674 5.7
oil-native 118,863 14.0

(zsh isn't measured because zsh -n unexpectedly forks.)

Average Parsing Rate, Measured on Two Machines (lines/ms)

Shell startup time is included in the elapsed time measurements, but long files are chosen to minimize its effect.

shell label total lines host broome host lenny
dash 111,674 1,534.7 3,362.1
mksh 111,674 592.7 1,147.9
bash 118,863 350.1 608.7
oil-native 118,863 248.1 422.8
zsh 118,863 104.8 286.5
osh-cpython 118,863 7.4 9.6
osh-ovm 118,863 6.6 8.7

Breakdown By File

Instructions Per Line (in thousands)

bash dash mksh oil-native num lines filename
15.6 1.3 4.4 7.1 1,679 t4014-format-patch.sh
16.7 2.1 8.4 21.4 1,733 functions
23.7 NA NA 19.8 2,029 configure-helper.sh
17.8 2.1 8.8 22.5 2,512 abuild
13.6 1.8 5.9 16.0 2,698 Build.sh
13.7 1.0 3.3 4.6 3,123 t9300-fast-import.sh
32.0 NA NA 24.3 5,160 test-cmd-util.sh
18.2 1.8 6.1 14.7 9,661 ltmain.sh
17.4 1.9 6.4 15.3 20,489 configure
12.9 1.5 5.4 12.7 69,779 configure-coreutils

Elasped Time (milliseconds)

host label bash dash mksh zsh osh-ovm osh-cpython oil-native osh to bash ratio num lines filename
broome 9 2 4 99 158 164 7 0.9 1,679 t4014-format-patch.sh
broome 10 3 7 11 459 428 16 1.6 1,733 functions
broome 15 NA NA 19 428 391 17 1.1 2,029 configure-helper.sh
broome 15 4 11 16 651 584 21 1.4 2,512 abuild
broome 12 3 7 78 444 408 19 1.5 2,698 Build.sh
broome 11 2 5 112 135 138 8 0.7 3,123 t9300-fast-import.sh
broome 32 NA NA 28 1,345 1,165 32 1.0 5,160 test-cmd-util.sh
broome 32 8 22 41 1,509 1,349 37 1.1 9,661 ltmain.sh
broome 61 17 39 118 3,302 2,902 87 1.4 20,489 configure
broome 141 35 93 614 9,659 8,576 235 1.7 69,779 configure-coreutils
lenny 3 1 2 41 93 97 3 0.9 1,679 t4014-format-patch.sh
lenny 5 1 3 3 309 280 8 1.8 1,733 functions
lenny 7 NA NA 8 329 303 11 1.7 2,029 configure-helper.sh
lenny 7 1 4 6 460 417 13 1.8 2,512 abuild
lenny 7 1 3 30 332 312 9 1.4 2,698 Build.sh
lenny 6 1 2 40 99 102 3 0.6 3,123 t9300-fast-import.sh
lenny 18 NA NA 10 1,008 920 23 1.3 5,160 test-cmd-util.sh
lenny 21 4 12 16 1,158 1,018 26 1.2 9,661 ltmain.sh
lenny 38 7 22 45 2,519 2,340 51 1.3 20,489 configure
lenny 84 17 51 216 7,374 6,605 133 1.6 69,779 configure-coreutils

Parsing Rate (lines/ms)

host label bash dash mksh zsh osh-ovm osh-cpython oil-native num lines filename
broome 196.1 911.5 406.6 17.0 10.6 10.3 230.3 1,679 t4014-format-patch.sh
broome 167.9 649.6 234.2 157.7 3.8 4.0 105.3 1,733 functions
broome 135.7 NA NA 109.3 4.7 5.2 118.5 2,029 configure-helper.sh
broome 165.7 702.5 224.6 160.2 3.9 4.3 117.0 2,512 abuild
broome 216.1 845.5 362.9 34.7 6.1 6.6 145.2 2,698 Build.sh
broome 272.2 1,429.3 595.1 28.0 23.2 22.6 391.1 3,123 t9300-fast-import.sh
broome 160.4 NA NA 186.8 3.8 4.4 161.2 5,160 test-cmd-util.sh
broome 299.0 1,222.9 440.8 237.3 6.4 7.2 264.6 9,661 ltmain.sh
broome 336.9 1,213.2 531.6 173.5 6.2 7.1 236.0 20,489 configure
broome 493.9 2,021.6 753.9 113.6 7.2 8.1 297.2 69,779 configure-coreutils
lenny 526.7 2,669.3 1,021.3 41.3 18.1 17.4 617.7 1,679 t4014-format-patch.sh
lenny 383.5 1,633.4 615.2 501.3 5.6 6.2 213.8 1,733 functions
lenny 309.0 NA NA 254.9 6.2 6.7 180.0 2,029 configure-helper.sh
lenny 350.0 1,701.9 691.2 449.9 5.5 6.0 195.2 2,512 abuild
lenny 396.2 1,972.2 962.5 90.6 8.1 8.6 290.0 2,698 Build.sh
lenny 541.7 3,831.9 1,493.5 78.7 31.4 30.5 940.4 3,123 t9300-fast-import.sh
lenny 285.5 NA NA 503.0 5.1 5.6 219.9 5,160 test-cmd-util.sh
lenny 451.7 2,619.6 819.4 601.2 8.3 9.5 376.9 9,661 ltmain.sh
lenny 535.9 2,865.2 941.3 453.3 8.1 8.8 398.8 20,489 configure
lenny 835.1 4,097.9 1,375.2 322.7 9.5 10.6 524.5 69,779 configure-coreutils

Memory Usage (Max Resident Set Size in MB)

Again, Oil uses a different algorithm (and language) than POSIX shells. It builds an AST in memory rather than just validating the code line-by-line.

host label bash dash mksh zsh osh-ovm osh-cpython oil-native num lines filename
broome 3.4 1.7 1.8 4.0 14.3 18.2 5.5 1,679 t4014-format-patch.sh
broome 3.4 1.8 1.9 3.9 17.2 21.1 8.2 1,733 functions
broome 3.5 NA NA 4.0 17.7 21.8 8.6 2,029 configure-helper.sh
broome 3.4 1.7 1.9 4.1 19.4 23.4 10.4 2,512 abuild
broome 3.4 1.7 2.0 4.1 18.0 21.8 9.1 2,698 Build.sh
broome 3.5 1.6 1.9 4.0 14.7 18.5 5.7 3,123 t9300-fast-import.sh
broome 3.6 NA NA 4.1 28.4 33.4 18.6 5,160 test-cmd-util.sh
broome 4.5 2.2 2.9 4.5 29.8 35.5 21.2 9,661 ltmain.sh
broome 4.1 2.4 2.7 4.4 49.9 58.8 41.9 20,489 configure
broome 3.7 1.8 2.2 4.1 112.5 130.9 110.8 69,779 configure-coreutils
lenny 3.5 1.7 1.9 4.0 14.4 18.3 5.4 1,679 t4014-format-patch.sh
lenny 3.4 1.7 1.9 4.0 17.2 21.1 8.3 1,733 functions
lenny 3.5 NA NA 4.0 17.5 21.8 8.7 2,029 configure-helper.sh
lenny 3.5 1.8 1.9 4.0 19.4 23.5 10.5 2,512 abuild
lenny 3.4 1.7 2.0 3.9 18.0 22.1 9.1 2,698 Build.sh
lenny 3.4 1.7 1.9 4.0 14.7 18.5 5.9 3,123 t9300-fast-import.sh
lenny 3.6 NA NA 4.1 28.2 33.5 18.5 5,160 test-cmd-util.sh
lenny 4.4 2.1 3.0 4.5 29.8 35.5 21.2 9,661 ltmain.sh
lenny 4.0 2.5 2.7 4.5 49.8 58.8 41.9 20,489 configure
lenny 3.7 1.9 2.3 4.2 112.9 130.6 110.9 69,779 configure-coreutils

Shell and Host Details

shell label shell id
bash bash-9af8f89f
dash dash-9ff48631
mksh mksh-e3755400
oil-native osh_eval.stripped-5c1e24e4
host label host id
broome broome-8a85d07d
lenny lenny-23f35694

Raw Data

filename
no-host.2022-08-02__02-41-02.cachegrind.tsv
broome.2022-08-02__02-41-03.times.csv
lenny.2022-08-01__22-36-12.times.csv