OSH Parser Performance

We time $sh -n $file for various files under various shells, and repeat then run under cachegrind for stable metrics.

Summary

Instructions Per Line (via cachegrind)

Lower numbers are generally better, but each shell recognizes a different language, and Oil uses a more thorough parsing algorithm. In thousands of "I refs".

shell label total lines thousand irefs per line
bash 118,863 15.4
dash 111,674 1.6
mksh 111,674 5.7
oil-native 118,863 NA

(zsh isn't measured because zsh -n unexpectedly forks.)

Average Parsing Rate, Measured on Two Machines (lines/ms)

Shell startup time is included in the elapsed time measurements, but long files are chosen to minimize its effect.

shell label total lines host broome host lenny
dash 111,674 1,457.4 3,684.6
mksh 111,674 585.6 1,255.8
bash 118,863 345.9 547.5
zsh 118,863 106.3 312.3
oil-native 118,863 147.4 232.0
osh-cpython 118,863 7.4 9.9
osh-ovm 118,863 6.6 8.7

Breakdown By File

Instructions Per Line (in thousands)

bash dash mksh oil-native num lines filename
15.6 1.4 4.4 11.7 1,679 t4014-format-patch.sh
16.7 2.1 8.4 41.6 1,733 functions
23.7 NA NA 38.5 2,029 configure-helper.sh
17.9 2.2 8.9 43.9 2,512 abuild
13.6 1.8 5.9 31.0 2,698 Build.sh
13.7 1.0 3.3 8.0 3,123 t9300-fast-import.sh
32.0 NA NA 47.0 5,160 test-cmd-util.sh
18.2 1.8 6.1 29.7 9,661 ltmain.sh
17.3 1.9 6.4 30.8 20,489 configure
12.9 1.5 5.4 25.1 69,779 configure-coreutils

Elasped Time (milliseconds)

host label bash dash mksh zsh osh-ovm osh-cpython oil-native osh to bash ratio num lines filename
broome 9 2 4 100 150 159 10 1.2 1,679 t4014-format-patch.sh
broome 10 3 7 13 413 410 23 2.2 1,733 functions
broome 15 NA NA 22 447 421 27 1.8 2,029 configure-helper.sh
broome 12 4 11 17 648 586 35 3.0 2,512 abuild
broome 12 3 8 79 447 422 27 2.2 2,698 Build.sh
broome 13 2 5 107 139 136 12 1.0 3,123 t9300-fast-import.sh
broome 32 NA NA 27 1,360 1,149 70 2.2 5,160 test-cmd-util.sh
broome 34 9 23 38 1,532 1,360 81 2.4 9,661 ltmain.sh
broome 62 18 39 122 3,300 2,959 131 2.1 20,489 configure
broome 144 36 93 594 9,698 8,506 390 2.7 69,779 configure-coreutils
lenny 4 1 2 28 89 90 4 1.1 1,679 t4014-format-patch.sh
lenny 4 1 3 3 309 271 11 2.5 1,733 functions
lenny 6 NA NA 6 311 286 11 1.8 2,029 configure-helper.sh
lenny 6 1 5 7 493 406 17 2.7 2,512 abuild
lenny 5 1 3 23 347 300 13 2.4 2,698 Build.sh
lenny 7 1 2 32 96 96 4 0.6 3,123 t9300-fast-import.sh
lenny 27 NA NA 9 996 865 34 1.2 5,160 test-cmd-util.sh
lenny 30 3 9 14 1,127 1,008 43 1.4 9,661 ltmain.sh
lenny 46 6 17 54 2,476 2,168 93 2.0 20,489 configure
lenny 81 17 48 204 7,375 6,535 284 3.5 69,779 configure-coreutils

Parsing Rate (lines/ms)

host label bash dash mksh zsh osh-ovm osh-cpython oil-native num lines filename
broome 192.4 887.0 437.2 16.8 11.2 10.6 161.3 1,679 t4014-format-patch.sh
broome 168.1 643.0 239.1 134.5 4.2 4.2 76.4 1,733 functions
broome 136.2 NA NA 93.3 4.5 4.8 76.5 2,029 configure-helper.sh
broome 210.3 687.5 225.1 148.1 3.9 4.3 71.3 2,512 abuild
broome 216.8 864.2 345.5 34.3 6.0 6.4 98.2 2,698 Build.sh
broome 240.7 1,425.4 607.1 29.2 22.4 22.9 252.4 3,123 t9300-fast-import.sh
broome 161.2 NA NA 191.4 3.8 4.5 73.7 5,160 test-cmd-util.sh
broome 280.9 1,033.8 414.7 251.8 6.3 7.1 119.3 9,661 ltmain.sh
broome 332.1 1,132.6 521.8 168.4 6.2 6.9 156.8 20,489 configure
broome 483.8 1,958.2 750.9 117.5 7.2 8.2 179.0 69,779 configure-coreutils
lenny 463.7 2,639.9 1,107.5 59.3 18.9 18.7 432.0 1,679 t4014-format-patch.sh
lenny 405.7 1,985.1 634.6 556.7 5.6 6.4 162.1 1,733 functions
lenny 318.5 NA NA 328.1 6.5 7.1 176.7 2,029 configure-helper.sh
lenny 400.1 2,184.3 541.3 343.1 5.1 6.2 150.5 2,512 abuild
lenny 526.0 2,437.2 791.0 115.1 7.8 9.0 215.8 2,698 Build.sh
lenny 444.9 4,436.1 1,380.0 97.6 32.4 32.6 778.4 3,123 t9300-fast-import.sh
lenny 191.1 NA NA 606.6 5.2 6.0 153.9 5,160 test-cmd-util.sh
lenny 320.6 3,249.6 1,102.5 709.4 8.6 9.6 225.9 9,661 ltmain.sh
lenny 441.8 3,435.4 1,193.1 379.0 8.3 9.4 219.5 20,489 configure
lenny 862.5 4,128.7 1,440.8 342.0 9.5 10.7 246.1 69,779 configure-coreutils

Memory Usage (Max Resident Set Size in MB)

Again, Oil uses a different algorithm (and language) than POSIX shells. It builds an AST in memory rather than just validating the code line-by-line.

host label bash dash mksh zsh osh-ovm osh-cpython oil-native num lines filename
broome 3.4 1.7 1.9 4.1 14.5 17.9 5.7 1,679 t4014-format-patch.sh
broome 3.4 1.6 1.9 4.0 17.3 21.2 9.1 1,733 functions
broome 3.5 NA NA 4.0 17.9 21.6 9.7 2,029 configure-helper.sh
broome 3.4 1.8 1.9 4.0 19.5 23.7 12.3 2,512 abuild
broome 3.5 1.7 1.9 4.0 18.0 21.9 10.3 2,698 Build.sh
broome 3.5 1.7 1.8 4.0 14.9 18.4 6.1 3,123 t9300-fast-import.sh
broome 3.4 NA NA 4.0 28.4 33.4 22.0 5,160 test-cmd-util.sh
broome 4.5 2.2 2.9 4.4 29.9 35.6 26.4 9,661 ltmain.sh
broome 4.0 2.4 2.6 4.4 50.0 58.6 53.2 20,489 configure
broome 3.8 1.8 2.2 4.0 112.9 130.8 144.2 69,779 configure-coreutils
lenny 3.3 1.7 1.9 4.0 14.7 18.1 5.7 1,679 t4014-format-patch.sh
lenny 3.4 1.7 1.9 4.1 17.3 21.1 9.1 1,733 functions
lenny 3.5 NA NA 4.1 17.9 21.6 9.8 2,029 configure-helper.sh
lenny 3.5 1.7 1.9 4.1 19.4 23.7 12.3 2,512 abuild
lenny 3.4 1.8 2.0 4.1 18.0 22.1 10.2 2,698 Build.sh
lenny 3.3 1.7 2.0 4.0 14.9 18.6 6.2 3,123 t9300-fast-import.sh
lenny 3.6 NA NA 4.1 28.3 33.5 22.1 5,160 test-cmd-util.sh
lenny 4.4 2.2 3.0 4.6 29.9 35.5 26.3 9,661 ltmain.sh
lenny 4.0 2.5 2.6 4.5 50.1 58.7 53.0 20,489 configure
lenny 3.9 1.9 2.3 4.2 112.7 130.8 144.3 69,779 configure-coreutils

Shell and Host Details

shell label shell id
bash bash-9af8f89f
dash dash-9ff48631
mksh mksh-e3755400
oil-native osh_eval.stripped-15aebd46
host label host id
broome broome-8a85d07d
lenny lenny-9de4b19b

Raw Data

filename
no-host.2022-10-07__13-01-52.cachegrind.tsv
broome.2022-10-07__16-48-58.times.csv
lenny.2022-10-07__13-02-51.times.csv