OSH Parser Performance

We time $sh -n $file for various files under various shells, and repeat then run under cachegrind for stable metrics.

Summary

Instructions Per Line (via cachegrind)

Lower numbers are generally better, but each shell recognizes a different language, and Oil uses a more thorough parsing algorithm. In thousands of "I refs".

shell label total lines thousand irefs per line
bash 118,863 15.3
dash 111,674 1.6
mksh 111,674 5.7
oil-native 118,863 8.7

(zsh isn't measured because zsh -n unexpectedly forks.)

Average Parsing Rate, Measured on Two Machines (lines/ms)

Shell startup time is included in the elapsed time measurements, but long files are chosen to minimize its effect.

shell label total lines host broome host lenny
dash 111,674 1,464.5 3,745.9
mksh 111,674 598.0 1,199.9
bash 118,863 352.2 640.4
oil-native 118,863 308.9 639.2
zsh 118,863 110.1 306.0
osh-cpython 118,863 6.6 9.9
osh-ovm 118,863 5.9 9.1

Breakdown By File

Instructions Per Line (in thousands)

bash dash mksh oil-native num lines filename
15.7 1.4 4.4 4.8 1,679 t4014-format-patch.sh
16.7 2.1 8.4 13.6 1,733 functions
23.7 NA NA 12.3 2,029 configure-helper.sh
17.9 2.2 8.8 14.2 2,512 abuild
13.7 1.8 5.9 9.8 2,698 Build.sh
13.7 1.0 3.3 3.1 3,123 t9300-fast-import.sh
32.0 NA NA 15.1 5,160 test-cmd-util.sh
18.2 1.8 6.1 9.3 9,661 ltmain.sh
17.3 1.9 6.4 9.4 20,489 configure
12.9 1.5 5.4 7.8 69,779 configure-coreutils

Elasped Time (milliseconds)

host label bash dash mksh zsh osh-ovm osh-cpython oil-native osh to bash ratio num lines filename
broome 8 2 4 93 153 157 6 0.7 1,679 t4014-format-patch.sh
broome 10 3 7 14 473 458 14 1.4 1,733 functions
broome 14 NA NA 19 522 452 14 1.0 2,029 configure-helper.sh
broome 13 4 10 17 703 623 15 1.2 2,512 abuild
broome 12 3 8 71 501 455 12 1.0 2,698 Build.sh
broome 12 2 5 100 174 166 7 0.6 3,123 t9300-fast-import.sh
broome 32 NA NA 26 1,451 1,351 29 0.9 5,160 test-cmd-util.sh
broome 33 9 21 41 1,732 1,507 34 1.0 9,661 ltmain.sh
broome 61 18 38 117 3,666 3,271 68 1.1 20,489 configure
broome 141 36 93 582 10,782 9,618 184 1.3 69,779 configure-coreutils
lenny 3 1 2 31 88 89 2 0.7 1,679 t4014-format-patch.sh
lenny 4 1 2 4 292 274 5 1.2 1,733 functions
lenny 6 NA NA 6 307 287 5 0.9 2,029 configure-helper.sh
lenny 6 1 3 5 442 407 8 1.3 2,512 abuild
lenny 5 1 3 27 321 296 7 1.3 2,698 Build.sh
lenny 5 1 2 34 98 97 3 0.5 3,123 t9300-fast-import.sh
lenny 18 NA NA 9 949 853 16 0.9 5,160 test-cmd-util.sh
lenny 22 3 10 14 1,093 991 17 0.8 9,661 ltmain.sh
lenny 37 7 21 55 2,387 2,204 35 0.9 20,489 configure
lenny 79 16 50 203 7,071 6,537 89 1.1 69,779 configure-coreutils

Parsing Rate (lines/ms)

host label bash dash mksh zsh osh-ovm osh-cpython oil-native num lines filename
broome 200.5 879.5 423.1 18.0 11.0 10.7 274.0 1,679 t4014-format-patch.sh
broome 171.4 635.0 231.5 127.2 3.7 3.8 124.2 1,733 functions
broome 141.2 NA NA 108.4 3.9 4.5 146.2 2,029 configure-helper.sh
broome 194.5 699.3 240.9 144.8 3.6 4.0 166.9 2,512 abuild
broome 221.2 851.9 350.5 38.0 5.4 5.9 216.3 2,698 Build.sh
broome 254.2 1,447.2 596.8 31.3 17.9 18.8 430.5 3,123 t9300-fast-import.sh
broome 161.1 NA NA 195.5 3.6 3.8 178.1 5,160 test-cmd-util.sh
broome 289.7 1,037.1 470.9 237.3 5.6 6.4 281.2 9,661 ltmain.sh
broome 336.7 1,160.8 535.2 175.5 5.6 6.3 299.7 20,489 configure
broome 494.9 1,952.6 749.1 119.8 6.5 7.3 378.5 69,779 configure-coreutils
lenny 522.9 2,757.0 1,083.2 54.6 19.0 18.8 777.0 1,679 t4014-format-patch.sh
lenny 443.1 1,992.0 729.7 401.1 5.9 6.3 379.1 1,733 functions
lenny 361.5 NA NA 330.2 6.6 7.1 397.4 2,029 configure-helper.sh
lenny 402.7 1,945.8 733.4 499.9 5.7 6.2 320.1 2,512 abuild
lenny 532.4 2,652.9 931.0 99.5 8.4 9.1 409.2 2,698 Build.sh
lenny 603.4 3,808.5 1,702.8 92.9 31.9 32.1 1,230.5 3,123 t9300-fast-import.sh
lenny 280.2 NA NA 545.3 5.4 6.1 327.0 5,160 test-cmd-util.sh
lenny 449.3 3,359.2 963.4 692.7 8.8 9.7 568.5 9,661 ltmain.sh
lenny 546.5 3,108.6 995.4 371.7 8.6 9.3 587.0 20,489 configure
lenny 883.5 4,433.8 1,385.1 344.0 9.9 10.7 779.9 69,779 configure-coreutils

Memory Usage (Max Resident Set Size in MB)

Again, Oil uses a different algorithm (and language) than POSIX shells. It builds an AST in memory rather than just validating the code line-by-line.

host label bash dash mksh zsh osh-ovm osh-cpython oil-native num lines filename
broome 3.3 1.7 1.9 4.0 14.5 17.9 5.6 1,679 t4014-format-patch.sh
broome 3.4 1.8 1.9 4.0 17.2 21.0 7.9 1,733 functions
broome 3.4 NA NA 4.0 17.8 21.4 8.5 2,029 configure-helper.sh
broome 3.5 1.7 1.9 4.1 19.0 23.4 9.9 2,512 abuild
broome 3.4 1.7 2.0 4.1 17.8 21.9 8.6 2,698 Build.sh
broome 3.5 1.8 1.9 4.0 14.6 18.1 5.9 3,123 t9300-fast-import.sh
broome 3.6 NA NA 3.9 27.9 33.4 17.7 5,160 test-cmd-util.sh
broome 4.4 2.2 3.1 4.5 29.5 35.1 19.8 9,661 ltmain.sh
broome 4.0 2.5 2.6 4.4 49.8 58.4 38.2 20,489 configure
broome 3.6 1.9 2.1 4.1 112.4 130.6 103.3 69,779 configure-coreutils
lenny 3.5 1.8 1.8 4.1 14.4 17.7 5.6 1,679 t4014-format-patch.sh
lenny 3.5 1.8 1.9 3.9 16.9 20.7 8.0 1,733 functions
lenny 3.4 NA NA 4.0 17.7 21.4 8.2 2,029 configure-helper.sh
lenny 3.5 1.7 2.0 4.1 19.4 23.2 10.1 2,512 abuild
lenny 3.4 1.8 2.0 4.2 17.8 21.8 8.8 2,698 Build.sh
lenny 3.5 1.7 2.0 4.1 14.8 18.3 6.0 3,123 t9300-fast-import.sh
lenny 3.5 NA NA 4.0 27.9 33.4 17.6 5,160 test-cmd-util.sh
lenny 4.5 2.2 3.1 4.6 29.5 35.4 19.8 9,661 ltmain.sh
lenny 4.0 2.5 2.7 4.5 49.7 58.3 38.4 20,489 configure
lenny 3.8 1.9 2.3 4.1 112.7 130.5 103.1 69,779 configure-coreutils

Shell and Host Details

shell label shell id
bash bash-9af8f89f
dash dash-9ff48631
mksh mksh-e3755400
oil-native osh_eval.opt.stripped-217c42a2
host label host id
broome broome-d80ab850
lenny lenny-ec64f10b

Raw Data

filename
no-host.2022-01-27__21-01-19.cachegrind.tsv
broome.2022-01-28__02-04-13.times.csv
lenny.2022-01-27__21-01-59.times.csv