OSH Parser Performance

We time $sh -n $file for various files under various shells, and repeat then run under cachegrind for stable metrics.

Summary

Instructions Per Line (via cachegrind)

Lower numbers are generally better, but each shell recognizes a different language, and Oil uses a more thorough parsing algorithm. In thousands of "I refs".

shell label total lines thousand irefs per line
bash 118,863 15.4
dash 111,674 1.6
mksh 111,674 5.7
oil-native 118,863 44.4

(zsh isn't measured because zsh -n unexpectedly forks.)

Average Parsing Rate, Measured on Two Machines (lines/ms)

Shell startup time is included in the elapsed time measurements, but long files are chosen to minimize its effect.

shell label total lines host broome host lenny
dash 111,674 1,499.7 3,666.5
mksh 111,674 586.4 1,149.3
bash 118,863 350.0 637.1
zsh 118,863 104.6 293.3
oil-native 118,863 52.3 50.0
osh-cpython 118,863 7.4 9.6
osh-ovm 118,863 6.6 8.6

Breakdown By File

Instructions Per Line (in thousands)

bash dash mksh oil-native num lines filename
15.6 1.4 4.4 17.9 1,679 t4014-format-patch.sh
16.7 2.1 8.4 65.0 1,733 functions
23.7 NA NA 59.4 2,029 configure-helper.sh
17.9 2.2 8.9 70.1 2,512 abuild
13.6 1.8 5.9 50.1 2,698 Build.sh
13.7 1.0 3.3 12.1 3,123 t9300-fast-import.sh
32.0 NA NA 74.3 5,160 test-cmd-util.sh
18.2 1.8 6.1 46.8 9,661 ltmain.sh
17.3 1.9 6.4 48.7 20,489 configure
12.9 1.5 5.4 40.7 69,779 configure-coreutils

Elasped Time (milliseconds)

host label bash dash mksh zsh osh-ovm osh-cpython oil-native osh to bash ratio num lines filename
broome 9 2 4 97 122 158 13 1.5 1,679 t4014-format-patch.sh
broome 10 3 7 13 439 410 39 3.8 1,733 functions
broome 12 NA NA 21 427 414 42 3.4 2,029 configure-helper.sh
broome 12 3 10 18 613 602 68 5.7 2,512 abuild
broome 12 3 8 77 489 402 50 4.0 2,698 Build.sh
broome 13 2 5 113 176 137 20 1.6 3,123 t9300-fast-import.sh
broome 31 NA NA 27 1,358 1,211 137 4.4 5,160 test-cmd-util.sh
broome 35 9 24 40 1,513 1,341 163 4.7 9,661 ltmain.sh
broome 62 16 40 121 3,346 2,891 407 6.6 20,489 configure
broome 143 35 92 609 9,632 8,462 1,335 9.3 69,779 configure-coreutils
lenny 3 1 1 32 93 92 6 2.0 1,679 t4014-format-patch.sh
lenny 4 1 3 3 308 267 27 6.8 1,733 functions
lenny 5 NA NA 6 315 289 31 5.7 2,029 configure-helper.sh
lenny 5 1 3 5 463 412 46 8.3 2,512 abuild
lenny 5 1 3 26 335 299 33 7.2 2,698 Build.sh
lenny 5 1 2 34 98 98 9 1.7 3,123 t9300-fast-import.sh
lenny 20 NA NA 10 960 870 113 5.6 5,160 test-cmd-util.sh
lenny 23 3 10 16 1,137 998 153 6.6 9,661 ltmain.sh
lenny 33 6 22 52 2,576 2,250 424 12.7 20,489 configure
lenny 82 16 53 220 7,505 6,781 1,538 18.8 69,779 configure-coreutils

Parsing Rate (lines/ms)

host label bash dash mksh zsh osh-ovm osh-cpython oil-native num lines filename
broome 194.9 875.8 401.2 17.4 13.7 10.6 125.8 1,679 t4014-format-patch.sh
broome 168.3 627.9 239.2 135.4 3.9 4.2 44.1 1,733 functions
broome 163.2 NA NA 96.4 4.8 4.9 48.1 2,029 configure-helper.sh
broome 210.8 729.6 244.1 137.7 4.1 4.2 37.0 2,512 abuild
broome 217.2 845.8 347.5 34.9 5.5 6.7 53.8 2,698 Build.sh
broome 243.5 1,411.8 594.9 27.6 17.7 22.7 152.4 3,123 t9300-fast-import.sh
broome 164.9 NA NA 194.0 3.8 4.3 37.8 5,160 test-cmd-util.sh
broome 277.4 1,034.1 401.5 241.8 6.4 7.2 59.4 9,661 ltmain.sh
broome 332.2 1,255.1 514.7 168.8 6.1 7.1 50.4 20,489 configure
broome 487.0 1,978.1 759.9 114.7 7.2 8.2 52.3 69,779 configure-coreutils
lenny 549.2 2,789.0 1,141.4 51.9 18.0 18.3 268.6 1,679 t4014-format-patch.sh
lenny 434.0 1,971.6 669.1 554.4 5.6 6.5 63.9 1,733 functions
lenny 377.3 NA NA 315.9 6.4 7.0 66.5 2,029 configure-helper.sh
lenny 458.0 2,043.9 738.2 457.0 5.4 6.1 54.9 2,512 abuild
lenny 594.3 2,538.1 937.8 102.6 8.0 9.0 82.4 2,698 Build.sh
lenny 573.2 4,319.5 1,716.9 91.9 32.0 31.8 340.2 3,123 t9300-fast-import.sh
lenny 254.6 NA NA 534.4 5.4 5.9 45.7 5,160 test-cmd-util.sh
lenny 418.2 2,801.9 964.5 598.6 8.5 9.7 63.1 9,661 ltmain.sh
lenny 614.0 3,373.2 944.8 393.3 8.0 9.1 48.4 20,489 configure
lenny 851.8 4,244.5 1,309.1 317.7 9.3 10.3 45.4 69,779 configure-coreutils

Memory Usage (Max Resident Set Size in MB)

Again, Oil uses a different algorithm (and language) than POSIX shells. It builds an AST in memory rather than just validating the code line-by-line.

host label bash dash mksh zsh osh-ovm osh-cpython oil-native num lines filename
broome 3.4 1.8 1.9 4.1 14.5 18.0 6.4 1,679 t4014-format-patch.sh
broome 3.3 1.7 1.9 4.1 17.3 21.3 12.3 1,733 functions
broome 3.4 NA NA 3.9 17.9 21.8 12.9 2,029 configure-helper.sh
broome 3.4 1.8 1.7 4.1 19.5 23.7 17.4 2,512 abuild
broome 3.5 1.8 2.0 4.0 18.0 21.9 15.1 2,698 Build.sh
broome 3.4 1.7 1.8 4.1 14.8 18.3 7.1 3,123 t9300-fast-import.sh
broome 3.5 NA NA 4.1 28.3 33.4 33.0 5,160 test-cmd-util.sh
broome 4.4 2.1 2.9 4.5 29.8 35.5 38.9 9,661 ltmain.sh
broome 4.0 2.4 2.6 4.5 50.1 58.7 80.6 20,489 configure
broome 3.6 1.9 2.3 4.1 112.7 130.8 230.0 69,779 configure-coreutils
lenny 3.4 1.7 1.9 4.1 14.6 17.9 6.4 1,679 t4014-format-patch.sh
lenny 3.4 1.7 2.0 4.0 17.4 21.2 12.3 1,733 functions
lenny 3.5 NA NA 4.1 17.8 21.8 12.8 2,029 configure-helper.sh
lenny 3.4 1.8 1.9 4.0 19.4 23.6 17.5 2,512 abuild
lenny 3.4 1.7 2.0 4.2 18.0 22.1 15.1 2,698 Build.sh
lenny 3.5 1.6 1.8 3.9 15.0 18.6 7.0 3,123 t9300-fast-import.sh
lenny 3.5 NA NA 4.1 28.3 33.5 33.0 5,160 test-cmd-util.sh
lenny 4.2 2.1 3.1 4.5 29.9 35.6 38.9 9,661 ltmain.sh
lenny 4.0 2.5 2.7 4.5 50.1 58.5 80.6 20,489 configure
lenny 3.8 1.8 2.1 4.2 112.7 130.7 230.0 69,779 configure-coreutils

Shell and Host Details

shell label shell id
bash bash-9af8f89f
dash dash-9ff48631
mksh mksh-e3755400
oil-native osh_eval.stripped-850fa8be
host label host id
broome broome-8a85d07d
lenny lenny-b7f988ad

Raw Data

filename
no-host.2022-09-21__22-47-16.cachegrind.tsv
broome.2022-09-22__02-26-18.times.csv
lenny.2022-09-21__22-48-26.times.csv