OSH Parser Performance

We time $sh -n $file for various files under various shells, and repeat then run under cachegrind for stable metrics.

Summary

Instructions Per Line (via cachegrind)

Lower numbers are generally better, but each shell recognizes a different language, and Oil uses a more thorough parsing algorithm. In thousands of "I refs".

shell label total lines thousand irefs per line
bash 118,863 15.3
dash 111,674 1.6
mksh 111,674 5.7
oil-native 118,863 8.5

(zsh isn't measured because zsh -n unexpectedly forks.)

Average Parsing Rate, Measured on Two Machines (lines/ms)

Shell startup time is included in the elapsed time measurements, but long files are chosen to minimize its effect.

shell label total lines host flanders host lenny
dash 111,674 1,336.2 4,019.8
mksh 111,674 430.8 1,167.3
oil-native 118,863 216.9 661.8
bash 118,863 246.3 661.7
zsh 118,863 31.8 324.4
osh-cpython 118,863 3.7 10.3
osh-ovm 118,863 2.9 9.6

Breakdown By File

Instructions Per Line (in thousands)

bash dash mksh oil-native num lines filename
15.7 1.4 4.4 4.7 1,679 t4014-format-patch.sh
16.7 2.1 8.4 13.3 1,733 functions
23.7 NA NA 12.0 2,029 configure-helper.sh
17.9 2.2 8.8 13.9 2,512 abuild
13.6 1.8 5.9 9.6 2,698 Build.sh
13.7 1.0 3.3 3.1 3,123 t9300-fast-import.sh
32.0 NA NA 14.8 5,160 test-cmd-util.sh
18.2 1.8 6.1 9.1 9,661 ltmain.sh
17.3 1.9 6.4 9.2 20,489 configure
12.9 1.5 5.4 7.7 69,779 configure-coreutils

Elasped Time (milliseconds)

host label bash dash mksh zsh osh-ovm osh-cpython oil-native osh to bash ratio num lines filename
flanders 11 3 5 164 240 214 9 0.8 1,679 t4014-format-patch.sh
flanders 13 4 12 43 819 680 21 1.5 1,733 functions
flanders 21 NA NA 83 903 722 20 1.0 2,029 configure-helper.sh
flanders 20 4 12 63 1,402 1,026 26 1.3 2,512 abuild
flanders 18 3 13 153 1,006 828 18 1.1 2,698 Build.sh
flanders 15 3 8 202 306 266 13 0.9 3,123 t9300-fast-import.sh
flanders 43 NA NA 206 2,792 2,402 39 0.9 5,160 test-cmd-util.sh
flanders 43 11 31 239 3,270 2,490 51 1.2 9,661 ltmain.sh
flanders 86 17 54 593 7,047 5,734 96 1.1 20,489 configure
flanders 212 38 125 1,986 23,149 17,747 256 1.2 69,779 configure-coreutils
lenny 4 1 1 29 81 85 2 0.6 1,679 t4014-format-patch.sh
lenny 5 1 2 3 274 257 5 1.1 1,733 functions
lenny 7 NA NA 6 295 283 5 0.8 2,029 configure-helper.sh
lenny 7 1 4 5 430 381 7 1.1 2,512 abuild
lenny 7 1 3 24 303 285 6 0.8 2,698 Build.sh
lenny 6 1 2 32 92 91 3 0.4 3,123 t9300-fast-import.sh
lenny 16 NA NA 9 893 802 13 0.8 5,160 test-cmd-util.sh
lenny 17 3 11 14 1,054 947 16 0.9 9,661 ltmain.sh
lenny 34 6 25 46 2,325 2,095 34 1.0 20,489 configure
lenny 78 15 48 199 6,647 6,308 89 1.1 69,779 configure-coreutils

Parsing Rate (lines/ms)

host label bash dash mksh zsh osh-ovm osh-cpython oil-native num lines filename
flanders 149.0 657.9 307.8 10.2 7.0 7.8 195.2 1,679 t4014-format-patch.sh
flanders 128.6 468.6 139.7 40.5 2.1 2.5 84.1 1,733 functions
flanders 96.5 NA NA 24.3 2.2 2.8 101.4 2,029 configure-helper.sh
flanders 127.9 587.2 217.2 39.8 1.8 2.4 97.1 2,512 abuild
flanders 153.3 772.6 208.2 17.6 2.7 3.3 145.9 2,698 Build.sh
flanders 207.2 1,001.0 405.5 15.5 10.2 11.7 243.4 3,123 t9300-fast-import.sh
flanders 118.6 NA NA 25.1 1.8 2.1 132.0 5,160 test-cmd-util.sh
flanders 224.7 862.7 314.1 40.4 3.0 3.9 188.6 9,661 ltmain.sh
flanders 237.9 1,196.1 380.7 34.6 2.9 3.6 214.2 20,489 configure
flanders 329.2 1,831.3 560.3 35.1 3.0 3.9 273.0 69,779 configure-coreutils
lenny 435.5 2,673.6 1,265.3 58.7 20.6 19.8 762.5 1,679 t4014-format-patch.sh
lenny 375.6 1,770.2 770.9 574.8 6.3 6.7 350.5 1,733 functions
lenny 305.3 NA NA 344.9 6.9 7.2 384.0 2,029 configure-helper.sh
lenny 376.7 2,203.5 665.3 518.8 5.8 6.6 354.2 2,512 abuild
lenny 406.8 2,681.9 958.8 111.1 8.9 9.5 482.0 2,698 Build.sh
lenny 524.2 4,493.5 1,635.9 98.6 33.8 34.2 1,188.4 3,123 t9300-fast-import.sh
lenny 327.5 NA NA 551.0 5.8 6.4 392.5 5,160 test-cmd-util.sh
lenny 553.3 3,278.2 872.4 686.2 9.2 10.2 607.8 9,661 ltmain.sh
lenny 604.0 3,481.6 823.3 446.3 8.8 9.8 609.5 20,489 configure
lenny 893.2 4,812.0 1,464.9 351.0 10.5 11.1 782.2 69,779 configure-coreutils

Memory Usage (Max Resident Set Size in MB)

Again, Oil uses a different algorithm (and language) than POSIX shells. It builds an AST in memory rather than just validating the code line-by-line.

host label bash dash mksh zsh osh-ovm osh-cpython oil-native num lines filename
flanders 3.0 1.8 1.7 3.7 13.7 16.6 5.4 1,679 t4014-format-patch.sh
flanders 3.1 1.8 1.8 3.7 16.2 19.7 7.7 1,733 functions
flanders 3.2 NA NA 3.7 16.9 20.3 8.1 2,029 configure-helper.sh
flanders 3.1 1.7 1.7 3.6 18.4 21.7 9.7 2,512 abuild
flanders 3.1 1.8 1.8 3.7 17.0 20.4 8.5 2,698 Build.sh
flanders 3.0 1.7 1.7 3.7 13.8 17.4 5.6 3,123 t9300-fast-import.sh
flanders 3.3 NA NA 3.7 27.5 31.2 17.4 5,160 test-cmd-util.sh
flanders 4.2 2.1 2.9 4.3 28.9 32.7 19.5 9,661 ltmain.sh
flanders 3.9 2.3 2.4 4.0 49.2 54.5 38.0 20,489 configure
flanders 3.4 1.7 1.8 3.7 111.8 121.5 102.7 69,779 configure-coreutils
lenny 3.3 1.7 1.9 4.0 14.1 17.7 5.3 1,679 t4014-format-patch.sh
lenny 3.4 1.7 1.9 3.9 16.8 20.8 7.8 1,733 functions
lenny 3.4 NA NA 4.1 17.4 21.2 8.2 2,029 configure-helper.sh
lenny 3.3 1.7 1.9 4.0 18.9 22.9 9.8 2,512 abuild
lenny 3.3 1.7 2.0 4.1 17.7 21.6 8.5 2,698 Build.sh
lenny 3.3 1.7 1.9 4.1 14.4 18.1 5.9 3,123 t9300-fast-import.sh
lenny 3.6 NA NA 4.0 28.0 33.4 17.6 5,160 test-cmd-util.sh
lenny 4.4 2.2 3.1 4.6 29.4 35.0 19.7 9,661 ltmain.sh
lenny 4.1 2.5 2.7 4.3 49.7 58.2 38.1 20,489 configure
lenny 3.7 1.8 2.3 4.0 112.3 129.8 103.0 69,779 configure-coreutils

Shell and Host Details

shell label shell id
bash bash-9af8f89f
dash dash-9ff48631
mksh mksh-e3755400
oil-native osh_eval.opt.stripped-4a6fedf8
host label host id
flanders flanders-63f04b74
lenny lenny-c758ab9d

Raw Data

filename
no-host.2021-07-05__23-56-52.cachegrind.tsv
flanders.2021-07-05__23-43-18.times.csv
lenny.2021-07-05__23-57-34.times.csv