OSH Parser Performance

We time $sh -n $file for various files under various shells, and repeat then run under cachegrind for stable metrics.

Summary

Instructions Per Line (via cachegrind)

Lower numbers are generally better, but each shell recognizes a different language, and Oil uses a more thorough parsing algorithm. In thousands of "I refs".

shell label total lines thousand irefs per line
bash 118,863 15.5
dash 111,674 2.4
mksh 111,674 6.2
oil-native 118,863 8.0

(zsh isn't measured because zsh -n unexpectedly forks.)

Average Parsing Rate, Measured on Two Machines (lines/ms)

Shell startup time is included in the elapsed time measurements, but long files are chosen to minimize its effect.

shell label total lines host flanders host lisa
dash 111,674 1,307.8 3,166.5
mksh 111,674 447.0 1,066.2
bash 118,863 249.2 659.9
oil-native 118,863 246.0 656.4
zsh 118,863 31.6 70.7
osh-cpython 118,863 3.3 10.7
osh-ovm 118,863 3.0 9.5

Breakdown By File

Instructions Per Line (in thousands)

bash dash mksh oil-native num lines filename
15.4 1.9 3.8 4.5 1,679 t4014-format-patch.sh
17.2 2.8 8.9 12.4 1,733 functions
23.7 NA NA 11.2 2,029 configure-helper.sh
18.5 2.9 9.4 12.9 2,512 abuild
13.9 2.5 6.4 9.1 2,698 Build.sh
13.5 1.4 2.7 3.0 3,123 t9300-fast-import.sh
32.2 NA NA 13.8 5,160 test-cmd-util.sh
18.1 2.6 6.7 8.5 9,661 ltmain.sh
17.4 2.8 7.0 8.7 20,489 configure
13.1 2.3 6.0 7.3 69,779 configure-coreutils

Elasped Time (milliseconds)

host label bash dash mksh zsh osh-ovm osh-cpython oil-native osh to bash ratio num lines filename
flanders 14 3 6 164 295 211 7 0.5 1,679 t4014-format-patch.sh
flanders 15 4 9 45 909 661 18 1.2 1,733 functions
flanders 15 NA NA 83 967 735 18 1.3 2,029 configure-helper.sh
flanders 19 5 17 65 1,389 1,101 24 1.2 2,512 abuild
flanders 17 5 9 152 1,051 788 19 1.1 2,698 Build.sh
flanders 14 2 5 202 339 262 9 0.7 3,123 t9300-fast-import.sh
flanders 42 NA NA 209 3,334 2,224 38 0.9 5,160 test-cmd-util.sh
flanders 47 10 30 245 3,427 2,398 40 0.9 9,661 ltmain.sh
flanders 86 22 48 585 7,850 5,995 86 1.0 20,489 configure
flanders 207 35 126 2,009 19,813 21,914 224 1.1 69,779 configure-coreutils
lisa 3 1 1 58 100 93 2 0.8 1,679 t4014-format-patch.sh
lisa 3 1 3 20 278 256 5 1.4 1,733 functions
lisa 5 NA NA 38 293 268 5 0.9 2,029 configure-helper.sh
lisa 5 1 4 29 419 377 6 1.3 2,512 abuild
lisa 5 1 3 56 304 279 6 1.2 2,698 Build.sh
lisa 4 1 2 68 97 94 2 0.6 3,123 t9300-fast-import.sh
lisa 16 NA NA 105 913 799 14 0.9 5,160 test-cmd-util.sh
lisa 21 4 10 121 1,092 932 16 0.8 9,661 ltmain.sh
lisa 33 7 23 289 2,290 2,039 35 1.0 20,489 configure
lisa 84 19 59 896 6,738 5,971 90 1.1 69,779 configure-coreutils

Parsing Rate (lines/ms)

host label bash dash mksh zsh osh-ovm osh-cpython oil-native num lines filename
flanders 118.9 657.7 291.4 10.2 5.7 8.0 232.3 1,679 t4014-format-patch.sh
flanders 117.1 440.6 200.8 38.9 1.9 2.6 96.1 1,733 functions
flanders 138.2 NA NA 24.6 2.1 2.8 109.8 2,029 configure-helper.sh
flanders 129.6 458.0 145.4 38.8 1.8 2.3 104.3 2,512 abuild
flanders 155.1 536.3 299.6 17.7 2.6 3.4 145.7 2,698 Build.sh
flanders 225.7 1,311.6 666.7 15.5 9.2 11.9 336.8 3,123 t9300-fast-import.sh
flanders 122.7 NA NA 24.7 1.5 2.3 137.3 5,160 test-cmd-util.sh
flanders 205.2 994.0 321.6 39.5 2.8 4.0 240.0 9,661 ltmain.sh
flanders 237.2 952.9 423.6 35.0 2.6 3.4 238.0 20,489 configure
flanders 336.6 2,006.0 553.4 34.7 3.5 3.2 312.0 69,779 configure-coreutils
lisa 586.2 1,888.6 1,308.7 28.9 16.8 18.1 779.5 1,679 t4014-format-patch.sh
lisa 503.5 1,684.2 666.5 87.3 6.2 6.8 355.7 1,733 functions
lisa 387.1 NA NA 53.7 6.9 7.6 430.1 2,029 configure-helper.sh
lisa 500.1 1,826.9 632.3 86.8 6.0 6.7 395.5 2,512 abuild
lisa 533.1 2,121.1 965.3 47.8 8.9 9.7 444.3 2,698 Build.sh
lisa 769.6 3,860.3 2,065.5 45.6 32.2 33.3 1,285.7 3,123 t9300-fast-import.sh
lisa 315.2 NA NA 49.0 5.6 6.5 358.8 5,160 test-cmd-util.sh
lisa 465.0 2,694.1 924.3 79.7 8.8 10.4 610.7 9,661 ltmain.sh
lisa 613.6 2,824.5 884.7 70.8 8.9 10.0 590.2 20,489 configure
lisa 831.6 3,662.4 1,183.4 77.9 10.4 11.7 779.0 69,779 configure-coreutils

Memory Usage (Max Resident Set Size in MB)

Again, Oil uses a different algorithm (and language) than POSIX shells. It builds an AST in memory rather than just validating the code line-by-line.

host label bash dash mksh zsh osh-ovm osh-cpython oil-native num lines filename
flanders 3.1 1.7 1.8 3.7 13.4 16.6 5.3 1,679 t4014-format-patch.sh
flanders 3.1 1.7 1.8 3.7 16.3 19.5 7.3 1,733 functions
flanders 3.2 NA NA 3.8 16.5 20.2 7.7 2,029 configure-helper.sh
flanders 3.1 1.7 1.7 3.7 18.4 21.7 9.2 2,512 abuild
flanders 3.1 1.8 1.8 3.7 17.1 20.5 8.0 2,698 Build.sh
flanders 3.0 1.7 1.6 3.7 14.0 17.2 5.5 3,123 t9300-fast-import.sh
flanders 3.3 NA NA 3.7 27.0 31.0 16.5 5,160 test-cmd-util.sh
flanders 4.2 2.2 2.8 4.2 28.6 32.8 18.6 9,661 ltmain.sh
flanders 3.9 2.2 2.4 4.0 48.9 54.3 36.0 20,489 configure
flanders 3.4 1.8 1.8 3.7 111.2 121.0 97.1 69,779 configure-coreutils
lisa 3.2 1.5 1.8 3.7 13.4 16.6 5.2 1,679 t4014-format-patch.sh
lisa 3.2 1.5 1.8 3.6 16.2 19.6 7.3 1,733 functions
lisa 3.3 NA NA 3.6 16.9 19.9 7.7 2,029 configure-helper.sh
lisa 3.2 1.7 1.9 3.7 18.2 21.7 9.2 2,512 abuild
lisa 3.2 1.5 1.7 3.7 17.0 20.4 8.1 2,698 Build.sh
lisa 3.2 1.6 1.7 3.7 13.8 17.3 5.5 3,123 t9300-fast-import.sh
lisa 3.4 NA NA 3.6 27.0 31.2 16.6 5,160 test-cmd-util.sh
lisa 4.3 2.1 2.9 4.2 28.5 32.9 18.6 9,661 ltmain.sh
lisa 4.0 2.2 2.4 4.1 48.8 54.3 35.9 20,489 configure
lisa 3.5 1.7 1.9 3.7 111.1 121.0 97.0 69,779 configure-coreutils

Shell and Host Details

shell label shell id
bash bash-d685a43c
dash dash-2887d9e6
mksh mksh-ae7d165b
oil-native osh_eval.opt.stripped-c6b7a640
host label host id
flanders flanders-63f04b74
lisa lisa-63f04b74

Raw Data

filename
no-host.2021-04-23__14-21-32.cachegrind.tsv
flanders.2021-04-23__14-16-13.times.csv
lisa.2021-04-23__14-22-12.times.csv