OSH Parser Performance

We time $sh -n $file for various files under various shells, and repeat then run under cachegrind for stable metrics.

Summary

Instructions Per Line (via cachegrind)

Lower numbers are generally better, but each shell recognizes a different language, and Oil uses a more thorough parsing algorithm. In thousands of "I refs".

shell label total lines thousand irefs per line
bash 118,863 15.5
dash 111,674 2.4
mksh 111,674 6.2
oil-native 118,863 8.0

(zsh isn't measured because zsh -n unexpectedly forks.)

Average Parsing Rate, Measured on Two Machines (lines/ms)

Shell startup time is included in the elapsed time measurements, but long files are chosen to minimize its effect.

shell label total lines host flanders host lisa
dash 111,674 1,289.8 2,907.2
mksh 111,674 450.7 1,034.7
bash 118,863 258.5 600.4
oil-native 118,863 212.2 575.2
zsh 118,863 31.4 64.7
osh-cpython 118,863 3.6 10.1
osh-ovm 118,863 2.8 8.0

Breakdown By File

Instructions Per Line (in thousands)

bash dash mksh oil-native num lines filename
15.4 1.9 3.8 4.5 1,679 t4014-format-patch.sh
17.2 2.8 8.9 12.4 1,733 functions
23.7 NA NA 11.2 2,029 configure-helper.sh
18.5 2.9 9.4 12.9 2,512 abuild
13.9 2.5 6.4 9.1 2,698 Build.sh
13.5 1.4 2.7 3.0 3,123 t9300-fast-import.sh
32.2 NA NA 13.8 5,160 test-cmd-util.sh
18.1 2.6 6.7 8.5 9,661 ltmain.sh
17.4 2.8 7.0 8.7 20,489 configure
13.1 2.3 6.0 7.3 69,779 configure-coreutils

Elasped Time (milliseconds)

host label bash dash mksh zsh osh-ovm osh-cpython oil-native osh to bash ratio num lines filename
flanders 11 2 4 173 258 231 7 0.6 1,679 t4014-format-patch.sh
flanders 13 4 12 44 1,078 724 15 1.1 1,733 functions
flanders 17 NA NA 80 1,219 763 16 0.9 2,029 configure-helper.sh
flanders 19 4 11 65 1,767 1,208 25 1.3 2,512 abuild
flanders 10 3 13 146 1,096 999 19 2.0 2,698 Build.sh
flanders 14 2 5 204 333 325 9 0.6 3,123 t9300-fast-import.sh
flanders 42 NA NA 214 3,432 2,828 45 1.1 5,160 test-cmd-util.sh
flanders 44 14 26 234 4,152 2,993 48 1.1 9,661 ltmain.sh
flanders 85 19 49 577 8,717 5,990 98 1.2 20,489 configure
flanders 205 38 128 2,052 19,919 17,339 278 1.4 69,779 configure-coreutils
lisa 3 1 1 62 97 102 2 0.7 1,679 t4014-format-patch.sh
lisa 4 1 3 21 312 282 5 1.4 1,733 functions
lisa 8 NA NA 42 329 282 6 0.7 2,029 configure-helper.sh
lisa 6 1 4 33 434 394 8 1.4 2,512 abuild
lisa 4 1 3 58 314 293 6 1.4 2,698 Build.sh
lisa 5 1 2 73 99 98 3 0.6 3,123 t9300-fast-import.sh
lisa 15 NA NA 114 938 831 16 1.0 5,160 test-cmd-util.sh
lisa 24 4 11 140 1,076 975 18 0.7 9,661 ltmain.sh
lisa 35 9 24 300 2,855 2,135 41 1.2 20,489 configure
lisa 94 20 60 995 8,327 6,356 103 1.1 69,779 configure-coreutils

Parsing Rate (lines/ms)

host label bash dash mksh zsh osh-ovm osh-cpython oil-native num lines filename
flanders 150.2 820.6 404.6 9.7 6.5 7.3 237.9 1,679 t4014-format-patch.sh
flanders 130.4 416.5 144.9 39.4 1.6 2.4 115.1 1,733 functions
flanders 118.2 NA NA 25.5 1.7 2.7 127.9 2,029 configure-helper.sh
flanders 134.0 568.2 231.0 38.4 1.4 2.1 99.4 2,512 abuild
flanders 278.3 855.7 209.4 18.5 2.5 2.7 141.2 2,698 Build.sh
flanders 219.1 1,409.9 666.9 15.3 9.4 9.6 341.3 3,123 t9300-fast-import.sh
flanders 122.4 NA NA 24.1 1.5 1.8 113.7 5,160 test-cmd-util.sh
flanders 221.9 710.8 377.7 41.2 2.3 3.2 203.3 9,661 ltmain.sh
flanders 242.3 1,062.8 415.0 35.5 2.4 3.4 209.7 20,489 configure
flanders 339.9 1,850.1 544.0 34.0 3.5 4.0 251.1 69,779 configure-coreutils
lisa 531.8 1,970.7 1,234.6 27.2 17.3 16.5 734.8 1,679 t4014-format-patch.sh
lisa 464.7 1,449.0 580.6 82.3 5.6 6.2 343.5 1,733 functions
lisa 254.4 NA NA 48.3 6.2 7.2 362.4 2,029 configure-helper.sh
lisa 425.8 1,681.4 622.6 76.6 5.8 6.4 313.1 2,512 abuild
lisa 618.4 1,998.5 799.4 46.9 8.6 9.2 432.6 2,698 Build.sh
lisa 692.6 3,695.9 1,984.1 43.0 31.6 31.8 1,225.2 3,123 t9300-fast-import.sh
lisa 333.5 NA NA 45.4 5.5 6.2 320.9 5,160 test-cmd-util.sh
lisa 405.9 2,651.2 859.7 69.2 9.0 9.9 547.0 9,661 ltmain.sh
lisa 584.5 2,261.2 861.5 68.2 7.2 9.6 504.7 20,489 configure
lisa 742.2 3,494.0 1,171.1 70.1 8.4 11.0 680.3 69,779 configure-coreutils

Memory Usage (Max Resident Set Size in MB)

Again, Oil uses a different algorithm (and language) than POSIX shells. It builds an AST in memory rather than just validating the code line-by-line.

host label bash dash mksh zsh osh-ovm osh-cpython oil-native num lines filename
flanders 3.0 1.7 1.6 3.7 13.1 16.2 5.1 1,679 t4014-format-patch.sh
flanders 3.2 1.7 1.8 3.7 16.0 19.1 7.4 1,733 functions
flanders 3.1 NA NA 3.7 16.3 19.6 7.8 2,029 configure-helper.sh
flanders 3.2 1.8 1.8 3.7 18.0 21.3 9.2 2,512 abuild
flanders 3.2 1.7 1.8 3.8 16.8 20.2 8.2 2,698 Build.sh
flanders 3.2 1.6 1.7 3.7 13.6 16.8 5.6 3,123 t9300-fast-import.sh
flanders 3.2 NA NA 3.7 26.6 30.6 16.5 5,160 test-cmd-util.sh
flanders 4.2 2.2 2.8 4.2 28.5 32.5 18.6 9,661 ltmain.sh
flanders 3.9 2.3 2.3 4.0 48.3 53.8 36.1 20,489 configure
flanders 3.4 1.7 1.9 3.8 111.0 120.4 97.1 69,779 configure-coreutils
lisa 3.1 1.7 1.7 3.7 13.2 16.2 5.2 1,679 t4014-format-patch.sh
lisa 3.2 1.6 1.8 3.7 15.9 19.1 7.3 1,733 functions
lisa 3.3 NA NA 3.6 16.3 19.6 7.7 2,029 configure-helper.sh
lisa 3.2 1.6 1.8 3.7 17.7 21.2 9.1 2,512 abuild
lisa 3.3 1.5 1.9 3.7 16.7 20.0 8.0 2,698 Build.sh
lisa 3.2 1.6 1.7 3.7 13.4 16.7 5.6 3,123 t9300-fast-import.sh
lisa 3.4 NA NA 3.8 26.7 30.6 16.5 5,160 test-cmd-util.sh
lisa 4.3 2.0 3.0 4.1 28.5 32.5 18.6 9,661 ltmain.sh
lisa 4.0 2.1 2.4 4.0 48.4 53.6 36.0 20,489 configure
lisa 3.6 1.6 1.9 3.7 110.8 120.7 97.0 69,779 configure-coreutils

Shell and Host Details

shell label shell id
bash bash-d685a43c
dash dash-2887d9e6
mksh mksh-ae7d165b
oil-native osh_eval.opt.stripped-159be6f4
host label host id
flanders flanders-c6f24b89
lisa lisa-c6f24b89

Raw Data

filename
no-host.2021-04-12__15-19-26.cachegrind.tsv
flanders.2021-04-12__15-02-38.times.csv
lisa.2021-04-12__15-20-08.times.csv