OSH Parser Performance

We time $sh -n $file for various files under various shells, and repeat then run under cachegrind for stable metrics.

Source code: oil/benchmarks/osh-parser.sh

Summary

Instructions Per Line (via cachegrind)

Lower numbers are generally better, but each shell recognizes a different language, and Oil uses a more thorough parsing algorithm. In thousands of "I refs".

shell label total lines thousand irefs per line
bash 118,863 15.4
dash 111,674 1.6
mksh 111,674 5.7
osh-native 118,863 28.9

(zsh isn't measured because zsh -n unexpectedly forks.)

Average Parsing Rate, Measured on Two Machines (lines/ms)

Shell startup time is included in the elapsed time measurements, but long files are chosen to minimize its effect.

shell label total lines host broome host lenny
dash 111,674 1,543 2,778
mksh 111,674 583 1,095
bash 118,863 351 581
zsh 118,863 102 281
osh-native 118,863 140 202
osh-cpython 118,863 7 9
osh-ovm 118,863 7 8

Per-File Measurements

Instructions Per Line (in thousands)

bash dash mksh osh-native num lines filename
15.7 1.4 4.4 10.0 1,679 t4014-format-patch.sh
16.7 2.1 8.4 34.7 1,733 functions
23.7 NA NA 35.8 2,029 configure-helper.sh
17.9 2.2 8.9 45.3 2,512 abuild
13.6 1.8 5.9 28.6 2,698 Build.sh
13.7 1.0 3.3 6.4 3,123 t9300-fast-import.sh
32.0 NA NA 44.2 5,160 test-cmd-util.sh
18.2 1.8 6.1 30.3 9,661 ltmain.sh
17.4 1.9 6.4 31.1 20,489 configure
12.9 1.5 5.4 27.5 69,779 configure-coreutils

Elasped Time (milliseconds)

host label bash dash mksh zsh osh-ovm osh-cpython osh-native osh to bash ratio num lines filename
broome 9 2 4 101 153 158 9 1.1 1,679 t4014-format-patch.sh
broome 10 3 7 11 410 404 23 2.3 1,733 functions
broome 14 NA NA 21 453 426 26 1.9 2,029 configure-helper.sh
broome 13 3 9 16 620 586 35 2.6 2,512 abuild
broome 12 3 8 80 451 433 27 2.1 2,698 Build.sh
broome 13 2 5 114 141 137 11 0.8 3,123 t9300-fast-import.sh
broome 31 NA NA 28 1,326 1,203 61 2.0 5,160 test-cmd-util.sh
broome 34 9 23 41 1,558 1,324 78 2.3 9,661 ltmain.sh
broome 59 15 39 123 3,356 2,905 162 2.7 20,489 configure
broome 143 35 96 631 9,798 8,538 414 2.9 69,779 configure-coreutils
lenny 3 1 2 34 90 100 4 1.1 1,679 t4014-format-patch.sh
lenny 4 1 3 4 342 319 9 2.4 1,733 functions
lenny 6 NA NA 8 361 337 11 2.0 2,029 configure-helper.sh
lenny 6 2 5 6 507 464 17 2.9 2,512 abuild
lenny 6 2 4 34 373 340 13 2.1 2,698 Build.sh
lenny 5 1 2 50 109 100 4 0.7 3,123 t9300-fast-import.sh
lenny 19 NA NA 12 1,166 1,001 32 1.7 5,160 test-cmd-util.sh
lenny 21 3 11 15 1,326 1,153 44 2.1 9,661 ltmain.sh
lenny 38 8 23 52 2,908 2,511 100 2.6 20,489 configure
lenny 96 22 53 209 8,512 7,428 356 3.7 69,779 configure-coreutils

Parsing Rate (lines/ms)

host label bash dash mksh zsh osh-ovm osh-cpython osh-native num lines filename
broome 193 1,049 400 17 11 11 179 1,679 t4014-format-patch.sh
broome 168 635 239 159 4 4 74 1,733 functions
broome 149 NA NA 99 4 5 79 2,029 configure-helper.sh
broome 189 718 272 154 4 4 72 2,512 abuild
broome 216 826 349 34 6 6 101 2,698 Build.sh
broome 242 1,389 592 27 22 23 289 3,123 t9300-fast-import.sh
broome 167 NA NA 182 4 4 85 5,160 test-cmd-util.sh
broome 281 1,086 429 237 6 7 123 9,661 ltmain.sh
broome 345 1,354 528 167 6 7 127 20,489 configure
broome 488 1,993 724 111 7 8 168 69,779 configure-coreutils
lenny 508 2,648 974 49 19 17 468 1,679 t4014-format-patch.sh
lenny 435 1,192 572 482 5 5 185 1,733 functions
lenny 350 NA NA 270 6 6 178 2,029 configure-helper.sh
lenny 430 1,477 503 455 5 5 148 2,512 abuild
lenny 439 1,569 744 79 7 8 213 2,698 Build.sh
lenny 578 3,776 1,556 63 29 31 825 3,123 t9300-fast-import.sh
lenny 269 NA NA 422 4 5 162 5,160 test-cmd-util.sh
lenny 464 2,853 907 637 7 8 222 9,661 ltmain.sh
lenny 537 2,558 887 395 7 8 206 20,489 configure
lenny 727 3,106 1,321 335 8 9 196 69,779 configure-coreutils

Memory Usage (Max Resident Set Size in MB)

Again, Oil uses a different algorithm (and language) than POSIX shells. It builds an AST in memory rather than just validating the code line-by-line.

host label bash dash mksh zsh osh-ovm osh-cpython osh-native num lines filename
broome 3.4 1.8 1.9 4.1 14.6 18.0 6.2 1,679 t4014-format-patch.sh
broome 3.4 1.7 2.0 4.0 17.3 21.1 9.7 1,733 functions
broome 3.4 NA NA 4.1 17.9 21.9 11.1 2,029 configure-helper.sh
broome 3.4 1.8 2.0 3.9 19.6 23.4 11.2 2,512 abuild
broome 3.3 1.7 2.0 4.0 18.1 22.0 11.2 2,698 Build.sh
broome 3.5 1.7 1.9 4.0 14.9 18.7 6.5 3,123 t9300-fast-import.sh
broome 3.5 NA NA 4.1 28.3 33.4 17.2 5,160 test-cmd-util.sh
broome 4.3 2.2 3.0 4.5 29.8 35.4 23.0 9,661 ltmain.sh
broome 3.9 2.5 2.7 4.3 50.2 58.7 37.2 20,489 configure
broome 3.7 1.8 2.3 4.2 112.8 131.0 91.4 69,779 configure-coreutils
lenny 3.4 1.7 1.9 4.0 14.5 18.0 6.2 1,679 t4014-format-patch.sh
lenny 3.5 1.8 1.9 4.1 17.2 21.1 9.6 1,733 functions
lenny 3.4 NA NA 4.0 17.9 22.0 11.2 2,029 configure-helper.sh
lenny 3.6 1.7 2.0 4.1 19.5 23.5 11.2 2,512 abuild
lenny 3.3 1.7 1.9 4.0 18.0 22.1 11.1 2,698 Build.sh
lenny 3.4 1.8 1.9 4.1 15.0 18.4 6.4 3,123 t9300-fast-import.sh
lenny 3.5 NA NA 4.1 28.3 33.5 17.3 5,160 test-cmd-util.sh
lenny 4.4 2.2 3.0 4.5 29.8 35.4 23.0 9,661 ltmain.sh
lenny 4.1 2.5 2.7 4.5 50.2 58.6 37.2 20,489 configure
lenny 3.8 1.8 2.3 4.2 112.8 130.5 91.3 69,779 configure-coreutils

Shell and Host Details

shell label shell id
bash bash-9af8f89f
dash dash-9ff48631
mksh mksh-e3755400
zsh zsh-a785003a
osh-cpython osh-5edf5d16
osh-ovm osh-ecd4f2dc
osh-native osh-f7cd6d82
host label host id
broome broome-7a69aabc
lenny lenny-76afa3e0

Raw Data

filename
lenny.2023-01-21__22-57-21.cachegrind.tsv
broome.2023-01-22__04-11-06.times.csv
lenny.2023-01-21__22-57-21.times.csv