OSH Parser Performance

We time $sh -n $file for various files under various shells, and repeat then run under cachegrind for stable metrics.

Source code: oil/benchmarks/osh-parser.sh

Summary

Instructions Per Line (via cachegrind)

Lower numbers are generally better, but each shell recognizes a different language, and Oil uses a more thorough parsing algorithm. In thousands of "I refs".

shell label total lines thousand irefs per line
bash 118,863 15.4
dash 111,674 1.6
mksh 111,674 5.7
osh-native 118,863 18.2

(zsh isn't measured because zsh -n unexpectedly forks.)

Average Parsing Rate, Measured on Two Machines (lines/ms)

Shell startup time is included in the elapsed time measurements, but long files are chosen to minimize its effect.

shell label total lines host broome host lenny
dash 111,674 1,449 3,429
mksh 111,674 597 1,084
bash 118,863 346 526
zsh 118,863 104 273
osh-native 118,863 188 272
osh-cpython 118,863 7 8
osh-ovm 118,863 6 6

Per-File Measurements

Instructions Per Line (in thousands)

bash dash mksh osh-native num lines filename
15.7 1.4 4.4 9.5 1,679 t4014-format-patch.sh
16.7 2.1 8.4 27.8 1,733 functions
23.7 NA NA 25.5 2,029 configure-helper.sh
17.9 2.2 8.9 26.4 2,512 abuild
13.6 1.8 5.9 19.9 2,698 Build.sh
13.7 1.0 3.3 5.9 3,123 t9300-fast-import.sh
32.0 NA NA 30.7 5,160 test-cmd-util.sh
18.2 1.8 6.1 18.7 9,661 ltmain.sh
17.4 1.9 6.4 18.7 20,489 configure
12.9 1.5 5.4 17.0 69,779 configure-coreutils

Elasped Time (milliseconds)

host label bash dash mksh zsh osh-ovm osh-cpython osh-native osh to bash ratio num lines filename
broome 9 2 4 98 136 159 9 1.1 1,679 t4014-format-patch.sh
broome 10 3 7 13 483 423 17 1.7 1,733 functions
broome 15 NA NA 20 501 447 21 1.4 2,029 configure-helper.sh
broome 14 4 11 18 657 650 25 1.8 2,512 abuild
broome 13 3 7 77 515 496 19 1.5 2,698 Build.sh
broome 11 2 5 113 153 170 10 0.9 3,123 t9300-fast-import.sh
broome 30 NA NA 27 1,510 1,332 46 1.5 5,160 test-cmd-util.sh
broome 36 10 23 42 1,688 1,478 54 1.5 9,661 ltmain.sh
broome 62 17 37 119 3,545 3,108 104 1.7 20,489 configure
broome 144 37 92 617 10,355 8,893 327 2.3 69,779 configure-coreutils
lenny 5 1 2 36 118 126 4 0.8 1,679 t4014-format-patch.sh
lenny 6 1 3 4 393 333 8 1.5 1,733 functions
lenny 7 NA NA 11 434 357 11 1.7 2,029 configure-helper.sh
lenny 8 1 5 8 566 564 14 1.9 2,512 abuild
lenny 5 1 4 28 411 363 10 2.1 2,698 Build.sh
lenny 5 1 2 38 123 118 5 0.9 3,123 t9300-fast-import.sh
lenny 19 NA NA 9 1,252 1,206 33 1.7 5,160 test-cmd-util.sh
lenny 20 3 11 16 1,634 1,361 36 1.8 9,661 ltmain.sh
lenny 40 7 24 52 3,392 2,612 69 1.8 20,489 configure
lenny 113 17 52 233 10,142 8,566 246 2.2 69,779 configure-coreutils

Parsing Rate (lines/ms)

host label bash dash mksh zsh osh-ovm osh-cpython osh-native num lines filename
broome 188 877 394 17 12 11 178 1,679 t4014-format-patch.sh
broome 178 645 247 136 4 4 102 1,733 functions
broome 135 NA NA 100 4 5 99 2,029 configure-helper.sh
broome 186 670 235 136 4 4 101 2,512 abuild
broome 215 836 369 35 5 5 143 2,698 Build.sh
broome 278 1,409 570 28 20 18 307 3,123 t9300-fast-import.sh
broome 170 NA NA 189 3 4 111 5,160 test-cmd-util.sh
broome 271 1,012 429 232 6 7 180 9,661 ltmain.sh
broome 330 1,204 550 172 6 7 196 20,489 configure
broome 483 1,901 755 113 7 8 213 69,779 configure-coreutils
lenny 366 2,385 891 47 14 13 451 1,679 t4014-format-patch.sh
lenny 315 1,712 554 448 4 5 206 1,733 functions
lenny 303 NA NA 191 5 6 178 2,029 configure-helper.sh
lenny 331 1,820 494 314 4 4 173 2,512 abuild
lenny 550 2,243 710 96 7 7 268 2,698 Build.sh
lenny 599 3,586 1,702 82 25 26 666 3,123 t9300-fast-import.sh
lenny 275 NA NA 553 4 4 158 5,160 test-cmd-util.sh
lenny 482 2,904 897 600 6 7 265 9,661 ltmain.sh
lenny 516 3,085 850 393 6 8 295 20,489 configure
lenny 618 4,004 1,330 299 7 8 284 69,779 configure-coreutils

Memory Usage (Max Resident Set Size in MB)

Again, Oil uses a different algorithm (and language) than POSIX shells. It builds an AST in memory rather than just validating the code line-by-line.

host label bash dash mksh zsh osh-ovm osh-cpython osh-native num lines filename
broome 3.4 1.7 1.8 4.0 14.3 17.7 7.2 1,679 t4014-format-patch.sh
broome 3.4 1.7 1.8 4.1 16.4 20.1 8.2 1,733 functions
broome 3.5 NA NA 4.0 16.9 20.4 8.6 2,029 configure-helper.sh
broome 3.5 1.8 1.9 4.0 18.0 21.6 9.4 2,512 abuild
broome 3.5 1.7 1.9 3.9 17.0 20.7 8.5 2,698 Build.sh
broome 3.5 1.7 1.9 4.1 14.6 18.3 7.2 3,123 t9300-fast-import.sh
broome 3.6 NA NA 4.0 24.3 28.4 13.9 5,160 test-cmd-util.sh
broome 4.5 2.1 3.1 4.4 25.8 30.3 17.2 9,661 ltmain.sh
broome 4.0 2.5 2.7 4.5 41.8 47.9 26.8 20,489 configure
broome 3.7 1.8 2.1 4.2 91.9 103.6 69.0 69,779 configure-coreutils
lenny 3.4 1.7 1.9 3.9 14.2 17.9 7.2 1,679 t4014-format-patch.sh
lenny 3.3 1.8 1.9 4.1 16.3 19.9 8.3 1,733 functions
lenny 3.5 NA NA 4.1 16.7 20.5 8.7 2,029 configure-helper.sh
lenny 3.4 1.7 1.9 4.0 17.8 21.4 9.4 2,512 abuild
lenny 3.4 1.8 2.0 4.0 16.8 20.6 8.5 2,698 Build.sh
lenny 3.4 1.6 1.9 4.0 14.6 18.3 7.2 3,123 t9300-fast-import.sh
lenny 3.6 NA NA 4.0 24.2 28.5 14.0 5,160 test-cmd-util.sh
lenny 4.4 2.2 3.0 4.6 26.0 30.4 17.1 9,661 ltmain.sh
lenny 4.1 2.4 2.6 4.5 41.8 47.8 26.8 20,489 configure
lenny 3.7 1.8 2.1 4.1 92.0 103.5 69.1 69,779 configure-coreutils

Shell and Host Details

shell label shell id
bash bash-9af8f89f
dash dash-9ff48631
mksh mksh-e3755400
zsh zsh-a785003a
osh-cpython osh-66742cb0
osh-ovm osh-74ef0703
osh-native osh-f03c4b44
host label host id
broome broome-76a68468
lenny lenny-098a8e52

Raw Data

filename
lenny.2023-06-05__13-34-59.cachegrind.tsv
broome.2023-06-05__17-15-40.times.csv
lenny.2023-06-05__13-34-59.times.csv