OSH Parser Performance

We time $sh -n $file for various files under various shells, and repeat then run under cachegrind for stable metrics.

Source code: oil/benchmarks/osh-parser.sh

Summary

Instructions Per Line (via cachegrind)

Lower numbers are generally better, but each shell recognizes a different language, and Oil uses a more thorough parsing algorithm. In thousands of "I refs".

shell label total lines thousand irefs per line
bash 118,863 15.3
dash 111,674 1.6
mksh 111,674 5.7
osh-native 118,863 16.3

(zsh isn't measured because zsh -n unexpectedly forks.)

Average Parsing Rate, Measured on Two Machines (lines/ms)

Shell startup time is included in the elapsed time measurements, but long files are chosen to minimize its effect.

shell label total lines host hoover host lenny
dash 111,674 3,683 3,384
mksh 111,674 1,559 1,195
bash 118,863 1,604 566
osh-native 118,863 439 344
zsh 118,863 318 286
osh-cpython 118,863 12 8
osh-ovm 118,863 12 7

Per-File Measurements

Instructions Per Line (in thousands)

bash dash mksh osh-native num lines filename
15.6 1.3 4.4 8.4 1,679 t4014-format-patch.sh
16.7 2.1 8.4 25.6 1,733 functions
23.7 NA NA 20.5 2,029 configure-helper.sh
17.9 2.1 8.9 24.0 2,512 abuild
13.6 1.8 5.9 18.3 2,698 Build.sh
13.7 1.0 3.3 5.3 3,123 t9300-fast-import.sh
32.0 NA NA 30.1 5,160 test-cmd-util.sh
18.2 1.8 6.1 17.2 9,661 ltmain.sh
17.3 1.9 6.4 18.2 20,489 configure
12.9 1.5 5.4 14.6 69,779 configure-coreutils

Elapsed Time (milliseconds)

host label bash dash mksh zsh osh-ovm osh-cpython osh-native osh to bash ratio num lines filename
hoover 2 1 1 27 71 75 3 1.8 1,679 t4014-format-patch.sh
hoover 2 1 2 3 236 228 7 3.3 1,733 functions
hoover 3 NA NA 6 248 233 6 2.5 2,029 configure-helper.sh
hoover 3 1 3 4 351 337 8 3.1 2,512 abuild
hoover 2 1 2 23 255 252 7 3.1 2,698 Build.sh
hoover 2 1 2 32 79 81 3 1.6 3,123 t9300-fast-import.sh
hoover 6 NA NA 8 769 708 21 3.5 5,160 test-cmd-util.sh
hoover 7 3 8 29 836 824 22 3.1 9,661 ltmain.sh
hoover 14 7 15 52 1,863 1,793 49 3.5 20,489 configure
hoover 34 15 37 190 5,269 5,056 144 4.3 69,779 configure-coreutils
lenny 6 1 1 27 98 97 3 0.6 1,679 t4014-format-patch.sh
lenny 5 1 2 3 365 327 7 1.5 1,733 functions
lenny 7 NA NA 6 370 326 7 1.0 2,029 configure-helper.sh
lenny 7 1 3 5 543 494 9 1.4 2,512 abuild
lenny 5 1 3 25 409 355 9 1.8 2,698 Build.sh
lenny 5 1 2 74 117 118 4 1.0 3,123 t9300-fast-import.sh
lenny 16 NA NA 17 1,367 1,018 32 2.0 5,160 test-cmd-util.sh
lenny 22 4 9 13 1,408 1,209 26 1.2 9,661 ltmain.sh
lenny 40 8 18 47 3,097 2,643 64 1.6 20,489 configure
lenny 99 16 54 199 8,872 7,447 183 1.8 69,779 configure-coreutils

Parsing Rate (lines/ms)

host label bash dash mksh zsh osh-ovm osh-cpython osh-native num lines filename
hoover 1,028 2,025 1,189 61 24 22 586 1,679 t4014-format-patch.sh
hoover 828 1,752 744 609 7 8 248 1,733 functions
hoover 811 NA NA 338 8 9 322 2,029 configure-helper.sh
hoover 919 1,748 819 635 7 7 297 2,512 abuild
hoover 1,151 2,192 1,092 119 11 11 376 2,698 Build.sh
hoover 1,674 3,727 1,757 98 40 38 1,030 3,123 t9300-fast-import.sh
hoover 870 NA NA 612 7 7 249 5,160 test-cmd-util.sh
hoover 1,334 3,114 1,160 335 12 12 434 9,661 ltmain.sh
hoover 1,463 3,145 1,338 395 11 11 416 20,489 configure
hoover 2,067 4,538 1,889 367 13 14 485 69,779 configure-coreutils
lenny 302 2,717 1,314 62 17 17 495 1,679 t4014-format-patch.sh
lenny 372 1,972 767 576 5 5 243 1,733 functions
lenny 297 NA NA 322 5 6 284 2,029 configure-helper.sh
lenny 368 2,097 725 484 5 5 267 2,512 abuild
lenny 545 2,669 956 107 7 8 301 2,698 Build.sh
lenny 681 3,308 1,832 42 27 26 703 3,123 t9300-fast-import.sh
lenny 318 NA NA 301 4 5 162 5,160 test-cmd-util.sh
lenny 445 2,386 1,035 737 7 8 375 9,661 ltmain.sh
lenny 518 2,493 1,114 440 7 8 320 20,489 configure
lenny 703 4,338 1,287 351 8 9 381 69,779 configure-coreutils

Memory Usage (Max Resident Set Size in MB)

Again, Oil uses a different algorithm (and language) than POSIX shells. It builds an AST in memory rather than just validating the code line-by-line.

host label bash dash mksh zsh osh-ovm osh-cpython osh-native num lines filename
hoover 3.4 1.7 2.3 3.7 14.9 17.5 7.8 1,679 t4014-format-patch.sh
hoover 3.5 1.7 2.5 3.8 17.8 20.7 8.9 1,733 functions
hoover 3.6 NA NA 3.8 18.3 21.2 9.4 2,029 configure-helper.sh
hoover 3.5 1.8 2.6 3.6 20.0 22.7 10.5 2,512 abuild
hoover 3.4 1.7 2.5 3.7 18.3 21.4 9.4 2,698 Build.sh
hoover 3.6 1.8 2.4 3.6 15.3 17.9 8.0 3,123 t9300-fast-import.sh
hoover 3.6 NA NA 3.9 28.3 31.8 15.8 5,160 test-cmd-util.sh
hoover 4.4 2.1 3.4 4.6 30.5 34.2 17.8 9,661 ltmain.sh
hoover 4.2 2.5 3.2 4.4 49.8 55.7 27.1 20,489 configure
hoover 3.8 1.9 2.8 3.9 113.0 124.8 69.6 69,779 configure-coreutils
lenny 3.4 1.7 1.9 3.9 14.8 18.2 7.2 1,679 t4014-format-patch.sh
lenny 3.4 1.7 1.9 3.9 17.9 21.3 8.5 1,733 functions
lenny 3.4 NA NA 4.1 18.3 21.9 8.9 2,029 configure-helper.sh
lenny 3.4 1.7 2.0 4.1 19.9 23.1 10.0 2,512 abuild
lenny 3.5 1.7 1.8 3.9 18.6 22.0 9.1 2,698 Build.sh
lenny 3.5 1.7 1.9 4.1 15.4 18.6 7.5 3,123 t9300-fast-import.sh
lenny 3.6 NA NA 4.1 28.5 32.4 15.3 5,160 test-cmd-util.sh
lenny 4.3 2.1 3.1 4.5 30.5 34.9 17.2 9,661 ltmain.sh
lenny 4.1 2.5 2.7 4.5 49.8 56.3 26.7 20,489 configure
lenny 3.7 1.9 2.2 4.2 113.0 125.2 69.3 69,779 configure-coreutils

Shell and Host Details

shell label shell id
bash bash-0d42a1ee
dash dash-d40a0412
mksh mksh-84e6599c
zsh zsh-25b8aadb
osh-cpython osh-e9438750
osh-ovm osh-0a14ddc2
osh-native osh-4dab5cf8
bash bash-9af8f89f
dash dash-9ff48631
mksh mksh-e3755400
zsh zsh-a785003a
host label host id
hoover hoover-94601b4f
lenny lenny-098a8e52

Raw Data

filename
lenny.2023-09-13__01-29-50.cachegrind.tsv
hoover.2023-09-13__00-25-56.times.csv
lenny.2023-09-13__01-29-50.times.csv