OSH Parser Performance

We time $sh -n $file for various files under various shells, and repeat then run under cachegrind for stable metrics.

Source code: oil/benchmarks/osh-parser.sh

Summary

Instructions Per Line (via cachegrind)

Lower numbers are generally better, but each shell recognizes a different language, and Oil uses a more thorough parsing algorithm. In thousands of "I refs".

shell label total lines thousand irefs per line
bash 118,863 15.3
dash 111,674 1.6
mksh 111,674 5.7
osh-native 118,863 18.2

(zsh isn't measured because zsh -n unexpectedly forks.)

Average Parsing Rate, Measured on Two Machines (lines/ms)

Shell startup time is included in the elapsed time measurements, but long files are chosen to minimize its effect.

shell label total lines host hoover host lenny
dash 111,674 3,464 3,844
mksh 111,674 1,469 1,061
bash 118,863 1,493 573
zsh 118,863 305 324
osh-native 118,863 334 315
osh-cpython 118,863 13 9
osh-ovm 118,863 12 8

Per-File Measurements

Instructions Per Line (in thousands)

bash dash mksh osh-native num lines filename
15.6 1.3 4.4 9.3 1,679 t4014-format-patch.sh
16.7 2.1 8.4 27.7 1,733 functions
23.7 NA NA 25.4 2,029 configure-helper.sh
17.9 2.1 8.9 26.3 2,512 abuild
13.6 1.8 5.9 19.8 2,698 Build.sh
13.7 1.0 3.3 5.8 3,123 t9300-fast-import.sh
32.0 NA NA 30.7 5,160 test-cmd-util.sh
18.2 1.8 6.1 18.6 9,661 ltmain.sh
17.3 1.9 6.4 18.7 20,489 configure
12.9 1.5 5.4 17.0 69,779 configure-coreutils

Elasped Time (milliseconds)

host label bash dash mksh zsh osh-ovm osh-cpython osh-native osh to bash ratio num lines filename
hoover 2 1 1 35 77 76 4 2.2 1,679 t4014-format-patch.sh
hoover 2 1 2 3 218 233 8 3.5 1,733 functions
hoover 3 NA NA 7 240 262 8 2.6 2,029 configure-helper.sh
hoover 3 1 3 4 351 348 10 3.3 2,512 abuild
hoover 3 1 2 24 241 253 10 3.5 2,698 Build.sh
hoover 2 1 2 32 77 84 5 2.4 3,123 t9300-fast-import.sh
hoover 6 NA NA 10 741 691 24 4.1 5,160 test-cmd-util.sh
hoover 7 3 7 30 844 800 29 4.1 9,661 ltmain.sh
hoover 13 7 16 52 1,744 1,720 65 4.8 20,489 configure
hoover 39 16 41 193 5,233 4,945 194 5.0 69,779 configure-coreutils
lenny 6 1 1 28 99 93 3 0.6 1,679 t4014-format-patch.sh
lenny 7 1 2 3 358 305 7 1.1 1,733 functions
lenny 7 NA NA 6 360 330 8 1.2 2,029 configure-helper.sh
lenny 6 1 3 5 546 528 11 1.8 2,512 abuild
lenny 5 1 3 25 396 382 9 1.8 2,698 Build.sh
lenny 5 1 2 32 112 108 4 0.8 3,123 t9300-fast-import.sh
lenny 17 NA NA 9 1,203 1,046 24 1.4 5,160 test-cmd-util.sh
lenny 20 3 12 14 1,275 1,280 31 1.6 9,661 ltmain.sh
lenny 35 6 24 43 2,931 2,487 62 1.8 20,489 configure
lenny 101 16 57 203 8,533 7,294 218 2.2 69,779 configure-coreutils

Parsing Rate (lines/ms)

host label bash dash mksh zsh osh-ovm osh-cpython osh-native num lines filename
hoover 1,026 2,147 1,127 48 22 22 465 1,679 t4014-format-patch.sh
hoover 788 1,334 813 597 8 7 222 1,733 functions
hoover 630 NA NA 304 8 8 241 2,029 configure-helper.sh
hoover 870 1,712 735 587 7 7 264 2,512 abuild
hoover 981 2,123 1,188 112 11 11 280 2,698 Build.sh
hoover 1,525 3,749 1,859 98 41 37 634 3,123 t9300-fast-import.sh
hoover 890 NA NA 517 7 7 215 5,160 test-cmd-util.sh
hoover 1,362 3,083 1,294 325 11 12 329 9,661 ltmain.sh
hoover 1,525 2,930 1,276 397 12 12 316 20,489 configure
hoover 1,810 4,240 1,682 362 13 14 360 69,779 configure-coreutils
lenny 303 2,652 1,226 60 17 18 484 1,679 t4014-format-patch.sh
lenny 264 1,949 767 566 5 6 231 1,733 functions
lenny 296 NA NA 348 6 6 250 2,029 configure-helper.sh
lenny 423 2,003 723 557 5 5 235 2,512 abuild
lenny 547 2,273 972 108 7 7 311 2,698 Build.sh
lenny 673 4,051 1,545 98 28 29 873 3,123 t9300-fast-import.sh
lenny 302 NA NA 587 4 5 214 5,160 test-cmd-util.sh
lenny 491 3,203 777 704 8 8 316 9,661 ltmain.sh
lenny 587 3,534 840 476 7 8 331 20,489 configure
lenny 689 4,501 1,235 343 8 10 319 69,779 configure-coreutils

Memory Usage (Max Resident Set Size in MB)

Again, Oil uses a different algorithm (and language) than POSIX shells. It builds an AST in memory rather than just validating the code line-by-line.

host label bash dash mksh zsh osh-ovm osh-cpython osh-native num lines filename
hoover 3.4 1.8 2.3 3.8 14.6 17.3 7.6 1,679 t4014-format-patch.sh
hoover 3.5 1.7 2.3 3.9 16.6 19.3 8.6 1,733 functions
hoover 3.5 NA NA 3.8 16.9 19.9 9.0 2,029 configure-helper.sh
hoover 3.4 1.8 2.5 3.9 17.9 21.1 9.7 2,512 abuild
hoover 3.5 1.7 2.6 3.7 17.3 20.3 8.9 2,698 Build.sh
hoover 3.6 1.6 2.5 3.9 15.0 17.7 7.6 3,123 t9300-fast-import.sh
hoover 3.7 NA NA 4.0 24.5 27.9 14.3 5,160 test-cmd-util.sh
hoover 4.5 2.2 3.5 4.5 26.3 30.0 17.6 9,661 ltmain.sh
hoover 4.0 2.5 3.2 4.4 42.1 47.2 27.1 20,489 configure
hoover 3.8 1.9 2.8 4.0 92.0 103.2 70.6 69,779 configure-coreutils
lenny 3.4 1.7 1.9 4.0 14.3 17.8 7.3 1,679 t4014-format-patch.sh
lenny 3.4 1.7 1.9 4.1 16.8 20.2 8.3 1,733 functions
lenny 3.4 NA NA 4.1 17.2 20.6 8.8 2,029 configure-helper.sh
lenny 3.4 1.8 2.0 4.1 17.9 21.7 9.4 2,512 abuild
lenny 3.3 1.7 2.0 4.2 17.3 20.7 8.6 2,698 Build.sh
lenny 3.4 1.6 1.8 3.9 15.0 18.4 7.2 3,123 t9300-fast-import.sh
lenny 3.6 NA NA 4.0 24.6 28.6 14.0 5,160 test-cmd-util.sh
lenny 4.4 2.1 3.0 4.6 26.0 30.6 17.3 9,661 ltmain.sh
lenny 4.0 2.5 2.7 4.4 41.7 47.8 27.0 20,489 configure
lenny 3.8 1.8 2.3 4.2 92.1 103.8 69.2 69,779 configure-coreutils

Shell and Host Details

shell label shell id
bash bash-0d42a1ee
dash dash-d40a0412
mksh mksh-84e6599c
zsh zsh-25b8aadb
osh-cpython osh-c58c917a
osh-ovm osh-3ed2a1b7
osh-native osh-37d2a50c
bash bash-9af8f89f
dash dash-9ff48631
mksh mksh-e3755400
zsh zsh-a785003a
host label host id
hoover hoover-94601b4f
lenny lenny-098a8e52

Raw Data

filename
lenny.2023-07-26__13-48-15.cachegrind.tsv
hoover.2023-07-26__13-26-45.times.csv
lenny.2023-07-26__13-48-15.times.csv