OSH Parser Performance

We time $sh -n $file for various files under various shells, and repeat then run under cachegrind for stable metrics.

Source code: oil/benchmarks/osh-parser.sh

Summary

Instructions Per Line (via cachegrind)

Lower numbers are generally better, but each shell recognizes a different language, and Oil uses a more thorough parsing algorithm. In thousands of "I refs".

shell label total lines thousand irefs per line
bash 118,863 15.3
dash 111,674 1.6
mksh 111,674 5.7
osh-native 118,863 15.5

(zsh isn't measured because zsh -n unexpectedly forks.)

Average Parsing Rate, Measured on Two Machines (lines/ms)

Shell startup time is included in the elapsed time measurements, but long files are chosen to minimize its effect.

shell label total lines host hoover host lenny
dash 111,674 3,578 3,363
mksh 111,674 1,602 874
bash 118,863 1,534 564
osh-native 118,863 476 343
zsh 118,863 306 309
osh-cpython 118,863 13 8
osh-ovm 118,863 12 7

Per-File Measurements

Instructions Per Line (in thousands)

bash dash mksh osh-native num lines filename
15.6 1.3 4.4 7.4 1,679 t4014-format-patch.sh
16.7 2.1 8.4 23.9 1,733 functions
23.7 NA NA 18.8 2,029 configure-helper.sh
17.9 2.1 8.9 22.3 2,512 abuild
13.6 1.8 5.9 17.0 2,698 Build.sh
13.7 1.0 3.3 4.7 3,123 t9300-fast-import.sh
32.0 NA NA 28.3 5,160 test-cmd-util.sh
18.2 1.8 6.1 16.4 9,661 ltmain.sh
17.3 1.9 6.4 17.4 20,489 configure
12.9 1.5 5.4 14.0 69,779 configure-coreutils

Elapsed Time (milliseconds)

host label bash dash mksh zsh osh-ovm osh-cpython osh-native osh to bash ratio num lines filename
hoover 2 1 2 32 75 79 3 1.9 1,679 t4014-format-patch.sh
hoover 2 1 2 3 225 219 6 2.8 1,733 functions
hoover 3 NA NA 7 239 232 6 1.9 2,029 configure-helper.sh
hoover 3 1 3 4 340 322 8 2.6 2,512 abuild
hoover 2 1 2 25 244 240 7 3.0 2,698 Build.sh
hoover 2 1 2 31 81 84 3 1.7 3,123 t9300-fast-import.sh
hoover 7 NA NA 9 739 700 17 2.6 5,160 test-cmd-util.sh
hoover 8 3 8 31 834 778 23 2.9 9,661 ltmain.sh
hoover 14 6 16 54 1,815 1,700 47 3.3 20,489 configure
hoover 34 16 36 193 5,179 4,837 129 3.8 69,779 configure-coreutils
lenny 6 1 2 29 100 99 3 0.6 1,679 t4014-format-patch.sh
lenny 7 1 3 3 375 324 7 1.1 1,733 functions
lenny 11 NA NA 6 394 332 7 0.6 2,029 configure-helper.sh
lenny 10 1 5 5 633 493 9 1.0 2,512 abuild
lenny 6 1 4 24 393 345 11 1.8 2,698 Build.sh
lenny 4 1 2 32 130 116 4 1.0 3,123 t9300-fast-import.sh
lenny 15 NA NA 9 1,260 1,097 29 2.0 5,160 test-cmd-util.sh
lenny 17 3 13 13 1,574 1,374 30 1.8 9,661 ltmain.sh
lenny 38 7 34 60 3,404 2,730 65 1.7 20,489 configure
lenny 98 19 64 204 8,664 7,745 181 1.8 69,779 configure-coreutils

Parsing Rate (lines/ms)

host label bash dash mksh zsh osh-ovm osh-cpython osh-native num lines filename
hoover 1,090 1,577 1,108 52 23 21 562 1,679 t4014-format-patch.sh
hoover 777 1,731 792 665 8 8 281 1,733 functions
hoover 605 NA NA 307 8 9 326 2,029 configure-helper.sh
hoover 861 1,787 866 558 7 8 332 2,512 abuild
hoover 1,208 2,111 1,220 110 11 11 399 2,698 Build.sh
hoover 1,708 4,061 1,825 100 39 37 1,026 3,123 t9300-fast-import.sh
hoover 776 NA NA 595 7 7 295 5,160 test-cmd-util.sh
hoover 1,198 2,954 1,247 314 12 12 417 9,661 ltmain.sh
hoover 1,428 3,173 1,320 380 11 12 432 20,489 configure
hoover 2,034 4,371 1,943 362 13 14 542 69,779 configure-coreutils
lenny 300 2,473 1,037 58 17 17 517 1,679 t4014-format-patch.sh
lenny 264 1,985 603 576 5 5 247 1,733 functions
lenny 177 NA NA 321 5 6 275 2,029 configure-helper.sh
lenny 259 2,056 489 523 4 5 271 2,512 abuild
lenny 471 2,468 631 113 7 8 257 2,698 Build.sh
lenny 745 4,093 1,294 98 24 27 717 3,123 t9300-fast-import.sh
lenny 345 NA NA 588 4 5 176 5,160 test-cmd-util.sh
lenny 565 2,894 718 722 6 7 322 9,661 ltmain.sh
lenny 544 3,041 600 342 6 8 317 20,489 configure
lenny 714 3,772 1,094 343 8 9 386 69,779 configure-coreutils

Memory Usage (Max Resident Set Size in MB)

Again, Oil uses a different algorithm (and language) than POSIX shells. It builds an AST in memory rather than just validating the code line-by-line.

host label bash dash mksh zsh osh-ovm osh-cpython osh-native num lines filename
hoover 3.4 1.7 2.4 3.6 15.3 17.8 8.1 1,679 t4014-format-patch.sh
hoover 3.5 1.7 2.3 3.7 18.0 20.7 9.0 1,733 functions
hoover 3.6 NA NA 3.8 18.6 21.2 9.5 2,029 configure-helper.sh
hoover 3.5 1.8 2.5 3.8 20.1 22.7 10.6 2,512 abuild
hoover 3.5 1.8 2.4 3.9 18.7 21.4 9.6 2,698 Build.sh
hoover 3.6 1.7 2.3 3.8 15.6 18.1 8.1 3,123 t9300-fast-import.sh
hoover 3.7 NA NA 3.9 28.5 32.0 15.7 5,160 test-cmd-util.sh
hoover 4.4 2.1 3.4 4.5 30.7 34.2 17.6 9,661 ltmain.sh
hoover 4.2 2.5 3.0 4.3 50.3 55.9 27.2 20,489 configure
hoover 3.8 1.8 2.7 3.9 113.0 125.1 69.9 69,779 configure-coreutils
lenny 3.5 1.7 1.9 4.0 15.5 18.4 7.6 1,679 t4014-format-patch.sh
lenny 3.5 1.8 1.8 4.1 18.1 21.3 8.6 1,733 functions
lenny 3.6 NA NA 4.1 18.6 21.9 9.0 2,029 configure-helper.sh
lenny 3.4 1.8 1.9 4.1 20.1 23.3 10.1 2,512 abuild
lenny 3.3 1.7 1.9 4.0 18.9 22.3 9.2 2,698 Build.sh
lenny 3.3 1.8 1.9 4.1 15.5 18.6 7.7 3,123 t9300-fast-import.sh
lenny 3.6 NA NA 4.0 28.5 32.4 15.4 5,160 test-cmd-util.sh
lenny 4.5 2.2 3.1 4.5 30.7 35.0 17.2 9,661 ltmain.sh
lenny 4.0 2.5 2.6 4.5 50.2 56.6 27.0 20,489 configure
lenny 3.7 1.9 2.3 4.1 113.0 125.7 69.5 69,779 configure-coreutils

Shell and Host Details

shell label shell id
bash bash-9af8f89f
dash dash-9ff48631
mksh mksh-e3755400
zsh zsh-a785003a
osh-cpython osh-fe6227a2
osh-ovm osh-30efce40
osh-native osh-dfba53e5
bash bash-0d42a1ee
dash dash-d40a0412
mksh mksh-84e6599c
zsh zsh-25b8aadb
host label host id
lenny lenny-098a8e52
hoover hoover-94601b4f

Raw Data

filename
lenny.2024-02-04__00-13-26.cachegrind.tsv
lenny.2024-02-04__00-13-26.times.csv
hoover.2024-02-03__23-54-55.times.csv