OSH Parser Performance

We time $sh -n $file for various files under various shells, and repeat then run under cachegrind for stable metrics.

Source code: oil/benchmarks/osh-parser.sh

Summary

Instructions Per Line (via cachegrind)

Lower numbers are generally better, but each shell recognizes a different language, and OSH uses a more thorough parsing algorithm. In thousands of "I refs".

shell label total lines thousand irefs per line
bash 118,863 15.3
dash 111,674 1.6
mksh 111,674 5.7
osh-native 118,863 15.8

(zsh isn't measured because zsh -n unexpectedly forks.)

Average Parsing Rate, Measured on Two Machines (lines/ms)

Shell startup time is included in the elapsed time measurements, but long files are chosen to minimize its effect.

shell label total lines host hoover host lenny
dash 111,674 3,587 3,553
mksh 111,674 1,608 1,099
bash 118,863 1,466 536
osh-native 118,863 484 335
zsh 118,863 281 303
osh-cpython 118,863 12 8
osh-ovm 118,863 12 6

Per-File Measurements

Instructions Per Line (in thousands)

bash dash mksh osh-native num lines filename
15.6 1.3 4.4 7.5 1,679 t4014-format-patch.sh
16.7 2.1 8.4 24.4 1,733 functions
23.7 NA NA 19.4 2,029 configure-helper.sh
17.9 2.1 8.9 26.7 2,512 abuild
13.6 1.8 5.9 17.3 2,698 Build.sh
13.7 1.0 3.3 4.7 3,123 t9300-fast-import.sh
32.0 NA NA 28.9 5,160 test-cmd-util.sh
18.2 1.8 6.1 16.6 9,661 ltmain.sh
17.3 1.9 6.4 17.5 20,489 configure
12.9 1.5 5.4 14.2 69,779 configure-coreutils

Elapsed Time (milliseconds)

host label bash dash mksh zsh osh-ovm osh-cpython osh-native osh to bash ratio num lines filename
hoover 1 1 2 32 73 107 3 2.1 1,679 t4014-format-patch.sh
hoover 2 1 2 3 241 249 6 2.7 1,733 functions
hoover 2 NA NA 7 301 272 5 2.2 2,029 configure-helper.sh
hoover 3 1 3 4 387 342 9 2.8 2,512 abuild
hoover 2 1 2 26 264 262 7 2.8 2,698 Build.sh
hoover 2 1 2 34 87 93 3 1.5 3,123 t9300-fast-import.sh
hoover 6 NA NA 8 782 764 17 2.7 5,160 test-cmd-util.sh
hoover 9 4 8 29 853 818 23 2.5 9,661 ltmain.sh
hoover 16 6 14 53 1,845 1,768 45 2.9 20,489 configure
hoover 36 15 37 227 5,330 5,134 128 3.5 69,779 configure-coreutils
lenny 6 1 2 32 114 109 3 0.6 1,679 t4014-format-patch.sh
lenny 7 1 3 3 388 349 7 1.1 1,733 functions
lenny 11 NA NA 6 410 350 7 0.6 2,029 configure-helper.sh
lenny 11 1 4 5 675 533 11 0.9 2,512 abuild
lenny 6 1 3 23 492 357 8 1.2 2,698 Build.sh
lenny 6 1 2 35 127 121 3 0.5 3,123 t9300-fast-import.sh
lenny 15 NA NA 9 1,497 1,200 24 1.6 5,160 test-cmd-util.sh
lenny 18 3 9 14 1,656 1,306 28 1.6 9,661 ltmain.sh
lenny 37 6 19 58 3,132 2,801 63 1.7 20,489 configure
lenny 104 17 61 206 10,020 7,904 201 1.9 69,779 configure-coreutils

Parsing Rate (lines/ms)

host label bash dash mksh zsh osh-ovm osh-cpython osh-native num lines filename
hoover 1,130 1,616 1,110 52 23 16 541 1,679 t4014-format-patch.sh
hoover 793 1,530 762 505 7 7 297 1,733 functions
hoover 831 NA NA 281 7 7 374 2,029 configure-helper.sh
hoover 777 1,790 876 598 6 7 276 2,512 abuild
hoover 1,161 2,105 1,158 104 10 10 408 2,698 Build.sh
hoover 1,648 3,213 1,885 92 36 34 1,073 3,123 t9300-fast-import.sh
hoover 807 NA NA 646 7 7 298 5,160 test-cmd-util.sh
hoover 1,043 2,724 1,217 338 11 12 424 9,661 ltmain.sh
hoover 1,307 3,200 1,437 385 11 12 457 20,489 configure
hoover 1,928 4,544 1,907 308 13 14 547 69,779 configure-coreutils
lenny 300 2,352 1,099 52 15 15 492 1,679 t4014-format-patch.sh
lenny 263 1,776 641 549 4 5 248 1,733 functions
lenny 177 NA NA 329 5 6 299 2,029 configure-helper.sh
lenny 220 2,198 643 499 4 5 238 2,512 abuild
lenny 425 2,201 937 118 5 8 353 2,698 Build.sh
lenny 505 3,529 1,965 89 25 26 945 3,123 t9300-fast-import.sh
lenny 333 NA NA 594 3 4 213 5,160 test-cmd-util.sh
lenny 548 3,145 1,057 669 6 7 342 9,661 ltmain.sh
lenny 552 3,401 1,071 353 7 7 327 20,489 configure
lenny 671 4,012 1,149 338 7 9 347 69,779 configure-coreutils

Memory Usage (Max Resident Set Size in MB)

Again, OSH uses a different algorithm (and language) than POSIX shells. It builds an AST in memory rather than just validating the code line-by-line.

host label bash dash mksh zsh osh-ovm osh-cpython osh-native num lines filename
hoover 3.6 1.8 2.5 3.9 15.5 17.8 8.1 1,679 t4014-format-patch.sh
hoover 3.6 1.7 2.4 3.8 18.3 21.0 9.0 1,733 functions
hoover 3.7 NA NA 3.9 18.9 21.7 9.6 2,029 configure-helper.sh
hoover 3.5 1.7 2.5 3.9 20.5 23.2 10.7 2,512 abuild
hoover 3.6 1.8 2.5 3.8 18.8 21.8 9.6 2,698 Build.sh
hoover 3.6 1.7 2.3 3.9 15.8 18.4 8.3 3,123 t9300-fast-import.sh
hoover 3.7 NA NA 4.0 29.1 32.6 15.7 5,160 test-cmd-util.sh
hoover 4.5 2.2 3.6 4.6 31.0 34.7 18.0 9,661 ltmain.sh
hoover 4.1 2.5 3.2 4.3 50.6 56.3 27.7 20,489 configure
hoover 3.8 1.8 2.8 4.1 113.6 125.4 70.8 69,779 configure-coreutils
lenny 3.4 1.7 1.8 3.9 15.4 18.5 7.7 1,679 t4014-format-patch.sh
lenny 3.4 1.8 1.9 4.1 18.3 21.4 8.6 1,733 functions
lenny 3.4 NA NA 4.1 19.0 22.4 9.3 2,029 configure-helper.sh
lenny 3.4 1.8 1.9 4.1 20.3 23.8 10.4 2,512 abuild
lenny 3.5 1.8 1.8 4.2 19.0 22.3 9.1 2,698 Build.sh
lenny 3.3 1.8 1.9 4.0 16.1 19.2 7.8 3,123 t9300-fast-import.sh
lenny 3.6 NA NA 4.1 29.0 33.1 15.2 5,160 test-cmd-util.sh
lenny 4.4 2.1 2.9 4.6 31.0 35.5 17.7 9,661 ltmain.sh
lenny 4.1 2.5 2.7 4.3 50.6 56.9 27.4 20,489 configure
lenny 3.8 1.9 2.3 4.1 113.7 126.1 70.6 69,779 configure-coreutils

Shell and Host Details

shell label shell id
bash bash-9af8f89f
dash dash-9ff48631
mksh mksh-e3755400
zsh zsh-a785003a
osh-cpython osh-451031a6
osh-ovm osh-348dd5cd
osh-native osh-4cf62bfe
bash bash-0d42a1ee
dash dash-d40a0412
mksh mksh-84e6599c
zsh zsh-25b8aadb
host label host id
lenny lenny-098a8e52
hoover hoover-94601b4f

Raw Data

filename
lenny.2024-03-13__14-43-52.cachegrind.tsv
lenny.2024-03-13__14-43-52.times.csv
hoover.2024-03-13__14-14-15.times.csv