OSH Parser Performance

We time $sh -n $file for various files under various shells, and repeat then run under cachegrind for stable metrics.

Source code: oil/benchmarks/osh-parser.sh

Summary

Instructions Per Line (via cachegrind)

Lower numbers are generally better, but each shell recognizes a different language, and Oil uses a more thorough parsing algorithm. In thousands of "I refs".

shell label total lines thousand irefs per line
bash 118,863 15.4
dash 111,674 1.6
mksh 111,674 5.7
oil-native 118,863 21.8

(zsh isn't measured because zsh -n unexpectedly forks.)

Average Parsing Rate, Measured on Two Machines (lines/ms)

Shell startup time is included in the elapsed time measurements, but long files are chosen to minimize its effect.

shell label total lines host broome host lenny
dash 111,674 1,500 3,052
mksh 111,674 602 1,261
bash 118,863 357 545
zsh 118,863 111 310
oil-native 118,863 167 297
osh-cpython 118,863 6 9
osh-ovm 118,863 6 8

Per-File Measurements

Instructions Per Line (in thousands)

bash dash mksh oil-native num lines filename
15.7 1.4 4.4 10.2 1,679 t4014-format-patch.sh
16.7 2.1 8.4 32.9 1,733 functions
23.7 NA NA 30.3 2,029 configure-helper.sh
17.9 2.2 8.9 34.5 2,512 abuild
13.6 1.8 6.0 24.4 2,698 Build.sh
13.7 1.0 3.3 6.7 3,123 t9300-fast-import.sh
32.0 NA NA 37.2 5,160 test-cmd-util.sh
18.2 1.8 6.1 23.3 9,661 ltmain.sh
17.4 1.9 6.4 24.1 20,489 configure
12.9 1.5 5.4 19.6 69,779 configure-coreutils

Elasped Time (milliseconds)

host label bash dash mksh zsh osh-ovm osh-cpython oil-native osh to bash ratio num lines filename
broome 8 2 4 94 158 154 10 1.2 1,679 t4014-format-patch.sh
broome 10 3 7 12 480 444 21 2.0 1,733 functions
broome 14 NA NA 20 507 469 23 1.7 2,029 configure-helper.sh
broome 12 4 11 17 747 631 28 2.3 2,512 abuild
broome 11 3 8 73 506 472 22 1.9 2,698 Build.sh
broome 11 2 5 103 156 169 10 1.0 3,123 t9300-fast-import.sh
broome 30 NA NA 28 1,556 1,302 50 1.6 5,160 test-cmd-util.sh
broome 35 9 22 40 1,776 1,611 63 1.8 9,661 ltmain.sh
broome 62 19 36 113 3,814 3,365 131 2.1 20,489 configure
broome 139 34 92 573 11,124 9,709 355 2.5 69,779 configure-coreutils
lenny 5 1 2 29 88 89 3 0.7 1,679 t4014-format-patch.sh
lenny 6 1 3 3 311 270 8 1.5 1,733 functions
lenny 8 NA NA 6 319 295 9 1.2 2,029 configure-helper.sh
lenny 8 1 5 5 501 428 12 1.5 2,512 abuild
lenny 7 1 3 24 344 300 10 1.4 2,698 Build.sh
lenny 7 1 2 32 100 95 3 0.5 3,123 t9300-fast-import.sh
lenny 25 NA NA 9 1,056 867 26 1.1 5,160 test-cmd-util.sh
lenny 28 4 9 13 1,191 1,072 31 1.1 9,661 ltmain.sh
lenny 44 7 17 55 2,641 2,317 69 1.6 20,489 configure
lenny 81 21 46 207 7,959 6,857 228 2.8 69,779 configure-coreutils

Parsing Rate (lines/ms)

host label bash dash mksh zsh osh-ovm osh-cpython oil-native num lines filename
broome 208 861 391 18 11 11 173 1,679 t4014-format-patch.sh
broome 169 645 240 141 4 4 84 1,733 functions
broome 145 NA NA 103 4 4 87 2,029 configure-helper.sh
broome 204 717 238 151 3 4 90 2,512 abuild
broome 236 852 352 37 5 6 125 2,698 Build.sh
broome 290 1,477 596 30 20 18 298 3,123 t9300-fast-import.sh
broome 170 NA NA 186 3 4 104 5,160 test-cmd-util.sh
broome 277 1,132 436 244 5 6 154 9,661 ltmain.sh
broome 333 1,106 566 181 5 6 156 20,489 configure
broome 501 2,055 758 122 6 7 197 69,779 configure-coreutils
lenny 350 2,682 923 57 19 19 499 1,679 t4014-format-patch.sh
lenny 311 1,967 532 582 6 6 208 1,733 functions
lenny 249 NA NA 331 6 7 215 2,029 configure-helper.sh
lenny 309 1,887 500 479 5 6 206 2,512 abuild
lenny 397 2,286 779 113 8 9 282 2,698 Build.sh
lenny 443 3,781 1,347 99 31 33 925 3,123 t9300-fast-import.sh
lenny 208 NA NA 602 5 6 196 5,160 test-cmd-util.sh
lenny 348 2,594 1,097 761 8 9 308 9,661 ltmain.sh
lenny 464 2,750 1,173 370 8 9 297 20,489 configure
lenny 862 3,392 1,505 337 9 10 306 69,779 configure-coreutils

Memory Usage (Max Resident Set Size in MB)

Again, Oil uses a different algorithm (and language) than POSIX shells. It builds an AST in memory rather than just validating the code line-by-line.

host label bash dash mksh zsh osh-ovm osh-cpython oil-native num lines filename
broome 3.3 1.8 1.9 4.0 14.5 17.9 5.6 1,679 t4014-format-patch.sh
broome 3.4 1.7 1.9 4.0 17.3 21.2 9.2 1,733 functions
broome 3.4 NA NA 4.0 17.8 21.6 9.7 2,029 configure-helper.sh
broome 3.5 1.8 1.8 4.1 19.4 23.4 12.4 2,512 abuild
broome 3.4 1.8 1.8 4.0 18.1 22.1 10.2 2,698 Build.sh
broome 3.4 1.8 1.8 4.1 14.8 18.3 6.1 3,123 t9300-fast-import.sh
broome 3.7 NA NA 3.9 28.3 33.4 22.1 5,160 test-cmd-util.sh
broome 4.4 2.1 3.0 4.5 29.9 35.2 26.4 9,661 ltmain.sh
broome 4.0 2.5 2.7 4.5 50.0 58.6 53.1 20,489 configure
broome 3.6 1.8 2.2 4.2 112.8 130.5 144.3 69,779 configure-coreutils
lenny 3.4 1.8 1.9 4.1 14.4 18.0 5.9 1,679 t4014-format-patch.sh
lenny 3.4 1.8 1.8 4.0 17.3 21.1 9.1 1,733 functions
lenny 3.3 NA NA 4.0 17.8 21.8 9.8 2,029 configure-helper.sh
lenny 3.4 1.7 2.0 4.0 19.3 23.4 12.4 2,512 abuild
lenny 3.4 1.8 2.0 4.1 17.9 22.2 10.5 2,698 Build.sh
lenny 3.4 1.8 1.9 4.0 14.8 18.5 6.2 3,123 t9300-fast-import.sh
lenny 3.5 NA NA 4.1 28.3 33.4 22.0 5,160 test-cmd-util.sh
lenny 4.3 2.2 3.0 4.5 29.9 35.4 26.3 9,661 ltmain.sh
lenny 4.0 2.5 2.7 4.3 50.1 58.8 53.2 20,489 configure
lenny 3.7 1.8 2.2 4.1 112.8 130.7 144.3 69,779 configure-coreutils

Shell and Host Details

shell label shell id
bash bash-9af8f89f
dash dash-9ff48631
mksh mksh-e3755400
zsh zsh-a785003a
osh-cpython osh-b6fe450e
osh-ovm osh-d5fafee5
oil-native osh_eval.stripped-b4a35663
host label host id
broome broome-8a85d07d
lenny lenny-c6bbd9ee

Raw Data

filename
no-host.2022-11-10__11-53-07.cachegrind.tsv
broome.2022-11-10__16-38-01.times.csv
lenny.2022-11-10__11-54-03.times.csv

raw files