OSH Parser Performance

We time $sh -n $file for various files under various shells, and repeat then run under cachegrind for stable metrics.

Source code: oil/benchmarks/osh-parser.sh

Summary

Instructions Per Line (via cachegrind)

Lower numbers are generally better, but each shell recognizes a different language, and Oil uses a more thorough parsing algorithm. In thousands of "I refs".

shell label total lines thousand irefs per line
bash 118,863 15.4
dash 111,674 1.6
mksh 111,674 5.7
osh-native 118,863 22.1

(zsh isn't measured because zsh -n unexpectedly forks.)

Average Parsing Rate, Measured on Two Machines (lines/ms)

Shell startup time is included in the elapsed time measurements, but long files are chosen to minimize its effect.

shell label total lines host broome host lenny
dash 111,674 1,455 3,070
mksh 111,674 595 1,139
bash 118,863 352 600
zsh 118,863 103 308
osh-native 118,863 167 237
osh-cpython 118,863 7 8
osh-ovm 118,863 6 7

Per-File Measurements

Instructions Per Line (in thousands)

bash dash mksh osh-native num lines filename
15.7 1.4 4.4 9.9 1,679 t4014-format-patch.sh
16.7 2.1 8.4 34.2 1,733 functions
23.7 NA NA 27.5 2,029 configure-helper.sh
17.9 2.2 8.9 37.0 2,512 abuild
13.6 1.8 5.9 23.7 2,698 Build.sh
13.7 1.0 3.3 6.1 3,123 t9300-fast-import.sh
32.0 NA NA 33.7 5,160 test-cmd-util.sh
18.2 1.8 6.1 24.1 9,661 ltmain.sh
17.4 1.9 6.4 24.7 20,489 configure
12.9 1.5 5.4 20.1 69,779 configure-coreutils

Elasped Time (milliseconds)

host label bash dash mksh zsh osh-ovm osh-cpython osh-native osh to bash ratio num lines filename
broome 9 2 4 99 154 160 10 1.2 1,679 t4014-format-patch.sh
broome 10 3 8 13 418 414 20 1.9 1,733 functions
broome 13 NA NA 22 447 456 23 1.8 2,029 configure-helper.sh
broome 14 4 11 18 666 563 29 2.1 2,512 abuild
broome 11 3 8 73 477 454 24 2.1 2,698 Build.sh
broome 12 2 5 113 147 171 11 0.9 3,123 t9300-fast-import.sh
broome 28 NA NA 28 1,382 1,252 50 1.8 5,160 test-cmd-util.sh
broome 36 10 23 41 1,578 1,387 69 1.9 9,661 ltmain.sh
broome 61 18 37 123 3,453 2,998 139 2.3 20,489 configure
broome 143 36 92 623 9,648 8,529 337 2.4 69,779 configure-coreutils
lenny 5 1 2 30 104 98 4 0.9 1,679 t4014-format-patch.sh
lenny 5 1 5 3 371 310 10 1.9 1,733 functions
lenny 8 NA NA 6 390 352 9 1.2 2,029 configure-helper.sh
lenny 7 1 6 5 531 494 15 2.1 2,512 abuild
lenny 4 1 4 24 412 361 11 2.6 2,698 Build.sh
lenny 4 1 3 36 114 105 5 1.1 3,123 t9300-fast-import.sh
lenny 16 NA NA 10 1,267 1,008 27 1.7 5,160 test-cmd-util.sh
lenny 18 4 9 16 1,441 1,240 41 2.3 9,661 ltmain.sh
lenny 35 7 19 50 3,167 2,674 97 2.8 20,489 configure
lenny 96 20 51 206 8,271 7,683 283 2.9 69,779 configure-coreutils

Parsing Rate (lines/ms)

host label bash dash mksh zsh osh-ovm osh-cpython osh-native num lines filename
broome 193 858 394 17 11 10 164 1,679 t4014-format-patch.sh
broome 167 627 230 134 4 4 86 1,733 functions
broome 153 NA NA 94 5 4 87 2,029 configure-helper.sh
broome 181 715 232 139 4 4 87 2,512 abuild
broome 237 838 344 37 6 6 113 2,698 Build.sh
broome 269 1,392 571 28 21 18 294 3,123 t9300-fast-import.sh
broome 182 NA NA 185 4 4 103 5,160 test-cmd-util.sh
broome 272 1,016 421 236 6 7 140 9,661 ltmain.sh
broome 333 1,149 554 166 6 7 147 20,489 configure
broome 488 1,954 759 112 7 8 207 69,779 configure-coreutils
lenny 368 2,392 927 55 16 17 429 1,679 t4014-format-patch.sh
lenny 324 1,826 363 557 5 6 172 1,733 functions
lenny 259 NA NA 336 5 6 215 2,029 configure-helper.sh
lenny 356 1,700 437 526 5 5 172 2,512 abuild
lenny 655 1,862 632 113 7 7 255 2,698 Build.sh
lenny 706 3,123 1,121 88 27 30 667 3,123 t9300-fast-import.sh
lenny 324 NA NA 497 4 5 192 5,160 test-cmd-util.sh
lenny 543 2,650 1,107 618 7 8 237 9,661 ltmain.sh
lenny 587 2,798 1,107 411 6 8 211 20,489 configure
lenny 725 3,519 1,357 338 8 9 247 69,779 configure-coreutils

Memory Usage (Max Resident Set Size in MB)

Again, Oil uses a different algorithm (and language) than POSIX shells. It builds an AST in memory rather than just validating the code line-by-line.

host label bash dash mksh zsh osh-ovm osh-cpython osh-native num lines filename
broome 3.4 1.7 1.9 4.0 14.6 18.3 7.5 1,679 t4014-format-patch.sh
broome 3.3 1.7 1.7 4.1 17.7 21.3 8.9 1,733 functions
broome 3.5 NA NA 4.0 18.3 22.1 9.4 2,029 configure-helper.sh
broome 3.4 1.7 1.9 3.9 20.0 23.8 10.7 2,512 abuild
broome 3.3 1.8 1.9 4.1 18.7 22.2 9.6 2,698 Build.sh
broome 3.4 1.7 1.8 4.1 15.1 18.7 7.7 3,123 t9300-fast-import.sh
broome 3.6 NA NA 4.0 28.4 32.6 16.5 5,160 test-cmd-util.sh
broome 4.4 2.2 3.1 4.6 30.9 35.8 20.0 9,661 ltmain.sh
broome 4.1 2.5 2.5 4.4 51.6 58.4 31.3 20,489 configure
broome 3.7 1.9 2.2 4.1 117.1 130.5 81.0 69,779 configure-coreutils
lenny 3.4 1.7 1.9 4.1 14.7 18.4 7.7 1,679 t4014-format-patch.sh
lenny 3.3 1.8 2.0 3.8 17.6 21.3 8.8 1,733 functions
lenny 3.6 NA NA 4.0 18.3 22.0 9.5 2,029 configure-helper.sh
lenny 3.4 1.7 1.9 3.9 19.8 23.7 10.7 2,512 abuild
lenny 3.3 1.7 1.9 4.0 18.5 22.3 9.7 2,698 Build.sh
lenny 3.4 1.7 1.9 4.1 15.1 18.7 7.6 3,123 t9300-fast-import.sh
lenny 3.6 NA NA 3.9 28.3 32.6 16.6 5,160 test-cmd-util.sh
lenny 4.3 2.1 3.1 4.6 31.0 35.9 20.0 9,661 ltmain.sh
lenny 3.9 2.5 2.7 4.2 51.6 58.4 31.2 20,489 configure
lenny 3.9 1.9 2.2 4.1 117.1 130.3 81.3 69,779 configure-coreutils

Shell and Host Details

shell label shell id
bash bash-9af8f89f
dash dash-9ff48631
mksh mksh-e3755400
zsh zsh-a785003a
osh-cpython osh-424d86f7
osh-ovm osh-613703d2
osh-native osh-3c290bd3
host label host id
broome broome-7a69aabc
lenny lenny-5570ba51

Raw Data

filename
lenny.2023-05-03__15-25-16.cachegrind.tsv
broome.2023-05-03__19-01-59.times.csv
lenny.2023-05-03__15-25-16.times.csv