OSH Parser Performance

We time $sh -n $file for various files under various shells, and repeat then run under cachegrind for stable metrics.

Summary

Instructions Per Line (via cachegrind)

Lower numbers are generally better, but each shell recognizes a different language, and Oil uses a more thorough parsing algorithm. In thousands of "I refs".

shell label total lines thousand irefs per line
bash 118,863 15.4
dash 111,674 1.6
mksh 111,674 5.7
oil-native 118,863 16.4

(zsh isn't measured because zsh -n unexpectedly forks.)

Average Parsing Rate, Measured on Two Machines (lines/ms)

Shell startup time is included in the elapsed time measurements, but long files are chosen to minimize its effect.

shell label total lines host broome host lenny
dash 111,674 1,484.4 3,823.8
mksh 111,674 589.1 992.0
bash 118,863 349.8 626.3
oil-native 118,863 234.8 433.6
zsh 118,863 103.3 306.0
osh-cpython 118,863 7.3 9.2
osh-ovm 118,863 6.6 8.4

Breakdown By File

Instructions Per Line (in thousands)

bash dash mksh oil-native num lines filename
15.6 1.4 4.4 8.3 1,679 t4014-format-patch.sh
16.7 2.1 8.4 24.9 1,733 functions
23.7 NA NA 23.3 2,029 configure-helper.sh
17.9 2.2 8.9 26.1 2,512 abuild
13.6 1.8 5.9 18.7 2,698 Build.sh
13.7 1.0 3.3 5.4 3,123 t9300-fast-import.sh
32.0 NA NA 27.8 5,160 test-cmd-util.sh
18.2 1.8 6.1 17.3 9,661 ltmain.sh
17.3 1.9 6.4 18.0 20,489 configure
12.9 1.5 5.4 14.7 69,779 configure-coreutils

Elasped Time (milliseconds)

host label bash dash mksh zsh osh-ovm osh-cpython oil-native osh to bash ratio num lines filename
broome 9 2 4 102 157 164 8 0.9 1,679 t4014-format-patch.sh
broome 10 3 7 13 432 405 15 1.5 1,733 functions
broome 15 NA NA 22 446 392 18 1.2 2,029 configure-helper.sh
broome 15 4 11 18 614 586 23 1.6 2,512 abuild
broome 10 3 8 78 445 410 18 1.7 2,698 Build.sh
broome 12 2 5 111 140 138 9 0.7 3,123 t9300-fast-import.sh
broome 31 NA NA 27 1,343 1,221 36 1.2 5,160 test-cmd-util.sh
broome 34 9 24 42 1,553 1,363 48 1.4 9,661 ltmain.sh
broome 62 18 39 120 3,333 3,000 97 1.6 20,489 configure
broome 142 35 91 618 9,592 8,626 234 1.6 69,779 configure-coreutils
lenny 4 1 1 29 89 89 3 0.7 1,679 t4014-format-patch.sh
lenny 5 1 2 3 300 271 8 1.5 1,733 functions
lenny 7 NA NA 6 381 303 7 1.0 2,029 configure-helper.sh
lenny 8 1 3 5 471 472 9 1.2 2,512 abuild
lenny 7 1 3 25 361 317 9 1.2 2,698 Build.sh
lenny 8 1 2 31 113 95 3 0.4 3,123 t9300-fast-import.sh
lenny 17 NA NA 9 991 927 21 1.2 5,160 test-cmd-util.sh
lenny 18 3 14 18 1,142 1,153 25 1.4 9,661 ltmain.sh
lenny 34 7 28 67 2,641 2,395 53 1.5 20,489 configure
lenny 82 15 59 195 7,637 6,938 137 1.7 69,779 configure-coreutils

Parsing Rate (lines/ms)

host label bash dash mksh zsh osh-ovm osh-cpython oil-native num lines filename
broome 195.8 879.1 432.2 16.5 10.7 10.2 219.0 1,679 t4014-format-patch.sh
broome 169.4 653.7 239.3 136.2 4.0 4.3 112.9 1,733 functions
broome 138.1 NA NA 93.2 4.6 5.2 111.9 2,029 configure-helper.sh
broome 167.8 699.7 229.3 138.5 4.1 4.3 107.4 2,512 abuild
broome 257.3 838.1 346.5 34.4 6.1 6.6 154.1 2,698 Build.sh
broome 270.4 1,412.5 592.9 28.1 22.3 22.6 364.2 3,123 t9300-fast-import.sh
broome 165.0 NA NA 188.5 3.8 4.2 143.0 5,160 test-cmd-util.sh
broome 285.4 1,032.7 396.7 232.6 6.2 7.1 199.9 9,661 ltmain.sh
broome 331.5 1,154.4 529.9 171.4 6.1 6.8 211.4 20,489 configure
broome 490.0 2,019.9 763.2 112.9 7.3 8.1 297.8 69,779 configure-coreutils
lenny 421.4 2,826.6 1,231.8 57.8 18.9 18.9 603.7 1,679 t4014-format-patch.sh
lenny 333.8 2,003.5 751.5 546.7 5.8 6.4 215.5 1,733 functions
lenny 298.5 NA NA 333.8 5.3 6.7 302.8 2,029 configure-helper.sh
lenny 319.4 2,128.8 767.0 484.2 5.3 5.3 269.1 2,512 abuild
lenny 369.3 2,739.1 961.2 109.5 7.5 8.5 316.3 2,698 Build.sh
lenny 411.5 4,572.5 1,694.5 99.4 27.6 32.8 1,080.2 3,123 t9300-fast-import.sh
lenny 299.9 NA NA 553.0 5.2 5.6 247.9 5,160 test-cmd-util.sh
lenny 551.7 3,239.8 690.2 541.3 8.5 8.4 388.1 9,661 ltmain.sh
lenny 596.3 3,140.6 737.6 307.1 7.8 8.6 388.6 20,489 configure
lenny 851.1 4,533.5 1,178.7 357.9 9.1 10.1 507.7 69,779 configure-coreutils

Memory Usage (Max Resident Set Size in MB)

Again, Oil uses a different algorithm (and language) than POSIX shells. It builds an AST in memory rather than just validating the code line-by-line.

host label bash dash mksh zsh osh-ovm osh-cpython oil-native num lines filename
broome 3.3 1.7 1.9 4.0 14.6 18.0 5.5 1,679 t4014-format-patch.sh
broome 3.5 1.8 1.9 4.0 17.3 21.1 8.6 1,733 functions
broome 3.3 NA NA 4.0 17.8 21.6 9.2 2,029 configure-helper.sh
broome 3.4 1.7 1.9 4.1 19.5 23.5 11.5 2,512 abuild
broome 3.5 1.7 2.0 3.9 18.0 22.0 9.8 2,698 Build.sh
broome 3.3 1.8 1.9 4.1 15.0 18.4 5.8 3,123 t9300-fast-import.sh
broome 3.5 NA NA 4.0 28.3 33.5 20.7 5,160 test-cmd-util.sh
broome 4.5 2.2 3.0 4.5 29.9 35.4 24.1 9,661 ltmain.sh
broome 4.1 2.4 2.6 4.4 50.1 58.6 49.4 20,489 configure
broome 3.8 1.8 2.2 4.2 112.6 130.7 133.1 69,779 configure-coreutils
lenny 3.4 1.7 1.8 4.0 14.6 18.0 5.4 1,679 t4014-format-patch.sh
lenny 3.5 1.7 1.9 4.0 17.2 21.1 8.7 1,733 functions
lenny 3.5 NA NA 4.1 17.7 21.8 9.2 2,029 configure-helper.sh
lenny 3.4 1.7 1.9 4.1 19.5 23.6 11.4 2,512 abuild
lenny 3.3 1.8 2.0 4.2 18.0 22.1 9.9 2,698 Build.sh
lenny 3.5 1.7 1.7 4.0 14.8 18.3 5.9 3,123 t9300-fast-import.sh
lenny 3.5 NA NA 4.1 28.2 33.4 20.6 5,160 test-cmd-util.sh
lenny 4.4 2.2 3.0 4.6 29.9 35.4 24.5 9,661 ltmain.sh
lenny 4.0 2.4 2.7 4.5 50.0 58.6 49.3 20,489 configure
lenny 3.7 1.9 2.3 4.2 112.9 130.8 133.2 69,779 configure-coreutils

Shell and Host Details

shell label shell id
bash bash-9af8f89f
dash dash-9ff48631
mksh mksh-e3755400
oil-native osh_eval.stripped-cd062c64
host label host id
broome broome-8a85d07d
lenny lenny-b7f988ad

Raw Data

filename
no-host.2022-08-20__11-59-08.cachegrind.tsv
broome.2022-08-20__16-16-03.times.csv
lenny.2022-08-20__11-59-52.times.csv