OSH Parser Performance

We time $sh -n $file for various files under various shells, and repeat then run under cachegrind for stable metrics.

Summary

Instructions Per Line (via cachegrind)

Lower numbers are generally better, but each shell recognizes a different language, and Oil uses a more thorough parsing algorithm. In thousands of "I refs".

shell label total lines thousand irefs per line
bash 118,863 15.3
dash 111,674 1.6
mksh 111,674 5.7
oil-native 118,863 9.0

(zsh isn't measured because zsh -n unexpectedly forks.)

Average Parsing Rate, Measured on Two Machines (lines/ms)

Shell startup time is included in the elapsed time measurements, but long files are chosen to minimize its effect.

shell label total lines host broome host lenny
dash 111,674 1,445.5 2,982.2
mksh 111,674 581.7 963.3
bash 118,863 341.6 545.6
oil-native 118,863 296.8 542.4
zsh 118,863 102.6 246.9
osh-cpython 118,863 7.3 8.6
osh-ovm 118,863 6.5 8.0

Breakdown By File

Instructions Per Line (in thousands)

bash dash mksh oil-native num lines filename
15.6 1.3 4.4 4.9 1,679 t4014-format-patch.sh
16.7 2.1 8.4 14.0 1,733 functions
23.7 NA NA 12.7 2,029 configure-helper.sh
17.8 2.1 8.8 14.6 2,512 abuild
13.6 1.8 5.9 10.2 2,698 Build.sh
13.7 1.0 3.3 3.2 3,123 t9300-fast-import.sh
32.0 NA NA 15.9 5,160 test-cmd-util.sh
18.2 1.8 6.1 9.6 9,661 ltmain.sh
17.3 1.9 6.4 9.8 20,489 configure
12.9 1.5 5.4 8.1 69,779 configure-coreutils

Elasped Time (milliseconds)

host label bash dash mksh zsh osh-ovm osh-cpython oil-native osh to bash ratio num lines filename
broome 9 2 4 100 145 164 6 0.7 1,679 t4014-format-patch.sh
broome 10 3 7 13 412 412 13 1.3 1,733 functions
broome 15 NA NA 22 462 432 13 0.9 2,029 configure-helper.sh
broome 15 4 11 15 608 603 18 1.2 2,512 abuild
broome 12 3 8 78 447 446 16 1.3 2,698 Build.sh
broome 13 2 5 111 139 135 7 0.5 3,123 t9300-fast-import.sh
broome 31 NA NA 28 1,330 1,161 37 1.2 5,160 test-cmd-util.sh
broome 34 9 23 41 1,580 1,383 34 1.0 9,661 ltmain.sh
broome 62 18 41 121 3,416 2,924 67 1.1 20,489 configure
broome 146 36 92 629 9,716 8,543 187 1.3 69,779 configure-coreutils
lenny 4 1 12 45 106 108 2 0.6 1,679 t4014-format-patch.sh
lenny 4 1 3 3 323 335 6 1.4 1,733 functions
lenny 7 NA NA 8 347 325 6 0.9 2,029 configure-helper.sh
lenny 6 2 4 7 497 509 8 1.2 2,512 abuild
lenny 6 2 3 37 355 398 6 1.0 2,698 Build.sh
lenny 6 1 2 45 108 131 3 0.4 3,123 t9300-fast-import.sh
lenny 21 NA NA 12 1,055 971 17 0.8 5,160 test-cmd-util.sh
lenny 21 4 13 18 1,203 1,095 20 0.9 9,661 ltmain.sh
lenny 43 8 23 55 2,741 2,451 40 0.9 20,489 configure
lenny 99 19 56 252 8,157 7,537 110 1.1 69,779 configure-coreutils

Parsing Rate (lines/ms)

host label bash dash mksh zsh osh-ovm osh-cpython oil-native num lines filename
broome 195.7 840.8 402.3 16.7 11.6 10.2 268.3 1,679 t4014-format-patch.sh
broome 170.0 635.5 238.1 135.1 4.2 4.2 129.7 1,733 functions
broome 136.8 NA NA 93.1 4.4 4.7 151.0 2,029 configure-helper.sh
broome 168.4 712.6 225.0 168.0 4.1 4.2 139.5 2,512 abuild
broome 217.8 831.4 348.7 34.5 6.0 6.1 167.3 2,698 Build.sh
broome 243.3 1,415.0 591.8 28.2 22.5 23.1 451.6 3,123 t9300-fast-import.sh
broome 164.8 NA NA 181.9 3.9 4.4 137.9 5,160 test-cmd-util.sh
broome 282.0 1,029.2 412.7 234.1 6.1 7.0 280.1 9,661 ltmain.sh
broome 329.1 1,128.3 503.4 169.4 6.0 7.0 305.3 20,489 configure
broome 476.8 1,937.9 756.4 111.0 7.2 8.2 372.5 69,779 configure-coreutils
lenny 450.1 2,020.5 142.2 37.6 15.8 15.5 708.7 1,679 t4014-format-patch.sh
lenny 413.7 1,574.0 508.5 506.4 5.4 5.2 293.0 1,733 functions
lenny 290.3 NA NA 244.2 5.9 6.2 321.7 2,029 configure-helper.sh
lenny 399.2 1,350.5 558.2 374.5 5.1 4.9 324.0 2,512 abuild
lenny 455.3 1,415.5 953.0 72.6 7.6 6.8 443.6 2,698 Build.sh
lenny 485.8 2,766.2 1,677.2 69.6 28.9 23.9 1,157.5 3,123 t9300-fast-import.sh
lenny 248.0 NA NA 443.0 4.9 5.3 300.6 5,160 test-cmd-util.sh
lenny 450.4 2,284.5 756.8 534.6 8.0 8.8 476.1 9,661 ltmain.sh
lenny 477.0 2,660.9 906.0 375.8 7.5 8.4 506.3 20,489 configure
lenny 704.1 3,733.3 1,243.0 276.9 8.6 9.3 633.9 69,779 configure-coreutils

Memory Usage (Max Resident Set Size in MB)

Again, Oil uses a different algorithm (and language) than POSIX shells. It builds an AST in memory rather than just validating the code line-by-line.

host label bash dash mksh zsh osh-ovm osh-cpython oil-native num lines filename
broome 3.5 1.8 1.9 4.0 14.5 18.2 5.5 1,679 t4014-format-patch.sh
broome 3.5 1.8 1.9 4.0 17.3 21.1 7.9 1,733 functions
broome 3.4 NA NA 4.1 17.6 21.9 8.4 2,029 configure-helper.sh
broome 3.2 1.7 1.7 4.0 19.3 23.4 10.1 2,512 abuild
broome 3.4 1.8 2.0 4.0 17.8 22.0 8.7 2,698 Build.sh
broome 3.5 1.8 1.9 3.9 14.7 18.4 5.9 3,123 t9300-fast-import.sh
broome 3.6 NA NA 4.1 28.3 33.6 18.0 5,160 test-cmd-util.sh
broome 4.4 2.2 3.1 4.4 29.9 35.5 20.3 9,661 ltmain.sh
broome 4.0 2.4 2.7 4.5 49.8 58.6 39.7 20,489 configure
broome 3.8 1.8 2.1 4.2 112.6 130.7 106.8 69,779 configure-coreutils
lenny 3.4 1.8 1.9 4.1 14.6 18.2 5.5 1,679 t4014-format-patch.sh
lenny 3.4 1.8 1.8 3.9 17.1 21.0 7.8 1,733 functions
lenny 3.5 NA NA 4.1 17.8 21.7 8.4 2,029 configure-helper.sh
lenny 3.4 1.7 1.9 4.1 19.4 23.2 10.1 2,512 abuild
lenny 3.5 1.7 2.0 4.0 18.0 21.9 8.8 2,698 Build.sh
lenny 3.5 1.7 1.8 4.0 14.9 18.4 6.0 3,123 t9300-fast-import.sh
lenny 3.6 NA NA 4.0 28.2 33.3 18.0 5,160 test-cmd-util.sh
lenny 4.3 2.2 3.1 4.6 29.8 35.5 20.2 9,661 ltmain.sh
lenny 4.0 2.5 2.6 4.3 49.9 58.7 39.7 20,489 configure
lenny 3.7 1.8 2.2 4.2 112.5 130.6 107.0 69,779 configure-coreutils

Shell and Host Details

shell label shell id
bash bash-9af8f89f
dash dash-9ff48631
mksh mksh-e3755400
oil-native osh_eval.stripped-5daf0f08
host label host id
broome broome-8a85d07d
lenny lenny-23f35694

Raw Data

filename
no-host.2022-07-30__01-43-51.cachegrind.tsv
broome.2022-07-30__01-43-51.times.csv
lenny.2022-07-29__21-35-40.times.csv