OSH Parser Performance

We time $sh -n $file for various files under various shells, and repeat then run under cachegrind for stable metrics.

Summary

Instructions Per Line (via cachegrind)

Lower numbers are generally better, but each shell recognizes a different language, and Oil uses a more thorough parsing algorithm. In thousands of "I refs".

shell label total lines thousand irefs per line
bash 118,863 15.4
dash 111,674 1.6
mksh 111,674 5.7
oil-native 118,863 14.0

(zsh isn't measured because zsh -n unexpectedly forks.)

Average Parsing Rate, Measured on Two Machines (lines/ms)

Shell startup time is included in the elapsed time measurements, but long files are chosen to minimize its effect.

shell label total lines host broome host lenny
dash 111,674 1,531.3 3,886.9
mksh 111,674 594.0 1,208.7
bash 118,863 352.0 639.0
oil-native 118,863 247.6 483.3
zsh 118,863 105.2 327.4
osh-cpython 118,863 7.3 9.9
osh-ovm 118,863 6.5 8.9

Breakdown By File

Instructions Per Line (in thousands)

bash dash mksh oil-native num lines filename
15.6 1.4 4.4 7.2 1,679 t4014-format-patch.sh
16.7 2.1 8.4 21.5 1,733 functions
23.7 NA NA 19.8 2,029 configure-helper.sh
17.9 2.2 8.9 22.6 2,512 abuild
13.6 1.8 5.9 16.0 2,698 Build.sh
13.7 1.0 3.3 4.6 3,123 t9300-fast-import.sh
32.0 NA NA 24.3 5,160 test-cmd-util.sh
18.2 1.8 6.1 14.8 9,661 ltmain.sh
17.3 1.9 6.4 15.3 20,489 configure
12.9 1.5 5.4 12.7 69,779 configure-coreutils

Elasped Time (milliseconds)

host label bash dash mksh zsh osh-ovm osh-cpython oil-native osh to bash ratio num lines filename
broome 9 2 4 98 125 160 7 0.9 1,679 t4014-format-patch.sh
broome 9 3 7 12 442 416 14 1.6 1,733 functions
broome 15 NA NA 22 491 418 18 1.2 2,029 configure-helper.sh
broome 15 4 11 15 655 560 22 1.5 2,512 abuild
broome 12 3 8 76 445 405 18 1.5 2,698 Build.sh
broome 11 2 5 116 172 161 8 0.7 3,123 t9300-fast-import.sh
broome 29 NA NA 28 1,355 1,205 35 1.2 5,160 test-cmd-util.sh
broome 32 9 23 40 1,533 1,374 40 1.2 9,661 ltmain.sh
broome 61 17 40 119 3,285 2,981 92 1.5 20,489 configure
broome 145 34 90 605 9,684 8,624 225 1.6 69,779 configure-coreutils
lenny 4 1 1 29 87 101 3 0.7 1,679 t4014-format-patch.sh
lenny 5 1 3 3 298 268 6 1.3 1,733 functions
lenny 7 NA NA 8 313 281 7 1.0 2,029 configure-helper.sh
lenny 6 1 3 5 450 419 9 1.5 2,512 abuild
lenny 5 1 3 27 329 291 7 1.4 2,698 Build.sh
lenny 5 1 2 31 96 94 3 0.5 3,123 t9300-fast-import.sh
lenny 16 NA NA 8 969 849 18 1.1 5,160 test-cmd-util.sh
lenny 18 3 10 13 1,119 989 22 1.2 9,661 ltmain.sh
lenny 37 6 20 42 2,454 2,171 47 1.3 20,489 configure
lenny 82 16 50 196 7,269 6,534 125 1.5 69,779 configure-coreutils

Parsing Rate (lines/ms)

host label bash dash mksh zsh osh-ovm osh-cpython oil-native num lines filename
broome 195.9 998.8 400.0 17.1 13.4 10.5 228.4 1,679 t4014-format-patch.sh
broome 190.7 650.0 247.4 149.7 3.9 4.2 122.7 1,733 functions
broome 136.4 NA NA 93.2 4.1 4.9 112.3 2,029 configure-helper.sh
broome 167.4 680.0 237.7 165.1 3.8 4.5 113.6 2,512 abuild
broome 219.2 843.1 344.6 35.4 6.1 6.7 146.4 2,698 Build.sh
broome 277.4 1,390.5 593.4 26.9 18.2 19.4 399.7 3,123 t9300-fast-import.sh
broome 178.3 NA NA 187.4 3.8 4.3 146.8 5,160 test-cmd-util.sh
broome 300.3 1,050.8 414.9 240.9 6.3 7.0 241.5 9,661 ltmain.sh
broome 337.8 1,240.6 510.2 172.8 6.2 6.9 221.7 20,489 configure
broome 482.0 2,068.6 777.9 115.4 7.2 8.1 310.6 69,779 configure-coreutils
lenny 421.4 2,784.4 1,224.7 57.3 19.3 16.6 618.6 1,679 t4014-format-patch.sh
lenny 369.1 2,003.5 685.0 581.5 5.8 6.5 291.5 1,733 functions
lenny 292.4 NA NA 270.5 6.5 7.2 298.3 2,029 configure-helper.sh
lenny 417.8 2,188.2 720.8 519.8 5.6 6.0 287.4 2,512 abuild
lenny 544.4 2,459.4 1,046.5 98.9 8.2 9.3 401.7 2,698 Build.sh
lenny 596.0 4,572.5 1,816.8 99.2 32.6 33.1 1,178.5 3,123 t9300-fast-import.sh
lenny 321.5 NA NA 607.3 5.3 6.1 284.5 5,160 test-cmd-util.sh
lenny 523.1 3,269.4 933.2 755.3 8.6 9.8 434.9 9,661 ltmain.sh
lenny 551.0 3,498.2 1,018.7 483.3 8.3 9.4 433.4 20,489 configure
lenny 845.9 4,495.2 1,388.8 356.0 9.6 10.7 559.3 69,779 configure-coreutils

Memory Usage (Max Resident Set Size in MB)

Again, Oil uses a different algorithm (and language) than POSIX shells. It builds an AST in memory rather than just validating the code line-by-line.

host label bash dash mksh zsh osh-ovm osh-cpython oil-native num lines filename
broome 3.4 1.7 1.9 4.1 14.6 18.4 5.5 1,679 t4014-format-patch.sh
broome 3.4 1.8 1.9 4.0 17.2 21.0 8.1 1,733 functions
broome 3.4 NA NA 4.1 17.9 21.6 8.7 2,029 configure-helper.sh
broome 3.5 1.8 1.8 4.1 19.4 23.6 10.5 2,512 abuild
broome 3.3 1.8 1.9 4.0 18.1 22.1 9.1 2,698 Build.sh
broome 3.3 1.7 2.0 4.1 15.0 18.2 5.7 3,123 t9300-fast-import.sh
broome 3.5 NA NA 4.1 28.3 33.4 18.3 5,160 test-cmd-util.sh
broome 4.3 2.2 3.0 4.6 29.9 35.5 21.1 9,661 ltmain.sh
broome 4.0 2.5 2.6 4.3 50.2 58.7 41.9 20,489 configure
broome 3.8 1.8 2.2 4.0 112.7 130.8 111.1 69,779 configure-coreutils
lenny 3.4 1.7 1.9 4.0 14.4 18.2 5.6 1,679 t4014-format-patch.sh
lenny 3.3 1.7 1.9 3.9 17.2 21.0 8.2 1,733 functions
lenny 3.6 NA NA 4.0 17.8 21.8 8.7 2,029 configure-helper.sh
lenny 3.6 1.7 1.9 4.0 19.4 23.5 10.7 2,512 abuild
lenny 3.5 1.8 2.0 4.1 17.9 21.9 9.1 2,698 Build.sh
lenny 3.5 1.7 1.8 4.1 14.9 18.3 5.6 3,123 t9300-fast-import.sh
lenny 3.6 NA NA 4.1 28.3 33.4 18.3 5,160 test-cmd-util.sh
lenny 4.4 2.2 3.0 4.5 29.9 35.4 21.4 9,661 ltmain.sh
lenny 4.0 2.5 2.6 4.4 50.1 58.8 42.1 20,489 configure
lenny 3.8 1.9 2.1 4.0 112.8 130.8 110.9 69,779 configure-coreutils

Shell and Host Details

shell label shell id
bash bash-9af8f89f
dash dash-9ff48631
mksh mksh-e3755400
oil-native osh_eval.stripped-0d75ac5a
host label host id
broome broome-8a85d07d
lenny lenny-23f35694

Raw Data

filename
no-host.2022-08-03__12-42-03.cachegrind.tsv
broome.2022-08-03__16-29-11.times.csv
lenny.2022-08-03__12-42-49.times.csv