OSH Parser Performance

We time $sh -n $file for various files under various shells, and repeat then run under cachegrind for stable metrics.

Summary

Instructions Per Line (via cachegrind)

Lower numbers are generally better, but each shell recognizes a different language, and Oil uses a more thorough parsing algorithm. In thousands of "I refs".

shell label total lines thousand irefs per line
bash 118,863 15.3
dash 111,674 1.6
mksh 111,674 5.7
oil-native 118,863 8.7

(zsh isn't measured because zsh -n unexpectedly forks.)

Average Parsing Rate, Measured on Two Machines (lines/ms)

Shell startup time is included in the elapsed time measurements, but long files are chosen to minimize its effect.

shell label total lines host broome host spring
dash 111,674 1,522.2 1,515.5
mksh 111,674 594.4 583.7
bash 118,863 339.1 359.6
oil-native 118,863 328.0 315.6
zsh 118,863 113.3 111.1
osh-cpython 118,863 6.6 6.3
osh-ovm 118,863 5.9 6.0

Breakdown By File

Instructions Per Line (in thousands)

bash dash mksh oil-native num lines filename
15.6 1.3 4.4 4.8 1,679 t4014-format-patch.sh
16.7 2.1 8.4 13.5 1,733 functions
23.7 NA NA 12.2 2,029 configure-helper.sh
17.8 2.1 8.9 14.1 2,512 abuild
13.6 1.8 5.9 9.8 2,698 Build.sh
13.7 1.0 3.3 3.1 3,123 t9300-fast-import.sh
32.0 NA NA 15.1 5,160 test-cmd-util.sh
18.2 1.8 6.1 9.3 9,661 ltmain.sh
17.4 1.9 6.4 9.3 20,489 configure
12.9 1.5 5.4 7.8 69,779 configure-coreutils

Elasped Time (milliseconds)

host label bash dash mksh zsh osh-ovm osh-cpython oil-native osh to bash ratio num lines filename
broome 8 2 3 90 154 152 6 0.7 1,679 t4014-format-patch.sh
broome 10 3 7 10 482 433 12 1.2 1,733 functions
broome 12 NA NA 19 492 444 12 1.0 2,029 configure-helper.sh
broome 13 4 11 16 716 626 18 1.4 2,512 abuild
broome 12 3 8 71 518 458 15 1.2 2,698 Build.sh
broome 13 2 5 99 179 153 7 0.5 3,123 t9300-fast-import.sh
broome 31 NA NA 25 1,473 1,365 28 0.9 5,160 test-cmd-util.sh
broome 34 9 22 40 1,711 1,527 32 0.9 9,661 ltmain.sh
broome 62 16 38 114 3,674 3,273 58 0.9 20,489 configure
broome 155 35 94 565 10,649 9,526 174 1.1 69,779 configure-coreutils
spring 6 2 4 89 158 165 6 1.1 1,679 t4014-format-patch.sh
spring 10 3 7 11 494 461 15 1.5 1,733 functions
spring 15 NA NA 18 531 496 14 0.9 2,029 configure-helper.sh
spring 14 3 11 17 712 669 16 1.2 2,512 abuild
spring 12 3 8 69 525 511 14 1.2 2,698 Build.sh
spring 11 2 5 103 172 171 7 0.6 3,123 t9300-fast-import.sh
spring 31 NA NA 26 1,478 1,434 31 1.0 5,160 test-cmd-util.sh
spring 35 7 22 43 1,703 1,556 38 1.1 9,661 ltmain.sh
spring 59 18 39 115 3,508 3,389 64 1.1 20,489 configure
spring 139 36 95 579 10,540 9,971 172 1.2 69,779 configure-coreutils

Parsing Rate (lines/ms)

host label bash dash mksh zsh osh-ovm osh-cpython oil-native num lines filename
broome 199.5 888.4 534.7 18.7 10.9 11.0 266.1 1,679 t4014-format-patch.sh
broome 169.8 647.9 241.9 167.2 3.6 4.0 142.7 1,733 functions
broome 169.6 NA NA 107.0 4.1 4.6 171.2 2,029 configure-helper.sh
broome 195.7 699.1 227.0 153.2 3.5 4.0 142.5 2,512 abuild
broome 220.1 873.1 349.6 38.1 5.2 5.9 184.4 2,698 Build.sh
broome 249.3 1,473.8 588.0 31.6 17.5 20.4 454.4 3,123 t9300-fast-import.sh
broome 163.9 NA NA 202.6 3.5 3.8 185.2 5,160 test-cmd-util.sh
broome 281.7 1,038.1 442.0 243.8 5.6 6.3 297.7 9,661 ltmain.sh
broome 332.9 1,278.4 543.0 179.9 5.6 6.3 350.5 20,489 configure
broome 450.2 2,013.1 743.2 123.5 6.6 7.3 400.7 69,779 configure-coreutils
spring 296.9 944.9 436.1 18.9 10.6 10.2 282.2 1,679 t4014-format-patch.sh
spring 170.5 642.6 236.6 163.3 3.5 3.8 115.3 1,733 functions
spring 137.7 NA NA 112.7 3.8 4.1 147.0 2,029 configure-helper.sh
spring 183.2 767.0 232.0 145.9 3.5 3.8 158.5 2,512 abuild
spring 234.3 846.6 326.2 39.2 5.1 5.3 191.3 2,698 Build.sh
spring 283.4 1,439.2 594.9 30.4 18.1 18.2 467.6 3,123 t9300-fast-import.sh
spring 167.5 NA NA 196.3 3.5 3.6 166.0 5,160 test-cmd-util.sh
spring 279.9 1,397.3 433.1 224.1 5.7 6.2 257.0 9,661 ltmain.sh
spring 346.7 1,133.4 531.4 177.9 5.8 6.0 319.9 20,489 configure
spring 500.8 1,960.5 734.9 120.5 6.6 7.0 404.5 69,779 configure-coreutils

Memory Usage (Max Resident Set Size in MB)

Again, Oil uses a different algorithm (and language) than POSIX shells. It builds an AST in memory rather than just validating the code line-by-line.

host label bash dash mksh zsh osh-ovm osh-cpython oil-native num lines filename
broome 3.4 1.7 1.9 4.1 14.2 17.7 5.5 1,679 t4014-format-patch.sh
broome 3.3 1.7 1.8 3.9 17.2 20.7 8.0 1,733 functions
broome 3.4 NA NA 4.1 17.4 21.4 8.2 2,029 configure-helper.sh
broome 3.4 1.7 1.8 4.1 19.2 23.2 10.0 2,512 abuild
broome 3.5 1.7 2.0 4.1 17.9 21.9 8.7 2,698 Build.sh
broome 3.4 1.7 1.9 4.1 14.7 18.2 6.0 3,123 t9300-fast-import.sh
broome 3.6 NA NA 4.1 27.9 33.3 17.7 5,160 test-cmd-util.sh
broome 4.4 2.2 3.0 4.5 29.5 35.1 19.9 9,661 ltmain.sh
broome 4.1 2.6 2.5 4.4 49.8 58.3 38.3 20,489 configure
broome 3.7 1.8 2.2 4.0 112.5 130.4 103.0 69,779 configure-coreutils
spring 3.3 1.7 1.8 4.1 14.4 17.9 5.6 1,679 t4014-format-patch.sh
spring 3.3 1.8 2.0 4.1 17.2 20.7 7.8 1,733 functions
spring 3.4 NA NA 4.1 17.7 21.5 8.2 2,029 configure-helper.sh
spring 3.4 1.8 1.9 4.1 19.1 23.2 9.9 2,512 abuild
spring 3.4 1.8 1.9 4.0 18.0 21.7 8.7 2,698 Build.sh
spring 3.3 1.7 2.0 4.0 14.7 18.3 5.7 3,123 t9300-fast-import.sh
spring 3.5 NA NA 4.0 27.9 33.4 17.6 5,160 test-cmd-util.sh
spring 4.3 2.1 3.0 4.5 29.4 35.1 19.8 9,661 ltmain.sh
spring 4.0 2.4 2.7 4.4 49.7 58.3 38.2 20,489 configure
spring 3.7 1.9 2.3 4.1 112.4 130.4 103.2 69,779 configure-coreutils

Shell and Host Details

shell label shell id
bash bash-9af8f89f
dash dash-9ff48631
mksh mksh-e3755400
oil-native osh_eval.opt.stripped-e36eda9a
host label host id
broome broome-d80ab850
spring spring-d80ab850

Raw Data

filename
no-host.2021-11-29__01-40-27.cachegrind.tsv
broome.2021-11-29__07-08-59.times.csv
spring.2021-11-29__01-41-35.times.csv