OSH Parser Performance

We time $sh -n $file for various files under various shells, and repeat then run under cachegrind for stable metrics.

Summary

Instructions Per Line (via cachegrind)

Lower numbers are generally better, but each shell recognizes a different language, and Oil uses a more thorough parsing algorithm. In thousands of "I refs".

shell label total lines thousand irefs per line
bash 118,863 15.3
dash 111,674 1.6
mksh 111,674 5.7
oil-native 118,863 8.6

(zsh isn't measured because zsh -n unexpectedly forks.)

Average Parsing Rate, Measured on Two Machines (lines/ms)

Shell startup time is included in the elapsed time measurements, but long files are chosen to minimize its effect.

shell label total lines host broome host spring
dash 111,674 1,491.2 1,576.6
mksh 111,674 585.8 598.5
bash 118,863 350.9 331.7
oil-native 118,863 333.6 311.4
zsh 118,863 112.4 112.2
osh-cpython 118,863 6.4 6.3
osh-ovm 118,863 6.0 5.8

Breakdown By File

Instructions Per Line (in thousands)

bash dash mksh oil-native num lines filename
15.6 1.3 4.4 4.8 1,679 t4014-format-patch.sh
16.7 2.1 8.4 13.5 1,733 functions
23.7 NA NA 12.2 2,029 configure-helper.sh
17.8 2.1 8.9 14.1 2,512 abuild
13.6 1.8 5.9 9.7 2,698 Build.sh
13.7 1.0 3.3 3.1 3,123 t9300-fast-import.sh
32.0 NA NA 15.0 5,160 test-cmd-util.sh
18.2 1.8 6.1 9.2 9,661 ltmain.sh
17.4 1.9 6.4 9.3 20,489 configure
12.9 1.5 5.4 7.8 69,779 configure-coreutils

Elasped Time (milliseconds)

host label bash dash mksh zsh osh-ovm osh-cpython oil-native osh to bash ratio num lines filename
broome 8 2 4 91 138 165 6 0.8 1,679 t4014-format-patch.sh
broome 10 3 7 10 461 441 14 1.4 1,733 functions
broome 14 NA NA 19 489 484 15 1.1 2,029 configure-helper.sh
broome 14 4 11 17 703 662 18 1.3 2,512 abuild
broome 12 3 8 71 518 465 14 1.2 2,698 Build.sh
broome 12 2 5 104 152 164 7 0.6 3,123 t9300-fast-import.sh
broome 31 NA NA 25 1,474 1,323 31 1.0 5,160 test-cmd-util.sh
broome 35 9 23 38 1,747 1,550 29 0.8 9,661 ltmain.sh
broome 60 17 37 117 3,626 3,348 67 1.1 20,489 configure
broome 142 35 95 565 10,526 9,834 154 1.1 69,779 configure-coreutils
spring 9 2 4 87 148 164 6 0.7 1,679 t4014-format-patch.sh
spring 9 3 7 12 475 459 12 1.3 1,733 functions
spring 13 NA NA 20 500 478 10 0.7 2,029 configure-helper.sh
spring 13 4 11 15 771 673 17 1.3 2,512 abuild
spring 13 3 8 66 530 508 14 1.1 2,698 Build.sh
spring 13 2 6 99 178 154 7 0.5 3,123 t9300-fast-import.sh
spring 31 NA NA 25 1,535 1,401 29 0.9 5,160 test-cmd-util.sh
spring 34 9 21 39 1,733 1,677 35 1.0 9,661 ltmain.sh
spring 61 16 38 115 3,750 3,542 71 1.2 20,489 configure
spring 161 32 92 582 10,933 9,902 180 1.1 69,779 configure-coreutils

Parsing Rate (lines/ms)

host label bash dash mksh zsh osh-ovm osh-cpython oil-native num lines filename
broome 206.6 882.8 405.0 18.4 12.1 10.2 263.8 1,679 t4014-format-patch.sh
broome 172.2 664.2 240.8 166.3 3.8 3.9 121.1 1,733 functions
broome 144.7 NA NA 105.0 4.2 4.2 133.2 2,029 configure-helper.sh
broome 179.9 685.2 238.4 148.4 3.6 3.8 136.4 2,512 abuild
broome 222.5 850.3 351.6 38.0 5.2 5.8 186.4 2,698 Build.sh
broome 253.5 1,432.6 592.5 30.0 20.5 19.0 460.9 3,123 t9300-fast-import.sh
broome 164.9 NA NA 206.9 3.5 3.9 167.8 5,160 test-cmd-util.sh
broome 276.4 1,070.7 416.0 257.2 5.5 6.2 330.4 9,661 ltmain.sh
broome 342.5 1,183.4 552.3 175.3 5.7 6.1 306.4 20,489 configure
broome 491.3 1,992.5 730.7 123.5 6.6 7.1 453.5 69,779 configure-coreutils
spring 179.4 896.4 386.1 19.3 11.3 10.2 270.2 1,679 t4014-format-patch.sh
spring 185.0 633.4 240.2 140.5 3.7 3.8 140.3 1,733 functions
spring 154.3 NA NA 102.4 4.1 4.2 211.9 2,029 configure-helper.sh
spring 189.6 692.8 238.4 169.6 3.3 3.7 150.8 2,512 abuild
spring 215.1 853.3 349.0 40.6 5.1 5.3 188.0 2,698 Build.sh
spring 236.9 1,441.8 562.5 31.6 17.5 20.3 469.1 3,123 t9300-fast-import.sh
spring 164.7 NA NA 207.3 3.4 3.7 176.8 5,160 test-cmd-util.sh
spring 284.8 1,114.6 468.8 249.9 5.6 5.8 272.4 9,661 ltmain.sh
spring 334.0 1,245.6 537.6 178.3 5.5 5.8 289.4 20,489 configure
spring 433.6 2,170.4 754.4 120.0 6.4 7.0 386.8 69,779 configure-coreutils

Memory Usage (Max Resident Set Size in MB)

Again, Oil uses a different algorithm (and language) than POSIX shells. It builds an AST in memory rather than just validating the code line-by-line.

host label bash dash mksh zsh osh-ovm osh-cpython oil-native num lines filename
broome 3.4 1.8 1.9 4.0 14.1 17.9 5.7 1,679 t4014-format-patch.sh
broome 3.4 1.7 1.9 4.0 16.9 20.8 7.9 1,733 functions
broome 3.3 NA NA 4.1 17.5 21.4 8.3 2,029 configure-helper.sh
broome 3.5 1.7 2.0 4.0 19.0 23.1 9.8 2,512 abuild
broome 3.4 1.8 2.0 4.1 17.7 21.6 8.6 2,698 Build.sh
broome 3.4 1.7 1.9 4.1 14.5 18.1 5.8 3,123 t9300-fast-import.sh
broome 3.6 NA NA 4.1 27.7 33.2 17.5 5,160 test-cmd-util.sh
broome 4.2 2.1 2.9 4.5 29.6 34.9 19.8 9,661 ltmain.sh
broome 4.0 2.4 2.7 4.5 49.7 58.2 38.2 20,489 configure
broome 3.7 1.8 2.1 4.1 112.4 130.0 103.0 69,779 configure-coreutils
spring 3.5 1.7 1.9 4.1 14.1 17.6 5.6 1,679 t4014-format-patch.sh
spring 3.4 1.8 1.9 4.1 17.0 20.8 7.8 1,733 functions
spring 3.4 NA NA 4.1 17.4 21.4 8.4 2,029 configure-helper.sh
spring 3.4 1.7 1.8 4.0 19.1 23.1 10.0 2,512 abuild
spring 3.6 1.7 2.0 4.1 17.7 21.7 8.7 2,698 Build.sh
spring 3.3 1.7 1.8 4.1 14.4 18.2 6.0 3,123 t9300-fast-import.sh
spring 3.6 NA NA 4.1 27.9 33.3 17.6 5,160 test-cmd-util.sh
spring 4.4 2.1 3.0 4.5 29.3 35.0 20.0 9,661 ltmain.sh
spring 3.9 2.5 2.6 4.4 49.7 58.1 38.2 20,489 configure
spring 3.7 1.9 2.1 4.2 112.3 130.1 103.1 69,779 configure-coreutils

Shell and Host Details

shell label shell id
bash bash-9af8f89f
dash dash-9ff48631
mksh mksh-e3755400
oil-native osh_eval.opt.stripped-145143a1
host label host id
broome broome-d80ab850
spring spring-d80ab850

Raw Data

filename
no-host.2021-10-04__22-26-57.cachegrind.tsv
broome.2021-10-05__02-19-49.times.csv
spring.2021-10-04__22-28-04.times.csv