blog | oilshell.org

What's Happened Since February?

2019-06-13

I haven't published a blog post in nearly four months, but the Oil project is still alive!

I've been busy coding, trying to push out a polished 0.6.0 release. I thought that would happen soon after I announced that OSH can run thousands of lines of interactive shell programs.

But it turned out that there were a lot of features, fixes, and polish necessary on top of that, and I was also derailed by a performance problem.

I'll review most of that work in this post.

I've been struggling with the fact that making a usable shell is a lot of work. One way to think about it is that bash is a 30-year-old program consisting of 140K lines of C code, and Oil has to replicate most of that functionality [1].

On the positive side, early users have filed high-quality bug reports, which have kept me busy. I need more help like this! Read on for details.

Table of Contents
Seven Pre-Releases
0.6.pre16 on March 3rd
0.6.pre17 on March 21st
0.6.pre18 on April 20th
0.6.pre19 on May 9th
0.6.pre20 on May 14th
0.6.pre21 on June 4th
0.6.pre22 on June 11th
Contributors
Updated Docs and New Wiki Pages
How to Help
Bug Labels
What's Next?
Conclusion
Notes
Appendix: Blog Post Drafts

Seven Pre-Releases

Since announcing the 0.6.pre15 release in February, I've made 7 more pre-releases. Here I mention the general areas changed in each release, and link to the git log for those who want details.

0.6.pre16 on March 3rd

I worked on more aspects of the interactive shell, e.g.

0.6.pre17 on March 21st

I statically typed the whole OSH front end with MyPy, which is around 10K lines of code. This work had four parts:

Josh Nelson implemented:

0.6.pre18 on April 20th

As mentioned in February, the shell is too slow, and I'm eager to fix that problem.

During this release period, I prototyped "mycpp", a program that walks MyPy's typed AST and emits C++ code. This is one possible way to use types to speed up the OSH interpreter.

It can translate many of the programs in the mycpp/examples/ dir accurately, and there are significant speedups in most cases.

I haven't yet applied mycpp to Oil's source, but I plan to shortly.

Note: I mentioned OVM2 last fall, but consider it dormant for now. I intend to take the shortest path to speed up the shell, and leveraging MyPy and a C++ compiler should get us there sooner than trying to write my own VM.

0.6.pre19 on May 9th

I made a pass over the whole codebase, checking that every error message is informative and has location information. These tests try to tickle every error:

Good news: OSH objectively has the most precise error messages of any shell. For example:

# opt=typo
set -o $opt
       ^~~~
foo.sh:2: 'set' got invalid option 'typo'

I'll show more demos of this with the 0.6.0 release. I decided that the headline for that release will be OSH Has Useful Error Messages.

Bad news: After this release, I noticed a performance regression running Python's configure. After some debugging, it turns out that there was no regression. Instead, there had been a bug in the benchmark for months.

What changed? This release fixed the behavior of the $LINENO variable. Apparently, autoconf generates code to detect whether $LINENO works. When it didn't work in OSH, the configure script exec'd itself with /bin/sh, so it ran more quickly.

Fixing $LINENO means that OSH now runs the whole configure script, which reveals the slowness. :-( I hope that the mycpp work will fix this problem.

0.6.pre20 on May 14th

0.6.pre21 on June 4th

More on that later.

0.6.pre22 on June 11th

Contributors

Thanks again to the following people, mentioned above:

Please continue to test OSH and file bugs!

Updated Docs and New Wiki Pages

Oil's documentation is currently sparse, but I'm not losing track of important points:

Designs on the Wiki:

Developer tips:

How to Help

I think Oil need more developers, but people are unlikely to become developers unless they're also users.

I've been suggesting that potential developers install OSH (which takes 30 seconds), and then:

If you're already working with the code, and have problems figuring it out, please ping me on Zulip.

Bug Labels

I'm making consistent use of Github's issue tracker.

The #good-first-issue and #help-wanted labels generally up-to-date. Again, feel free to ping me on oilshell.zulipchat.com for ideas.

Other labels:

What's Next?

Conclusion

I went over many topics quickly in this post. If you're interested in more detail, leave a comment or leave a message on #oil-discuss in Zulip.

Notes

[1] Wikipedia says that the first release of bash was on June 8th, 1989. So its 30th birthday was a few days ago. Happy Birthday bash!

Appendix: Blog Post Drafts

Although I plan to publish the FAQ first, I have drafts of the following blog posts:

If you have experience implementing garbage collection and Python-like data structures, I'd love to chat!