blog | oilshell.org

Summer Status Update

I plan to release OSH 0.5 next week! This post is an update on what's happened since the last release.

Recurse Center

In late March, I wrote a post to relieve myself of Oil-related "duties" while attending the Recurse Center.

But a few things happened in the meantime, and I ended up working on Oil a lot:

• Guido van Rossum tweeted about my project [1], which attracted new contributors. I kept my promise to review their pull requests quickly.

  I'm embarrassed that these commits have been sitting unreleased in the repo since May, but I will fix this shortly.

• Pair programming is a core part of the Recurse Center experience. It turned out that many people were interested in pairing on Oil. You'll see this in the release notes.

• After the handling the logistics of traveling and housing, it felt good to work on Oil, rather than something more speculative and unfamilar.

Oil Subproject Review

In the same post on Recurse Center (abbreviated RC), I wrote about these subprojects:

• A shell trace tool, i.e. a better sh -x.
• Static analysis of shell scripts, e.g. for deploying self-contained app bundles.
• The OPy compiler.
• A source code browser.

I've decided to focus on two of these projects. They address the main problem I see: that OSH is basically a slower and less featureful version of bash right now!

While I've built a solid foundation for the project, I'm not surprised that it doesn't have regular users. I want to change that in the coming months.

Shell Trace Tool

This is a "carrot" for OSH, or at least a candidate for one. In other words, I want OSH to do something that bash doesn't do.

(The Oil language is still the main "carrot", but it's a longer-term design and implementation effort.)

So, before leaving for New York, I prototyped a web-based shell trace tool with my friend Eric.

I conjectured that we could use bash as the client, rather than Oil. I came up with some horrible hacks involving the $PS4 variable, which controls the output of sh -x.

We have a working demo, but I'm not sure how successful this experiment was. When I get back to San Francisco, I plan to release it and solicit feedback.

OPy

The OPy compiler is necessary to turn OSH into a "production quality" shell. It's a risky and unusual implementation strategy, but I'm optimistic based on the progress so far.

I haven't worked on it too much this summer, but I had some realizations based on conversations with Darius Bacon, who wrote one of the articles I mentioned here:

• Compiling to C code is probably easier than compiling to bytecode and writing my own VM. It's more direct and will solve the "double interpretation" problem.
• There are different ways to compile a subset of Python to C code. PyPy compiles a control flow graph to C, but higher-level translations are also possible.
• There's a lot of work in common between the C code strategy and the bytecode strategy, so I don't have to make this decision right now. For example, I want to remove the BREAK_LOOP and CONTINUE_LOOP opcodes and express them in terms of jumps. This change will be made in Python 3.8, and I plan to reproduce it in OPy (credit for bringing this to my attention).

Next

The next post will announce OSH 0.5, a significant release with many new contributors!

Footnote

[1] I e-mailed Guido about his recent blog post on pgen, the bespoke parser generator that powers Python. (Oil uses the Python version, pgen2).

Regular readers know that I'm interested in #parsing — specifically the difference between theory and practice. There are a bunch of interesting issues in Python's parser that I'd like to write about.