Why Sponsor Oils? | blog | oilshell.org

Review of Roadmap #3

Yesterday I glossed over a dozen or so blog posts, clearing some mental space for coding.

I want to write a Roadmap #4 to describe what I expect to implement in the next few months. But let's first review November's Roadmap #3. In that post, I stated two priorities:

1. Test the Oil Language Design by automatically translating real programs to it.
2. Fill out the Shell Runtime. I talked about publishing the spec test results as HTML and accepting code contributions.

I stated a third priority in this status update:

3. Serialize the AST to a Binary Format for execution in C++.

I went into even more detail in Roadmap #3 Continued, listing tasks in these categories:

1. Ongoing Work
2. Short-Term Plans
3. Medium-Term Plans
4. Long-Term Plans

How Did We Do?

Surprisingly, I've mostly followed the plan. The biggest divergence is that I thought I would translate OSH to Oil by simple printing the OSH AST used for execution to source code.

That didn't happen because it was naive — we need multiple representations. I developed ASDL to specify these representations, and changed the AST to an LST.

Here's what I did with respect to each of the top priorities:

1. I wrote code to automatically translate OSH to Oil. Although it's only about 25% done, I think the approach is sound. I showed translations of real programs in Translating Shell to Oil (part two).
2. In the last few days, I wrote code to publish the spec tests as HTML. This will make it easier for others to help me finish the shell runtime in Python. I'll write about it shortly.
3. I implemented and tested the OHeap file format and the associated C++ code generator. This allows us to write a tree-walking interpreter in C++.

Now let's review the detailed task list:

1. Ongoing Work

This work is straightforward but not urgent, so not much happened:

• I didn't polish the error messages, but the Lossless Syntax Tree refined the data structures for tracking source code locations, which are required for good error messages.
• I didn't implement many remaining shell features. As mentioned, I hope the spec tests will allow this work to be parallelized.
• My writing has been directed at the blog and e-mail correspondence, rather than publishing Oil documentation.

2. Short Term Work

Mostly done:

• As mentioned at the end of ASDL Implementation and Use Cases, I abandoned type checking with MyPy. MyPy conflicts with metaprogramming, and the stronger (but dynamic) type checking of ASDL turned out to be a good replacement.
• I implemented pretty printing to text, ANSI color, and HTML, with a full format and an abbreviated format. I'm pleased with results and expect them to pay dividends when working with the OSH LST, Oil LST, and OVM.
• Testing tools: I will publish the spec tests, but I need to do the same thing with the "wild tests", which test the OSH parser on source code found in the wild.
• Globbing: Not done. (I single this out because it has some architectural implications.)

3. Medium Term Work.

I sketched different strategies for porting various components to C++, e.g. using re2c to implement the lexer.

I haven't yet done that, but I did implement a C++ code generator for core/id_kind.py as part of the diversion on Bespoke Superoptimization.

And, as mentioned, the C++ API for OHeap is tested and ready to use.

4. Long Term Work

This part is unchanged. I still want to subsume Awk, subsume Make, and implement an interactive shell — but I want to write this code in Oil. Oil is a shell, but it will also be a programming language like Python or ML.

So these tasks are blocking on the Oil parser and interpreter.

5. Even Longer Term

I wrote about some even loftier goals in this blog post. I don't want to talk too much about work that's years in the future, but I think it's important to have a vision for the design. Oil is about respecting the past, but not calcifying it!

In particular, the shell is naturally related to distributed computing.