Review of Roadmap #3


Yesterday I glossed over a dozen or so blog posts, clearing some mental space for coding.

I want to write a Roadmap #4 to describe what I expect to implement in the next few months. But let's first review November's Roadmap #3. In that post, I stated two priorities:

  1. Test the Oil Language Design by automatically translating real programs to it.
  2. Fill out the Shell Runtime. I talked about publishing the spec test results as HTML and accepting code contributions.

I stated a third priority in this status update:

  1. Serialize the AST to a Binary Format for execution in C++.

I went into even more detail in Roadmap #3 Continued, listing tasks in these categories:

  1. Ongoing Work
  2. Short-Term Plans
  3. Medium-Term Plans
  4. Long-Term Plans

How Did We Do?

Surprisingly, I've mostly followed the plan. The biggest divergence is that I thought I would translate OSH to Oil by simple printing the OSH AST used for execution to source code.

That didn't happen because it was naive — we need multiple representations. I developed ASDL to specify these representations, and changed the AST to an LST.

Here's what I did with respect to each of the top priorities:

  1. I wrote code to automatically translate OSH to Oil. Although it's only about 25% done, I think the approach is sound. I showed translations of real programs in Translating Shell to Oil (part two).
  2. In the last few days, I wrote code to publish the spec tests as HTML. This will make it easier for others to help me finish the shell runtime in Python. I'll write about it shortly.
  3. I implemented and tested the OHeap file format and the associated C++ code generator. This allows us to write a tree-walking interpreter in C++.

Now let's review the detailed task list:

1. Ongoing Work

This work is straightforward but not urgent, so not much happened:

2. Short Term Work

Mostly done:

3. Medium Term Work.

I sketched different strategies for porting various components to C++, e.g. using re2c to implement the lexer.

I haven't yet done that, but I did implement a C++ code generator for core/id_kind.py as part of the diversion on Bespoke Superoptimization.

And, as mentioned, the C++ API for OHeap is tested and ready to use.

4. Long Term Work

This part is unchanged. I still want to subsume Awk, susbume Make, and implement an interactive shell — but I want to write this code in Oil. Oil is a shell, but it will also be a programming language like Python or ML.

So these tasks are blocking on the Oil parser and interpreter.

5. Even Longer Term

I wrote about some even loftier goals in this blog post. I don't want to talk too much about work that's years in the future, but I think it's important to have a vision for the design. Oil is about respecting the past, but not calcifying it!

In particular, the shell is naturally related to distributed computing.