Review of Roadmap #3
Yesterday I glossed over a dozen or so blog posts, clearing some
mental space for coding.
I want to write a Roadmap #4 to describe what I expect to implement in
the next few months. But let's first review November's
Roadmap #3. In that post, I stated two priorities:
- Test the Oil Language Design by automatically translating real
programs to it.
- Fill out the Shell Runtime. I talked about publishing the
spec test results as HTML and accepting code contributions.
I stated a third priority in this status update:
- Serialize the AST to a Binary Format for execution in C++.
I went into even more detail in Roadmap #3 Continued,
listing tasks in these categories:
- Ongoing Work
- Short-Term Plans
- Medium-Term Plans
- Long-Term Plans
How Did We Do?
Surprisingly, I've mostly followed the plan. The biggest divergence is that
I thought I would translate OSH to Oil by simple printing the OSH AST used
for execution to source code.
That didn't happen because it was naive — we need multiple
representations. I developed ASDL to specify these representations, and
changed the AST to an LST.
Here's what I did with respect to each of the top priorities:
- I wrote code to automatically translate OSH to Oil. Although it's only
about 25% done, I think the approach is sound. I showed translations of
real programs in Translating Shell to Oil (part two).
- In the last few days, I wrote code to publish the spec tests
as HTML. This will make it easier for others to help me finish the shell
runtime in Python. I'll write about it shortly.
- I implemented and tested the OHeap file format and the associated
C++ code generator. This allows us to write a tree-walking interpreter in
Now let's review the detailed task list:
1. Ongoing Work
This work is straightforward but not urgent, so not much happened:
- I didn't polish the error messages, but the Lossless Syntax Tree
refined the data structures for tracking source code locations, which
are required for good error messages.
- I didn't implement many remaining shell features. As mentioned, I hope the
spec tests will allow this work to be parallelized.
- My writing has been directed at the blog and e-mail correpondence, rather
than publishing Oil documentation.
2. Short Term Work
- As mentioned at the end of ASDL Implementation and Use
Cases, I abandoned type checking with MyPy. MyPy
conflicts with metaprogramming, and the
stronger (but dynamic) type checking of ASDL turned out to be a good
- I implemented pretty printing to text, ANSI color, and
HTML, with a full format and an abbreviated format. I'm pleased with
results and expect them to pay dividends when working with the OSH LST,
Oil LST, and OVM.
- Testing tools: I will publish the spec tests, but I need to do
the same thing with the "wild tests", which test the OSH parser on
source code found in the wild.
- Globbing: Not done. (I single this out because it has some architectural
3. Medium Term Work.
I sketched different strategies for porting various components to C++, e.g.
using re2c to implement the lexer.
I haven't yet done that, but I did implement a C++ code generator for
core/id_kind.py as part of the diversion on Bespoke
And, as mentioned, the C++ API for OHeap is tested and ready to use.
4. Long Term Work
This part is unchanged. I still want to subsume Awk, susbume Make, and
implement an interactive shell — but I want to write this code in
Oil. Oil is a shell, but it will also be a programming language like
Python or ML.
So these tasks are blocking on the Oil parser and interpreter.
5. Even Longer Term
I wrote about some even loftier goals in this blog
post. I don't want to talk too much about work that's years
in the future, but I think it's important to have a vision for the design.
Oil is about respecting the past, but not calcifying it!
In particular, the shell is naturally related to distributed computing.