blog |

Roadmap #3: Two Top Priorities


Yesterday I wrote a retrospective, and today I will sketch a roadmap for the next few months.

I accomplished most of the things on the last roadmap: I fixed up both the ${} parser and arithmetic parser, which allowed me to parse most of git. I wrote about about weird shell syntax.

I didn't repost messages from Reddit or Hacker News, mostly because I edited them into higher quality blog entries. (For example, this post should be turned into a blog entry or two.)

Test the Oil Language Design

Now that I've laboriously parsed real shell scripts into a clean AST, I have a fun project ahead of me: print the AST into the new oil language.

I've designed the language with a long document, and now I can test it out by automatically porting real programs. Here's a good test for the language: the converted programs should be immediately readable for expert shell users, but also approachable for non-shell programmers.

I'm thinking of the Python/Ruby/Perl/JavaScript programmers who shy away from shell due to its prickly nature, preferring to write verbose "shell scripts" in those languages. Or people who think its a "simpler" to embed shell code strings in JSON or other config files.

Fill Out the Shell Runtime

The current code does a lot already: external commands/builtins/functions, pipelines, subshells, redirects, command sub, here docs, variables, etc. But there is plenty more to do — the runtime is much less complete than the parser.

Both the osh and oil languages will share the same runtime — they should remain purely front ends. There will need to be compatibility flags, but that shouldn't be a big deal.

Here is the plan:

  1. Publish the "spec" test results as HTML. There are 37 tests/* files totalling 3500 lines in the oil repository. They thoroughly test corner cases in shell; hence most are currently failing.
  2. Document the test framework so people can write more tests.
  3. Accept contributions that make existing tests pass. This will involve patches to the Python code in core/*_{eval,exec}.py.
  4. Also accept contributions that add new tests, to document a feature's behavior. Making the tests actually pass is preferred, but not required!

Currently there is no C++ skeleton, but I want to work on that soon. However, I do think it makes sense to get tests passing with a Python implementation, even if it will all be ported C++ eventually.

Maybe it's just me, but it seems to make things go faster overall. Implementing a shell is software archaeology, and iteration speed counts.

The runtime should be compatible with bash (which has its own compatibility flags like set -o posix, although I'm testing dash, mksh, and zsh and considering their behavior as well.

If you're interested in contributing, please send mail to the list (or to

I will end today's post here, and sketch the rest of the roadmap tomorrow. Some of it is more like a TODO list. I think it's important to focus on the top things, and these two things alone are plenty of work.