Roadmap #3: Two Top Priorities

2016-11-21

Yesterday I wrote a retrospective, and today I will sketch a roadmap for the next few months.

I accomplished most of the things on the last roadmap: I fixed up both the ${} parser and arithmetic parser, which allowed me to parse most of git. I wrote about about weird shell syntax.

I didn't repost messages from Reddit or Hacker News, mostly because I edited them into higher quality blog entries. (For example, this post should be turned into a blog entry or two.)

Test the Oil Language Design

Now that I've laboriously parsed real shell scripts into a clean AST, I have a fun project ahead of me: print the AST into the new oil language.

I've designed the language with a long document, and now I can test it out by automatically porting real programs. Here's a good test for the language: the converted programs should be immediately readable for expert shell users, but also approachable for non-shell programmers.

I'm thinking of the Python/Ruby/Perl/JavaScript programmers who shy away from shell due to its prickly nature, preferring to write verbose "shell scripts" in those languages. Or people who think its a "simpler" to embed shell code strings in JSON or other config files.

Fill Out the Shell Runtime

The current code does a lot already: external commands/builtins/functions, pipelines, subshells, redirects, command sub, here docs, variables, etc. But there is plenty more to do — the runtime is much less complete than the parser.

Both the osh and oil languages will share the same runtime — they should remain purely front ends. There will need to be compatibility flags, but that shouldn't be a big deal.

Here is the plan:

Publish the "spec" test results as HTML. There are 37 tests/*.test.sh files totalling 3500 lines in the oil repository. They thoroughly test corner cases in shell; hence most are currently failing.
Document the sh_spec.py test framework so people can write more tests.
Accept contributions that make existing tests pass. This will involve patches to the Python code in core/*_{eval,exec}.py.
Also accept contributions that add new tests, to document a feature's behavior. Making the tests actually pass is preferred, but not required!

Currently there is no C++ skeleton, but I want to work on that soon. However, I do think it makes sense to get tests passing with a Python implementation, even if it will all be ported C++ eventually.

Maybe it's just me, but it seems to make things go faster overall. Implementing a shell is software archaeology, and iteration speed counts.

The runtime should be compatible with bash (which has its own compatibility flags like set -o posix, although I'm testing dash, mksh, and zsh and considering their behavior as well.

If you're interested in contributing, please send mail to the oil-discuss@oilshell.org list (or to andy@oilshell.org).

I will end today's post here, and sketch the rest of the roadmap tomorrow. Some of it is more like a TODO list. I think it's important to focus on the top things, and these two things alone are plenty of work.