Why Sponsor Oils? | blog | oilshell.org

Roadmap #3 Continued

In the last post, I mentioned the two top priorities for the project: refining the design of the oil language by automatically converting shell programs to it, and fleshing out the runtime portion of the shell and spec test suite.

In this post, I outline more things to do. I'm sure I'll end up dropping some of them for lack of time, but it won't hurt to publish a TODO list.

Ongoing Work

This is work that I expect to be spread out over a period of time.

• Polish parse error messages. Continue to attach position information throughout the code.
• Report runtime errors with stack traces, like Python.
• Implement remaining shell features first in the parser, and then in the runtime. This will be done on-demand, since the parser is pretty complete. Unimplemented features:
  • extended glob (probably involves an additional lexer mode, for a total of 14)
  • Array L-values: array[i + 1]=x
  • Keyed array literals: a=([5]=x [i + 1]=y) (can be an indexed array or associative array)
  • the let keyword, an alternative to ((
  • brace expansion: {alice,bob}@example.com
  • coprocesses (using bash syntax?)
• Publish parts of the old repo:
  • Docs on oil design, implementation, etc.
  • C++ code, which will form the seed of a C++ implementation.
  • Research scripts written in shell/awk/make/etc. (doing strace, ltrace, etc.)

Short-Term Plans

• Type check the code with Python 3 type annotations. (Although this hit a snag which I'll describe tomorrow.)
• Better AST printing. After the oil printer is done, a three-column format would be nice: osh, AST, oil. This will give users a flavor for the language.
• Testing tools:
  • More automation of parsing real projects with wild.sh, and print the results into an HTML table.
  • More automation of spec.sh, and print the results into an HTML table. (Could also try out asciinema.)
• Fix and enhance globbing. Right now we glob every word, even say echo. The algorithm for when to glob is probably something like: look for *, ?, and balanced []. Research how other shells do it. Add options like set -o nullglob.

Medium-Term Plans

C++ port. The current plan is to write each component in a sublanguage of Python, and then, using the Python AST module, write very specific translators to C++. Explaining this will take a few blog entries.

• core/id_kind.py is easily translated to a big set of enums and strings.
• osh/lex.py will be done with re2c.
• I'm currently making the AST more regular, in preparation for translation (core/*_node.py). I may use Python as something like Zephyr ASDL, which the Python interpreter itself uses. Stretch goal: export the AST to OCaml for things like unused and undefined variable analysis.
• The four parsers are an open question, but it's probably possible to translate them (osh/*_parse.py).
• The evaluators will be hand-written, because they interact closely with the OS and C++-style strings.

Long-Term Plans

• Add functionality equivalent to Awk and Make. To limit the scope, I don't think I will automatically translate them, in the style of osh and oil.

  Justification: relative to shell, there's much less Awk code in the wild. Sometimes Make is written by hand, but it's often generated by autoconf or CMake.

• Interactive shell -- I want to have this bootstrapped in the oil language, adding C bindings to libraries like readline. User-defined completion functions are probably the first use case for an improved language.

I could add more, but that's enough of the TODO list for now.

Since releasing the code, I made some big changes to the enums used in tokens and nodes.

And I'm currently working on polishing the AST format. One issue is that I want to support printing the AST to oil, as well as executing the AST directly. These requirements pull the AST in different directions. I will have more to say on this later.

Tomorrow I will give a roadmap of future blog topics, as opposed to the project design and implementation roadmap in this post.