Blog Retrospective #1

2016-11-20

Now that I've released the code, let's take a minute to look back. Here are the blog posts so far grouped into five general themes:

1. Project Updates

oil is a new Unix shell. It parses real programs, including Aboriginal Linux, /etc/init.d scripts, debootstrap, git, and other projects.
The source code has been published.

2. Shell Trivia / WTFs

Shell is composed of four interleaved sublanguages.
Unix shell has a bad syntax. For example, the # character has five different meanings inside ${}, and the / character has three meanings. I found an example of the latter in the wild.
There are many incorrect ways to use arrays.
Here docs are processed in a post-order traversal of the AST, and use temp files.
In parsing git, we saw that comments are not lexical constructs, and that shell is parsed and executed line-by-line (but this distinction isn't important for scripts which use functions.)
In Problems with $((, I touched on the difference between operators and reserved words, e.g. subshell () vs. grouping { }. Some tokens that look like operators aren't actually operators.
Writing a grammar for variable substutitions revealed that bash parses but doesn't execute some constructs like ${a[0][1][2]}.

Future posts on shell trivia:

The non-operator issue comes up in FOO=bar and 2> /dev/null
Arithmetic in shell is dynamically parsed. For example:

$ myvar=123
> name=var
> echo $((my$name * 2))  # name of a variable is dynamically constructed
246

3. Language Design

The osh language is mostly identical to bash, and requires two tokens of lookahead to parse.

It can be automatically be converted to the oil language, which is still yet to be defined. But I've given a sneak peek of oil in some posts:

Oil will have methods on strings instead of custom syntax like ${var/pat/replace}, discussed here
Arrays will use the @ splice operator, discussed here
Problems with $(( talked about a more consistent grouping and subshell syntax: do {} and shell {}

Oil won't be just a shell, because Shell, Awk, and Make should be combined (example code).

4. Static Parsing

Static Parsing is parsing up front, in a single pass, which is aided by lexical state.

First, I gave a demonstration that other shells don't statically parse programs. Instead, they interleave parsing and execution.
There is a precedent for this distinction because [[ is statically parsed, but [ isn't.
Strings used to index associative arrays should be quoted to allow static parsing.

Future post: oil parses regexes statically, but other shells don't.

5. Parsing Algorithms and Techniques

A lexical state is now called a lexer mode. osh requires thirteen different modes!
Pratt Parsing is a simple and extensible algorithm for parsing expressions, and it subsumes the later "precedence climbing" algorithm. I published demo code, which among other things shows that Pratt parsers statically typed.
Oil treats $(( more uniformly, which allows osh to be parsed with two tokens of lookahead instead of arbitrary lookahead.

Future post: Shell should be parsed with a top-down algorithm. Bash uses yacc, which uses a bottom-up algorithm. This has implications for both static parsing and completion engines.

These five themes have been covered fairly thoroughly, although I've made note of a few more posts that would be nice to have. Tomorrow I'll outline some new, bigger themes for the blog.