blog | oilshell.org

February Recap / Brain Dump

2020-02-20

I outlined a dozen or so blog posts in last month's roadmap, and I ended up finishing seven of them.

The last one was discussed on Hacker News three weeks ago. The discussion covered many of the remaining blog topics, like which features I've cut from Oil, and opportunities for a better interactive hsell.

So rather than write five more posts, this post summarizes important comments in that thread.

I also explain some recent progress in translation to C++, and summarize two releases I've made in the last month.

Table of Contents
Important Comments
"Fish Oil" Is A Good Idea (link)
Why Oil Won't Have Types in the Near Future (link)
shopt -s nullglob and #real-problems (link)
January Blog Recap
Recent Progress in Translation to C++
Unfinished Posts
Posts I Still Want to Write
Recent Releases
0.8.pre1 on February 17th
0.7.0 on January 20th: 6 months of Work
What's Next

Important Comments

"Fish Oil" Is A Good Idea (link)

I'd like to expose an API that will let others build a fish-like interactive shell on top of Oil.

Start a new thread on Zulip if you're interested in shaping the API. Here's one thread about the interactive shell.

I'm not sure exactly what this API looks like. It feels complicated because I learned that shells and libraries like GNU readline fight over SIGWINCH, the Unix signal that notifies a program when the terminal has been resized.

Context: I'm reducing the scope of the project to get it "done" in 2020, but I think Oil is substantial enough for bigger projects to grow around it.

Why Oil Won't Have Types in the Near Future (link)

This is the most important comment regarding language design.

It relates to experiences I had with Python 3 strings and the "third style" of HTML processing I'm now using to generate this blog (source code in lazylex/ and doctools/).

I tried to mint a slogan Bijections That Aren't Bijections, but I'm not sure it's a useful one.

I also updated the Structured Data in Oil wiki page with links to related projects involving CSV, JSON, recutils, and binary formats. I think these projects are more closely related to the kind of "types" Oil should have, in contrast to Python-like data types.

(Ironically, after writing this comment, I did refactor Oil to have something more like a proper integer type.)

Discussion on structured data is welcome. Prototypes with working code are even more welcome!

shopt -s nullglob and #real-problems (link)

Oil now has shopt -s nullglob on by default, which addresses the quoting problem described in A Surprisingly Common Mistake Involving Wildcards & The Find Command.

oil$ find . -name *.jpg    # unquoted arg expands to nothing
find: missing argument to `-name'

oil$ find . -name '*.jpg'  # correct usage of 'find'
foo.jpg

Details on Zulip.

I've also created the #real-problems tag, which contains this post and posts describing shopt -s strict_argv and strict_control_flow. Whenever I encounter a complaint about shell in the wild, I try to address it in Oil.

January Blog Recap

Recent Progress in Translation to C++

I said I would write Recent Progress in Oil, but let me focus on just the translation process, since I'll continue to mention it in the blog.

I wrote bin/osh_eval.py a few weeks ago, which includes parsers for all shell sublanguages and evaluators for some of them. Oil uses a clean dependency injection style, so it's easy to create subsets of the interpreter.

This program translates to 16,648 lines of C++ code, compared with 9,735 lines in bin/osh_parse.py, which I got working in December with Oil 0.7.pre9.

The initial translation on February 3rd produced 296 compile errors. Last night there were 38 compile errors. Much of this work involved rewriting Python reflection as textual code generation of C++ and Python. Details on Zulip.

But compiling isn't the end of the translation process: it still has to run! I often write stubs with assert(0) bodies to get the translated code to compile.

In summary: I'm making progress on translation, but it's not clear how long it will take.

Unfinished Posts

I mentioned these posts in the January blog roadmap, but I won't finish most of them:

In summary, I want to translate more of Oil to C++ before making any predictions. If you want the pithy answer, it's:

Translation is required to make Oil a production-quality shell, and whatever else fits in 2020 is what will get done.

Posts I Still Want to Write

On the other hand, these posts are worth the time:

Recent Releases

0.8.pre1 on February 17th

I'll describe this release in detail tomorrow, but the summary is:

0.7.0 on January 20th: 6 months of Work

I didn't announce the 0.7.0 release on January 20th, but here's a summary of the 0.7.x series, which started in July 2019:

Even though I feel this project has taken too long, that looks like an impressive 6 months of progress to me! The Oil language didn't exist in July.

(For comparison, the 0.6.x series from 2018-2019 was about the #interactive-shell and the OSH language features to support it.)

What's Next

I had more #zulip-links in this post, but I moved them to tomorrow's post. I want to make the project friendly to contributors, and one way to do that is by explaining its components and their status.

I'll describe the Oil 0.8.pre1 release in more detail. I want to credit contributors and review the metrics. This release spanned about 30 days, so analyzing it will suggest what we can finish in 2020.