blog | oilshell.org

Summer 2020 Blog Roadmap

2020-07-19

I just released Oil 0.8.pre8, and it reached a significant milestone! The oil-native binary can now run Crestwave's brainfuck interpreter in bash, and it's even pretty fast.

So now is a good time to come up for air and write blog posts. This post is an outline of what I want to write. If the January Blog Roadmap is any indication, I'll publish a few posts, then go back to coding again.

Please ask questions in the comments section to help me prioritize the writing.

Table of Contents
A Plan for Oil 0.8 and 0.9
Features Cut in 2020
Making the Project Sustainable
How Can Oil Be Useful Before It's Complete?
How to Rewrite Oil in Nim, C++, D, or Rust (or C)
Accepting Donations for a Technical Writer
Some Kind of Teaching / Mentoring
Advocacy for Unix Shell and Oil
More Posts in #shell-the-good-parts
The Unix Shell Should Evolve Like Perl 5
Why Use Oil?
Oil Starts Fewer Processes Than Other Shells
What's Next?

A Plan for Oil 0.8 and 0.9

The C++ translation process is creating a fast and correct shell, which is exciting. The next two releases will continue along the same path:

Based on the last few months of progress, I believe this can be done by the end of the year. Recall that back in January, I said I wanted to get something concrete done in 2020. This is about what I expected.

Features Cut in 2020

I also want to reduce the scope of the project, and point out some problems.

Making the Project Sustainable

I've been working on Oil for over 4 years! I considered it an experiment for most of that time:

After the progress in 2019 and 2020, I'm confident that the answer is yes. There's no risk of failing to deliver 0.8 and 0.9 — it's just work.

And with enough people working on the project, I'd argue there's no risk to Oil 1.0 or Oil 2.0 either. However, even though 30+ people have contributed code to Oil, I'd estimate that 90% of the code has been touched only by me.

I want to change that. Here are some posts that address "project sustainability".

How Can Oil Be Useful Before It's Complete?

The lack of real Oil usage has been bugging me for a long time, and I've mentioned it when soliciting help. I don't have a solid answer to this, so I'm interested in your ideas.

Outline:

How to Rewrite Oil in Nim, C++, D, or Rust (or C)

Last week, I got a mail from a reader Andrew who's rewriting Oil in the Nim language! This is great, as I believe in the power of parallel open source development.

I'd like to spread the knowledge of shell implicit in Oil, and independent implementations are a great way to do that.

Roughly speaking, there are 4 possible levels of reuse:

  1. Reuse the list of syntactic ID elements. This is a hard-won list of 300 or so things a bash-compatible shell needs to handle!
  2. Reuse the generated lexer, which is pure C and can be integrated into any language. This will save a huge amount of "groveling through backslashes and braces", i.e. what bash does with ~10,000 lines of hand-written C code.
  3. Reuse the ASDL schemas, like frontend/syntax.asdl. In some sense, this is the OSH language definition.
  4. Reuse the entire architecture of the Oil interpreter, which uses dependency inversion of state and I/O. (For example, it doesn't constantly save and restore global variables like bash does.)

Addendum: Rob Landley and toybox should reuse some of this knowledge. He says he's been working on a bash-compatible shell on and off since 2006! And he has expressed doubt about finishing it, so sharing knowledge only makes sense. However I'm not sure if we can communicate on the same wavelength.

Accepting Donations for a Technical Writer

Last year, I was asked if I accept donations for the project. My answer was no, because Oil is still an experiment. (In other words, I wanted to be able to abandon the project at any moment without guilt.)

However, I now think the experiment is succeeding. It feels like the codebase is "just on the edge" of being a significant advance over bash, the most widely deployed shell in the world.

In contrast, the documentation is not close to being done. And I think attracting contributors to docs is harder than attracting contributors to the code.

So I'd like to solicit donations to hire a technical writer for Oil's documentation. I'm not sure when I will do this. I don't really want to be an administrator, but it seems necessary.

I'd appreciate advice from anyone who has raised enough money to pay someone. I imagine that this is around 6 months of full time work. (E-mail me or leave a comment.)

Some Kind of Teaching / Mentoring

I posted an offer to edit technical writing on lobste.rs. I can imagine something in this vein to on-board contributors to Oil, but I'm not sure what it would look like.

Though a major problem with this model is that it sets me up as some kind of authority, whereas I'd really like people to tell me what I'm doing wrong.

There have been multiple contributors, including Koiche Murase and Josh Nelson, who have fixed things that surprised me. And Crestwave has done a lot of difficult debugging on codebases I'm unfamiliar with.

In addition, our .travis.yml was contributed a few years ago, before I knew anything about Travis CI. It's gradually turned into a useful and functional continuous build, of course written in shell. (As of the last release, it runs the ble.sh test suite and the mycpp translation of oil-native).

Advocacy for Unix Shell and Oil

I did a good job of advocacy at the beginning of the project, but haven't had time for it lately. Here are some more ideas.

More Posts in #shell-the-good-parts

Headings for this post (yes, this is an outline for an outline!):

The Unix Shell Should Evolve Like Perl 5

This is a post derived from these links:

Summary and analogies:

In contrast, Oil has shopt -s oil:basic and shopt -s oil:all.

Why Use Oil?

I need to explain Oil from scratch again. Recent progress on the implementation has made it more clear what Oil is and what it isn't. Draft on the wiki:

I want to condense those points and put them boldly and clearly on the Oil home page.

Oil Starts Fewer Processes Than Other Shells

The comics post back in April alluded to this result, but I never published/explained it. This is mainly because I wanted to gather data on real shell scripts in the wild. For now, you can see the benchmarks:

Summary: Across several dozen test cases, Oil starts fewer processes than every shell except the uncommon yash shell. I could have optimized it more, but I didn't want to game the benchmark.

For context, I want Oil to be faster than bash and other shells along three dimensions:

  1. Parsing. This has been achieved, although changing the memory allocation strategy will affect the results.
  2. Running programs that do computation without I/O, like autocompletion scripts.
  3. Running programs that do a lot of I/O, like building Linux distros.

What's Next?

I outlined 8 or 9 posts here, but there are still two more with #blog-topics:

After I publish those, I'll write the plan for Oil 0.8 and 0.9.

Let me know what you're thinking in the comments!