blog | oilshell.org

Recent Progress on the Oil Language

2021-06-08

After garbage collector progress in Oil 0.8.8, and addressing user feedback in the Oil 0.8.9, I started working on the Oil language again. For context, the language has only existed since 2019, despite the fact that the Oil project as a whole is 5 years old.

I ended up making rapid progress in several areas, and took notes on Zulip. This post expands on the thread Notes for Oil 0.9.0 (requires login).

I expect 0.9.0 to be one of the biggest releases ever!

Rather than fully explaining each feature, this post enumerates #blog-topics. Not all the work is done, but I want to broach these topics now so the code doesn't get too far ahead of the blog.

As always, feel free to ask questions in the comments.

Table of Contents
Posts That Are Ready To Write
New Features
Oil's Solutions To Real Problems
Real Problems From Chris Siebenmann
More Blog Updates
Work in Progress
Educational Posts to Support String Safety
The YAML Problem
Official Docs
Language Concepts
What's Next?

Posts That Are Ready To Write

The features in this section are mostly done, but still need user feedback.

New Features

Oil's Solutions To Real Problems

I use the #real-problems tag to describe Oil's solutions to problems found "in the wild" (via user feedback and production code).

Real Problems From Chris Siebenmann

Chris Siebenmann prolifically documents shell issues on his blog, a rich source of #real-problems.

He also wrote that The Bourne shell and Bash aren't the right languages for larger programs, and I agree. Oil addresses all of his complaints, though it's not all documented:

Here's a 2008 post from Chris's blog saying essentially the same thing:

Unfortunately I don't think there's a really good Unix programming language to replace the Bourne shell, which is one of the reasons that writing programs in the Bourne shell remains so tempting.

So that's why I say that these posts feel like "groundhog day", despite being valuable and correct. Instead of documenting shell's limitations for future generations, we should fix shell. That's not easy, as it has to be done in a compatible way.

But this is the purpose of the Oil project! Below I mention A Tour of Oil, which will show that Oil isn't encumbered by much legacy at all. (We extend bash's shopt mechanism to achieve this.)

More Blog Updates

I want to take another stab at Python vs. Shell question in the Oil FAQ. It's a persistent question, and came up again in the HotOS Slack channel.

This short response that found some agreement, and I will incorporate it into the FAQ:

I would say the biggest misconception is that it's either/or. Python and bash go well together. The basic calculus is to write 200 lines of shell calling 200 line Python script, rather than 1000 lines of Python, and the whole thing runs faster.

Work in Progress

The work in this section isn't done yet, but thinking about how to describe it is useful. I like documentation-driven development!

Educational Posts to Support String Safety

Shell is fundamentally string-based, which is error prone. Oil adds mechanisms to make this style correct. String safety is just as important than memory safety (which Rust has raised awareness of).

These mechanisms deserve a few pedagogical posts as background. Over years of professional programming, I've noticed that seemingly simple things like backslash escaping cause confusion in practice!

The YAML Problem

I mentioned this problem in Comments on Build Systems and CI Services, and I've made some progress recently.

I also call it the Shell-in-{YAML,Docker,systemd,Chef,Vagrant,Python} problem, and mentioned it at the end of my BayLisa 2019 Notes.

The declarative part of shell can be used for multiple purposes:

Official Docs

I started docs on these features and need to finish them:

The most important doc is:

Language Concepts

I notice that I'm reusing shell mechanisms to turn OSH into Oil, like shopt and source. This is good because it keeps the language smaller.

These generic concepts/mechanisms are an orthogonal axis to features like the module system and declarative configuration. I want to explain:

What's Next?

This post helped me write my TODO list:

When drafting this post, many other #blog-topics came up, like those related to distributed systems, #software-architecture and dev tools. I've moved those to the next post.

With apologies for these dense notes, let me know if you have questions!