blog | oilshell.org

Notes on the HotOS Unix Shell Panel

2021-06-05

On Thursday, I (virtually) attended the HotOS 2021 panel on Unix Shell, and took notes on Zulip. This post summarizes them, and elaborates on comments I made during the panel.

The event gathered a diverse crowd of researchers from operating systems, programming languages, and HCI. Bells Labs legend Doug McIlroy and bash maintainer Chet Ramey were also there!

Some of these notes may not make sense to people who weren't there. Feel free to ask questions in the comments.

Be warned: this post started out as fairly neutral notes, but as I edited it, I inserted several pronouncements about shell, and Oil. I invite disagreements, since will make the project better and help explain it :-)

Table of Contents
What I Took Away
There's a Lot of Interest in Unix Shell
Shell Is Hard To Use
Statements by Panelists
Comments by Doug McIlroy
My Messages for Researchers
Oil's Headless Mode Should Be Useful for UI Research
Oil Is Implemented Like a Programming Language
How Can Oil Support PaSh- and POSH-Like Annotations?
Papers to Read
Odds and Ends
Discussion: Does Shell Have A Heap?
Conclusion

What I Took Away

The panel attendees weren't the usual audience of this blog (programmers and practitioners from Hacker News, Reddit, lobste.rs), but they have many of the same issues with shell.

So I feel like the event validated the motivations for Oil, which has been in progress for over 5 years now (for better or worse)! I still need to write that "project retrospective", and do a better job of opening it up to more contributors.

There's a Lot of Interest in Unix Shell

Not only was the group diverse, but it was big! Many researchers wanted to talk about shell.

There were also many threads online about the paper that accompanied this project. It was at the top of Hacker News, Reddit, and lobste.rs multiple times!

Shell Is Hard To Use

The difficulty of learning and using shell was the overwhelming feedback I heard. (This may have been because I spent time in the 3rd breakout room, which had several HCI researchers.)

This of course isn't news to me -- in fact, it's the main motivation for the Oil project :-) But it was good to hear this feedback again from a different group.

I have two responses to this:

(1) Shell should be more like the dynamic languages that "won" (Python, JavaScript, Ruby, ...). As on Hacker News, there was talk about Python as a better shell, which I view as a fallacy. It's not "either-or". However, it's absolutely true that users try to "import" their Python knowledge into shell, fail, and become sorely disappointed.

Remember that Oil has a dual nature: it's an upgrade path from bash, but it's also a "clean slate" shell language. It's not obvious that this is possible, and I need to explain it more, but so far everything is working. In the last 2 months, I've gotten even deeper into the Oil language, and I still feel this way.

(TODO for me: write about 10 posts on many new features; write A Tour of Oil.)

A few days before the panel, I updated the Oil home page to reflect the "clean slate" viewpoint, in addition to the legacy viewpoint:

Oil is for Python and JavaScript users who avoid shell!

(2) Oil's new headless mode can (optionally) divorce the shell from the terminal, enabling better user interfaces. Shell is fundamentally a language-oriented interface, but it doesn't have to be a terminal-oriented one. That's sort of a historical accident.

This work is very recent, and I haven't written much about it. I'm excited about it because it simultaneously reduces the scope of the Oil project, while allowing other people to build interesting things on top. More on this below.

Statements by Panelists

There were statements by four panelists, and a few caught my interest:

Comments by Doug McIlroy

McIlroy wasn't one of the four panelists, but he made interesting comments:

Related: I played with Doug's 2014 code a few months ago, but didn't blog about it!

My Messages for Researchers

Oil's Headless Mode Should Be Useful for UI Research

The headless name is an analogy to a headless browser, like Chrome. Just like you can have a web browser without the UI, you can have a shell without the UI.

I want the UI to (optionally) live in a separate process from the shell itself, which enables it to be written in a different language against a stable IPC interface.

I mentioned headless mode during the panel, but I don't think the message will get through without a concrete demo. Thankfully Subhav Ramachandran has started one! He also deserves a lot of thanks for motivating this work in the first place.

I need to write properly about this later, but here are some preliminary docs:

And comments on what you can do with it:

A GUI around a shell can show state. I believe that will help users considerably. Right now experienced shell users just "imagine" what state they are in, with a tiny bit of help from the prompt.

Let me know if this sounds interesting! I need help.

Oil Is Implemented Like a Programming Language

I suggest that Oil's implementation is more amenable to research hacking than bash, or even dash. (Although dgsh is notably a bash fork!)

A concrete way to see this is with osh -n:

# print the syntax tree (in abbreviated form)
$ osh -n -c 'echo "hello $name"'
(C {<echo>} {(DQ <'hello '> ($  Id.VSub_DollarName '$name'))})

Reasons:

How Can Oil Support PaSh- and POSH-Like Annotations?

This issue was discussed multiple times during the panel: it's not possible to apply traditional compiler analyses to shell because external commands like grep are opaque to it. If you don't know what side effects happen, then you don't know if parallelization is safe, etc.

These recent research projects both rely on command annotations to solve that problem:

Building such parallelization and distribution directly into Oil probably isn't in scope, but the annotations are (similar to Python supporting multiple external type checkers with its type annotation syntax). I filed this issue:

I also wonder if Oil should expose its internals for metaprogramming:

I've been working on features to let Oil evaluate untrusted config files, and some of that work should be useful for annotations. Annotations on proc arguments can also be used for docstrings, completion hints, and even types / content types.

Let me know what you think! The PB&J paper in the next section seems related to PaSh and POSH.

Papers to Read

There were pointers to recent shell research I wasn't aware of. It would be nice to have a "shell bibliography" on a Wiki page or in a git repo!

Odds and Ends

I think I want to double down on this analogy:

Oil is to bash as LLVM/Clang is to GCC.

The LLVM project was initiated in the early 2000's. After several years, it increased in scope to the Clang compiler (C, C++, Objective C). About 10 years later, it enabled Rust, Julia, and Zig.

Analogously, I hope that there will great things built on top of Oil, particularly the new headless mode. But I know it will take a long time! I don't have the bandwidth to do it, so I want open up the project more.

Discussion: Does Shell Have A Heap?

This was a short but useful exchange on language design and the philosophy of the shell. Paraphrasing:

Conclusion

Thanks to Michael Greenberg, Konstantinos Kallas, and Nikos Vasilakis for organizing. This was a great event with a lot of energy!

I want to follow up with 2 or 3 people mentioned above. It also caused me to resume a conversation with Stephen Kell on text vs. binary data. (That reminds me that I still need to write a post on this comment: text as a narrow waist.)

I imagine that some parts of this post didn't made sense to people who weren't there, so feel free to ask questions (and disagree) in the comments.