blog | oilshell.org

Summer Blog Backlog: Understanding and Using Shell

2021-07-08

Last month, I published Recent Progress on the Oil Language, which enumerated #blog-topics on the Oil language. It outlined posts on Oil's improvements over shell, as well as educational posts on concepts like string safety.

I started this post with remaining topics that aren't about the Oil language, and it ended with a focus on understanding and using shell in general. I list drafts and other topics as well.

As you can see, the blog is backlogged by 10 to 20 posts. This has happened before, so I think it makes sense to "declare bankruptcy" every year :-)

Let's go through the topics, and then I'll figure out what I can realistically write. If I have time in the future, I can return to some of them . In the meantime, I can refer to this post in the future, so the blog remains coherently connected.

Table of Contents
Spoiler
Drafts Written
Wiki: How to Understand Shell
Slogans
Fallacies
Concepts
Wiki: How to Use Shell
Shell Programming Patterns
Anti-Patterns
Continued Tomorrow

Spoiler

This post will meander through many ideas, so I'll state the conclusion up front. These are the two most important posts to write:

  1. Five Years of Oil Milestones (And Some Wrong Turns). It's past time to take stock of the project! How did we get here, and what did we learn?
  2. The Perlis-Thompson Principle. A #software-architecture principle that I've been hinting at in #comments. I've come up with a name, but not yet explained it. It has practical consequences: not just for Oil's design, but also for the design of languages, OSes, and distributed systems.

Now let's see what else belongs on the Oil blog.

Drafts Written

These two posts are fairly close to done:

  1. Oil Starts Fewer Processes than Other Shells. A worthwhile demonstration of how the #shell-runtime does basic optimizations. Surprisingly, all Bourne shells I tested do them! This draft is over a year old now, and I've already mentioned it in #blog-topics :-(
  2. Shell History and Trivia. This post links to #comments, including ones on Ken Thompson's original paper on Unix shell. The paper popped up in a few places recently, which is great, because it will form part of the explanation for the Perlis-Thompson Principle.

Wiki: How to Understand Shell

Readers have praised this blog over the years, but there's still a big explanatory gap regarding Unix shell. I say this because I see repetitive arguments for and against shell erupt on many discussion threads, like announcements for Oil and other alternative shells.

As another anecdote, this post from last month links to The Bourne Shell Is Not a Programming Language. It sounds like it could have been written today, but it's from 2008.

So I've been thinking of ways to advocate shell. I want users to enjoy it. I've tried to do that with #shell-the-good-parts posts, but I haven't published enough of them.

Maybe I should start a Twitter account for pithy explanations of shell. This wiki page has ideas:

Wiki: Slogans, Fallacies, and Concepts. Under construction.

The following sections sketch selected ideas I like.

Slogans

Shell Is Now a Good Language. There are a number of users who avoid Oil specifically because it's a shell! This ironic because Oil is supposed to fix the things about shell that make you avoid it. :-)

Shell Is the Language of Heterogeneity and Diversity. Data analysis, building Linux distros, and distributed systems are inherently heterogeneous problems. This trend will only increase in the future. We'll never be able to express these problems under a single type system. (Feel free to argue with me.)

Oil Is the Shitty Language Rehabilitation Project. Related: this comment on the YAML Problem.

Unix Programming Is Like Woodworking. Not everyone will get this lobste.rs comment, but I think it's worthwhile. Unix programmers and woodworkers make their own tools. Tools are made of wood; tools are made of text. You can make bespoke tools in IDEs, but in practice individual programmers don't do it.

Fallacies

Sometimes an inverted explanation of a concept is more memorable. That is, fallacies are a form of "negative knowledge".

Fallacy: Python Is A Good Shell Replacement. It's not either-or. I sketched a new answer in a post last month.

Fallacy: Unix Processes are Slow. They're often the only way to make use of your whole machine! Process-based concurrency is underrated.

Fallacy: Programming With Strings Is Slow. It's often much faster. The kernel natively deals with strings (byte streams), and the kernel is fast.

Fallacy: You Can Extend Your Type System Across the Network. Large distributed systems can't be atomically upgraded. (Example: the entire Internet, although it's also true at much smaller scales.) This conflicts with type safety as a global property of a program.

Concepts

The Perlis-Thompson Principle. I figured out a good way to explain this crucial idea in software architecture: by coining a name that refers to Alan Perlis and Ken Thompson! (Incidentally, they're both early Turing Award winners.)

Concept: String Hygiene (aka String Safety).

Concept: Process-Based Concurrency (e.g. with fork(), exec(), wait()). As mentioned above, it's underrated. It's also the oldest form of concurrency, and we can contrast it with threads and async.

A thread is a process that allows you to write race conditionshttps://t.co/0QvA07mjS4

— Oil Shell Blog (@oilshellblog) March 24, 2021

Concepts: Policy vs. Mechanism; Control Plane vs. Data Plane. Last year, I mentioned that factoring into processes is an underrated design skill. These related ideas may help you determine where draw the line between processes.

Wiki: How to Use Shell

Not only do I want to help users understand shell, I also want to show practical ways to use it. I started this wiki page:

Wiki: Patterns and Anti-Patterns. Under construction.

As with the fallacies, the negative knowledge of anti-patterns is also useful.

Shell Programming Patterns

The $0 Dispatch Pattern. I sketched this in Four More Posts in "Shell: The Good Parts" (February 2020), but I haven't written a post that explains and demonstrates it. Nearly all my shell scripts are written like this. Example: the sample code in Shell Has a Forth-Like Quality.

The PPPT Pattern: Parameterized Parallel Process Table. This pattern helps you invoke parallel processes and collect their results.

Anti-Patterns

Generating Code with Raw String Concatenation: Again, the Shell-in-{YAML,Docker,systemd,Chef,Vagrant,Python} Anti-Pattern is a primary example of this.

The Whitespace Joining Anti-Pattern. This came up recently in SSH quoting. Oil solves the analogous issue with echo and eval with shopt --set simple_echo simple_eval_builtin, which are on in option group oil:all.

Continued Tomorrow

This post got too big! There are still more #blog-topics to enumerate:

In the meantime, let me know if there are particular posts that caught your interest. Or let me know if none of this made sense!

The wiki pages Slogans, Fallacies, and Concepts and Patterns and Anti-Patterns are rough, but I expect that they'll grow into a central part of the project in the future. It's hard to explain why and how to use shell!