blog | oilshell.org

Why Create a New Unix Shell? (2021)

2021-01-27 (Last updated 2021-02-12)

This is an update of the popular 2018 FAQ. Details have changed, but the spirit of the project is the same.

Whenever I announce a new Oil release, some readers are confused by the project.

This post explains the project's motivation from several perspectives. Because Unix shell is an old and successful technology, there are many ways of looking at it.

Table of Contents
Introduction
Frequently Asked Questions
How is Oil different than bash or zsh?
I don't understand. Why not use a different a programming language?
Shouldn't we discourage people from writing shell scripts?
Shouldn't scripts over 100 lines be rewritten in Python or Ruby?
Are you reinventing Perl?
I'm still angry, and I don't want you to create a new shell.
Should I limit myself to POSIX shell scripts?
How can you design a good language on top of shell?
Does this mean that Oil isn't a good interactive shell?
Why is it written in Python?
2021 Updates
What can I use right now?
Where can I see some sample code?
Don't shell scripts have many dependencies?
Comparisons
What's the difference between Oil and fish?
What about PowerShell?
What about $OTHER_SHELL?
What about this shell-like library / embedded DSL?
Conclusion
Appendices
More Reading on Oil
More Ambitious Ideas
More Ways to Understand Oil

Introduction

Before explaining why I created Oil, let's review what it is. You can think of a Unix shell in two ways:

  1. As a text-based user interface. You communicate with the operating system by typing commands.
  2. As a language. It has variables, functions, and loops. Shell programs are text files that start with #!/bin/sh.

In this document, we'll think of Unix shells as languages. The Oil project actually has two languages: OSH and Oil. Let's define these terms, along with two others for context:

More about the Oil language:

The best description of the project is on the home page:

Oil is our upgrade path from bash to a better language and runtime.

Frequently Asked Questions

This section paraphrases questions I've received and summarizes the answers. In most cases, I link to the original comment thread, which you can read for details.

How is Oil different than bash or zsh?

Oil is taking shell seriously as a programming language, rather than treating it as a text-based UI that can be abused to write programs.

To see why this is valuable, consider these two groups of shell users:

  1. People who use shell to type a few commands here and there.
  2. People who write scripts, which may get into the hundreds or even thousands of lines.

Oil is aimed at group 2. If you're in group 1, there's admittedly no reason to use it right now.

However, group 2 writes scripts for group 1 to use! So I believe the benefits of Oil will eventually bubble up.

In other words, I'm building a solid foundation for a few more decades of shell usage.

I don't understand. Why not use a different a programming language?

It's important to be compatible with existing code. You might not personally use shell as a programming language, but all Unix users still rely on big shell programs. It's often used at build time, but it's still used at runtime too, e.g. on embedded Linux devices.

Some of this code is old, but much of it is new. It's not a small amount of code, either. Examples:

Original question and answer (reddit.com)

Shouldn't we discourage people from writing shell scripts?

There are three problems with that:

  1. It won't work. It would be like trying to convince productive PHP programmers not to use PHP. Many people have wasted breath on that, but important sites like Wikipedia are still written in hundreds of thousands of lines of PHP.

    Like PHP, the shell language is useful, ubiquitous, flawed, and being improved.

  2. Even if a new line of shell never gets written, there will still be a huge installed base of shell scripts that you may need to understand (e.g. when they don't work).

  3. Shell is still the best tool for many jobs. Most new "cloud" projects rely on Linux system images, in VMs or containers, and shell is the ideal language for creating such images. Even if you use a framework like Chef or Docker, you're still using bits of shell.

Shouldn't scripts over 100 lines be rewritten in Python or Ruby?

The biggest misconception about shell is that you write only in shell. Shell is about polyglot programming. It's a language that grows.

It's clear that I like Python because:

But I also use C, C++, JavaScript, R, and dozens of DSLs like SQL and HTML. They're often the best or only reasonable language for a particular task (for speed, available libraries, or runtime environment). I also use programs written in languages that I don't know how to read or modify.

Factoring into heterogeneous processes is a software design skill, particularly in the domain of distributed systems.


That said, rewriting a shell script is reasonable in some circumstances. If everyone on your dev team knows Python, maintaining a shell script can be more costly than maintaining a Python script.

But Python and Ruby aren't good shell replacements in general. Shell is a domain-specific language for dealing with concurrent processes and the file system. Python and Ruby have too much abstraction over these concepts, sometimes in the name of portability (e.g. to Windows). They hide what's really going on.

I encountered a nice blog post, Replacing Shell Scripts with Python, which, in my opinion, inadvertently proves the opposite point. The Python version is longer and has more dependencies. It's more difficult to write and maintain.

Are you reinventing Perl?

It's true that Perl is closer to shell than Python and Ruby are. For example, the perl -pie idiom can take the place of awk and sed. However, Perl isn't an acceptable shell either:

Also:

However, it's true that, in some respects, Oil is retreading the same ground as Perl. But Oil is more faithful to shell, and its syntax uses fewer punctuation characters. In other words, it's less like "line noise".

Threads:

I'm still angry, and I don't want you to create a new shell.

You might be angry because you had to maintain a nasty shell script written by a coworker.

If that's the case, you should be helping Oil succeed! The only way to "kill bash" is to:

  1. Reimplement it, then
  2. Gradually migrate away from it.

This is analogous to how Facebook is moving away from PHP by developing a similar, but cleaner, language called Hack.

Perl, Python, and Ruby have all existed for over 25 years, but they haven't replaced shell. New shell scripts are being written every day.

(Oil also has some similarity to CoffeeScript, which smoothed over some of JavaScript's rough edges and added syntactic sugar, but didn't stray from its core execution model. CoffeeScript was a success because it influenced subsequent versions of JavaScript.)

Should I limit myself to POSIX shell scripts?

I've seen this suggestion a lot, and there are entire books devoted to it. If your script is small, it may be a reasonable goal.

For bigger programs, limiting yourself to POSIX is not just inconvenient, it's also an ill-defined and virtually untestable concept. Evidence:

As of 2021, I believe that OSH is a "better POSIX". POSIX is a descriptive specification and not a normative one. That means that it's an observation of how popular shells like ksh and bash happened to behave at a certain time. In other words, it's a compromise.

Similarly, OSH is based on extensive testing of the behavior of bash, dash, mksh, zsh, and busybox ash. That is, it uses the same philosophy as POSIX, but it specifies more of the language. Roughly speaking, spec tests are an executable specification.

How can you design a good language on top of shell?

POSIX shell has global options that affect parsing and execution like set -o noglob. And bash extends them with options like shopt -s lastpipe.

Oil continues in the same vein, adding several shell options under shopt. However, they are organized into 2 binaries so you don't have to remember each one:

Historical note: I prototyped an OSH-to-Oil translator in 2017, but it's on hold for now. Contributors are welcome to try to revive it.

Does this mean that Oil isn't a good interactive shell?

Making shell a good programming language is a prerequisite for creating a good interactive shell. See posts tagged #interactive-shell for details on Oil's unique support:

Anyone who's written a bash completion script from scratch can also see this. Shell doesn't have true functions, so bash's mechanism involves reading global variables likes $COMP_CWORD and mutating ones like $COMP_REPLY.

Although I'm a Vim user, I'm sometimes jealous that Emacs has a better programming language for customizing the UI. Like Emacs, I expect that many of Oil's interactive features will be written in Oil, and not C or Python.

Why is it written in Python?

It's more accurate to say that Oil is written in a collection of DSLs based on Python:

  1. A subset of statically-typed Python that's translated to C++.
  2. Zephyr ASDL for describing typed data.
  3. Regular languages via re2c. This avoids "groveling through backslashes and braces one at a time".

In other words, the interpreter is written in high-level code for correctness, but compiled to native code for speed. You could say that Oil is an experiment in #metaprogramming. It's similar to PyPy in spirit, but not in the details.

This unusual style affects Oil developers, but not end users. End users get a source tarball that is compiled with a normal C++ toolchain.

These posts explain our use of Python in more detail:

In the distant future, Oil's metalanguages may be evolved into the Tea language, i.e. a form of bootstrapping. But you'll still get a "normal" source tarball.

2021 Updates

The questions below are inspired by threads like this one from Reddit.

What can I use right now?

See Why Use Oil? — I keep it up to date.

As of February 2021, Oil is best used as a dev tool alongside bash or POSIX shell. It will help you write better programs.

You can use Oil in production, but it's slow in some cases. The upcoming C++ translation will fix that.

I encourage people to test Oil, and to give feedback on the Oil language! I also recommend it to sophisticated shell users who understand Four Features That Justify a New Unix Shell.

Where can I see some sample code?

This program prints Hello world:

var name = 'World'
echo "Hello $name"

Here's a more complex program inspired by a lobste.rs thread. It deletes git branches that are merged, except for master:

git branch --merged | while read --line {
  var line = _line.strip()
  # The * prefix means the branch is unmerged
  if (line != 'master' and not line.startswith('*')) {
    echo $line
  }
} | readarray -t :branches

if (len(branches) == 0) {
  echo "No merged branches"
} else {
  git branch -D @branches
}
 
# $ git branch --merged
#   master
#   merged1
# * in-progress
#   merged2

Notice these differences between Oil and shell:

I plan to port this snippet to more languages and write a longer post about it. (I also think readarray -t should be read --lines.)

More code examples:

Don't shell scripts have many dependencies?

Yes, this is a problem. I commented on it in the last post. Oil should have some combination of these two solutions:

Note that Python, Ruby, and JavaScript also have dependency problems, and a better shell will help us fix those!

Related answer

Comparisons

This section will help you understand Oil vs. other shells. Here are the primary differences:

  1. Oil is a rich and reliable language for automation. (Related: #shell-the-good-parts)
  2. It's a thin layer over the Unix kernel.
  3. It provides a smooth upgrade path from POSIX shell and bash.

You may be happy using another shell! Feel free to let us know what you like, and maybe analogous features can be integrated into Oil.

What's the difference between Oil and fish?

Fish is a good interactive shell, but a poor language for automation. So it's the opposite of Oil, and complementary to it.

It would make sense to combine the projects in some way:

Related:

I love fish as a shell, I just don't think it's good as a scripting language

 

What about PowerShell?

See this section of the previous post. PowerShell is more natural on Windows, and it's embedded in a large VM.

What about $OTHER_SHELL?

I link to dozens of other shells on this wiki page:

Oil has taken cues from some of these projects (and vice versa). Again, the biggest difference is that Oil is the only shell that's a smooth upgrade from bash.

This post about Linux distros gives color on why Oil aims to be compatible. Most distros are based on shell, with languages like Python being either nonexistent or playing an ancillary role.

What about this shell-like library / embedded DSL?

I also maintain a list of dozens of shell DSLs in various languages:

A lot of effort was spent on these projects, so I have no doubt that they're useful. Embedding has advantages in some situations.

But I don't consider any of these libraries to be bash or shell replacements. A shell is a foundational, low-level component of a Unix system.

Conclusion

Leave a comment if there is something you don't understand, and I'll answer and possibly update the FAQ.

Appendices

More Reading on Oil

In addition to the simplest explanation and the idioms doc mentioned above, see:

Again, we're removing the problems with shell, preserving the good parts, and gracefully upgrading it!

More Ambitious Ideas

More Ways to Understand Oil

Future blog posts can explain Oil in different ways.