blog |

Ambitions for a Unix Shell


As discussed in the January blog roadmap, I want to concretely describe a reduced Oil language, and see if we can get it "done" in 2020.

So, to give context to upcoming posts about the language, let's review the project's goals from different perspectives:

  1. What languages and tools does Oil aim to "replace"?
  2. What problems does a shell help you with?
  3. What application areas is shell good for?

This post discusses just the first question.

Table of Contents
Three Levels
The Biggest Cut
Appendix: Comments That Add Color

Three Levels

I visualize the goals as a "ladder" to climb:

  1. Replace my use of bash. This the bare minimum. If it doesn't happen, then I'll feel like the project has failed.

    On the other hand, there are "backup plans" mentioned in the roadmap. In the worst case, I hope the source code, tests, and knowledge contained therein will be useful to another project.

  2. Replace the world's usage of bash. Replacing my usage isn't enough, because other projects I want to work on are blocked by gross shell scripts :-) For example, many embedded systems are now Linux systems.

    Another way to think about it: a programming language is a kind of intersubjective truth: its meaning depends on agreement in the minds of other people. It's not enough that a program can be executed by the computer; it has to be understandable and modifiable by people.

    But I realize that, in the best case, this will take decades. That's OK, but I want to get something concrete done in 2020.

  1. Replace my use of Python. I've written and maintain dozens of Python 2 programs, and Oil could provide an alternative upgrade path for them, as well as for similar programs with few dependencies.

    I may not have stated this goal explicitly, but I've had it in mind since the beginning. It's natural because Oil currently reuses parts of the CPython 2.7 interpreter.

    But I'm explicitly cutting this out of the project.

    I decided to cut it after making progress on it! I hacked on py_object_main.c, which compiles against a 57K line slice of the CPython interpreter.

    That showed me that it's very feasible, but will take months, not weeks. Implementing the equivalent of that 57K lines from scratch would take more than a year.

The Biggest Cut

So the biggest change is that Oil will be based on strings (and arrays of strings) like shell, rather than having structured data types like Python. It may have some awk-like numbers, but not immediately.

In particular, I no longer think Oil will reuse any of the CPython interpreter. Translating Oil with mycpp made reusing it less appealing. The reasons for this are complicated, but feel free to leave a comment if you're curious about it.

This is drastic, but upcoming posts will show that the language's power isn't reduced for the primary use cases.

Importantly, I didn't start writing a shell because I want to write all my code in shell.

I started writing a shell because I program regularly in Python, C and C++, JavaScript, R, and many DSLs. And I need a way to glue these programs together!


I have several drafts, and I'll try to arrange them in a coherent order:

Appendix: Comments That Add Color

So Oil will certainly be a "layer on top" at first, but I hope that eventually small systems can be built with a kernel, a compiler, and Oil.