blog | oilshell.org

Frequently Asked Questions (2019)

2019-06-17

I'm writing this FAQ for an upcoming AMA on Reddit's /r/linux, but I expect it to be useful for all readers. (AMA stands for "ask me anything"; it's like a group interview.)

Update: AMA: I spent 3 years creating a new bash-compatible shell called Oil

Table of Contents
Why is the Project Named "Oil"?
Why is it Written in Python?
Is it Written in Python 2?
Why Use It?
Why Not Use It?
What Happened to the Oil Language?
Shell Expressions
Oil Expressions
Links
How Do I Try it?
How Can I Help?
More Answers
Notes

Why is the Project Named "Oil"?

  1. Because Oil is a compliment to "metal"! Shell is an essential lubricant for software systems.

    Programs in C, C++, Java, Go, Rust, etc. are analogous to "metal". I also use shell to glue together Python, R, and JavaScript.

  2. Because oil is a short word, like cat, awk, or sed.

The name often reminds people of the company "Shell Oil". Of course, it has nothing to do with that.

I expect associations to change over time. Python creator Guido van Rossum was fighting "snakes" for a long time, but now the name stands on its own.

Why is it Written in Python?

Python let me implement a large portion of bash in five to seven times less code. Small programs are easier to write and maintain, and they have fewer bugs.

Current line counts [1]:

Physical Lines Significant Lines
bash 142K 101K
osh 26K 14K
Approximate ratio 5.5x 7.2x

Squeezing a backward-compatible shell into 14K lines of code (excluding whitespace and comments) is a great result! It gives me confidence that new features won't bloat the shell.

To be fair, a big chunk of the Python interpreter sits below that 14K lines. But I didn't have to write that code. And I think of it as having an 87K line "code budget" to work with (101K minus 14K).

The main downside of using Python is that the OSH interpreter is too slow. But now that OSH can run thousands of lines of bash scripts, speeding it up is the top priority. I aim to do this without rewriting the whole program by hand, using "metaprogramming" and DSLs.

More on Python:

Also note that I started the project with 3000 lines of C++, as mentioned in the very first post. But I realized I would never get it done at that pace!

Is it Written in Python 2?

See the FAQ in Building Oil with the OPy Bytecode Compiler.

Why Use It?

OSH is ready for early adopters to test. I started using it myself in February.

At the moment, there are a couple reasons to use it:

(1) As a development aid while writing shell scripts.

(2) To support a new project to replace bash. The wiki page Shell Programs That Run Under OSH is editable and I would love your help in making it longer.

I hope to get feedback from the AMA for a more featureful 0.7.0 release. Although I'll still be happy if 0.7.0 improves only in speed.

Why Not Use It?

If you're interested in helping with any of these things, feel free to join oilshell.zulipchat.com.

What Happened to the Oil Language?

I described a new, legacy-free shell language over 2 years ago, but I haven't worked on it much since then.

Instead, I worked on running existing bash scripts because I think that's more important. There are many alternative shells, but I don't think they will replace bash because of its strong network effect.

I still believe that many shell features hinge on a better language, including:

In the last post, I mentioned that I made some progress on Oil's expression language.

With the caveat that I just started implementing it, the two sections below give you a flavor for the language. In short, the syntax and semantics resemble Python and JavaScript. You'll be able to use the var keyword and then write a "normal looking" expression. There are also some elements of Ruby, Perl, and Awk.

Shell Expressions

The following POSIX shell code tests if a number is between 5 and 10:

n=$(( 3 + 4 ))
if [ 5 -le $n ] && [ $n -le 10 ]; then
  echo "$n is between 5 and 10"
fi
# Output: 7 is between 5 and 10

I'd guess that 50-90% of shell users couldn't write that correctly the first time. Reading it afterward is an additional challenge.

Here's how you do it in the ksh dialect that bash supports:

(( n = 3 + 4 ))
if (( 5 <= n && n <= 10 )); then
  echo "$n is between 5 and 10"
fi
# Output: 7 is between 5 and 10

This syntax is OK (ignoring some extra parentheses). But the semantics are bad in at least three ways.

  1. There are edge cases that are confusing to developers. What happens when n doesn't look like an integer? What happens when n is 0x0A, 0123, x, or x+8?
  2. The semantics are inefficient. The lack of true integers in shell means that the interpreter must do many string→int→string conversions.
  3. I (re-)discovered in January that this sublanguage is dangerously insecure. What if n is $(echo pwned), or a[$(echo 42 >pwned.txt)]?

I didn't have time to blog about the security issue, but the gist of it is that there's a hidden eval every time you use shell arithmetic.

A statement as simple as echo $((x + 1)) leads to the evaluation of arbitrary code in x! An adversary that controls x also controls your system.

Oil Expressions

Oil will have more familiar syntax and simpler, safer semantics:

var n = 3 + 4              # 'var' for real ints, floats, dicts, etc.
if (5 <= n and n <= 10) {  # 'and' in expressions, && in commands
  echo "$n is between 5 and 10"
}
# Output: 7 is between 5 and 10

That's all I want to say about the Oil language right now. That's because I don't know if I'll be able to finish it!

If you're interested in a saner and more powerful shell language, the last post has some notes on how to get involved. And feel free to ask questions on oilshell.zulipchat.com.

Links

How Do I Try it?

The home page links to the latest release. The wiki page How To Test OSH has details.

How Can I Help?

I wrote about this in the last post. Feedback is appreciated!

More Answers

I wrote the last FAQ in January 2018. It's by far the most popular page on this site.

Other blog posts with answers are labeled #FAQ and #why-a-new-shell.

Notes

[1] The line-counts/run.sh script in the oilshell/blog-code repo counts lines of bash code.

The pydeps-src-only function in metrics/tarball.sh counts line of Oil code. It reports slightly higher numbers than the src and osh-cloc metrics, which are published with every release.