The Long Slog Through a Shell -- Intro


I just fixed the hardest bug ever in OSH, and now OSH can run a real program I didn't write. It can run Python's 16,000 line configure script as well as the 1388 line config.status script it generates. (Both of these are generated by autoconf.)

It was a stupid bug with a trivial fix -- closing a file -- but in my experience, that's typical of bugs that cost you a lot of time. Grrrr.

Now that I've completed a major milestone, I feel comfortable spending time to edit the notes I've made since early May or so.

Where are we?

I listed four goals in [Working Toward an OSH release][]. I'm pretty satisfied with #3 and #4 -- spec test failures and running a real program I didn't write. I feel like they're in release quality.

This means I have to go back to #1 and #2, which I was working on before and during [vacation].

  1. Fix up configure system
  2. Error messages. I'm not going to be able to fix up all the error messages, but I want to demo really good error messages.

I also want to change #2 to "carrots" -- i.e. reasons that users would adopt OSH. But we can talk about that later.

I also introduced the test-driven style of working in these posts:

This is the long slog through a shell that I promised in the OSH Release post.

The post contains

If you don't want to read on, here are the main points:

Also, from the last post:

list of features

In the last post, I said I would give the criteria for an OSH release.

I have a draft of that post, but I started to "go wide" on a few of the issues encountered there.

In particular, "Running real programs" is a risk.

Parsing is 60% of the work. But there are still are roughly

This post is listing a bunch of corners of shell I found. It's for shell experts (I'm still in Twitter Mode).

Time keyword

Time is a block.

Running configure

negation with ! -- not that much interesting, except I figured out that you could do:

! false

This could be used to silence failures with errexit


grep foo bar || true ! grep foo bar || echo "succeeded but we thought it woudl fail"

Running test/wild.sh parse-usr-bin

shift builtin

The only thing interesting here was avoiding quadratic behavior.

Interestingly, Python's getopt has quadratic behavior. However it probably doesn't matter unless you have tons of flags.

read builtin

Didn't take multiple args.

Still doesn't respect IFS.

Trivia: if there's no trailing newline, read fails, but it still populates the variables. This interacts poorly with set -o errexit (see below).

set -u (nounset)

I implemented proper error messages.

Arithmetic Evaluation

Catching ZeroDivisionError -- one of TWO FATAL errors I could find.

Concept of Lvalue. I never really got this until now.

Integer overflow:

Fatal vs. Non-Fatal Errors

Almost all errors in bash are non-fatal by default. You have to

cat < nonexistent.txt # the failure to open means your program keeps chugging along. Python would raise an OSError exception.

Refactoring: Exceptions vs. Error Codes.

It's liberating to use exceptions!!!! I did the runtime; next I will do the parser.

1. set -e (errexit)

TODO: Link to blog posts on this. Copy from the other posts.

Recall what errexit does:

$ set -o errexit
> echo one
> false  # fails with exit code 1
> echo two
set -o errexit
if { false; set +o errexit; }; then  # errexit disabled for if condition
  echo OK


1. Pipeline Status

${PIPESTATUS[@]} in particular.

echo | wc -l
echo ${PIPESTATUS[@]}

This required introducing the Waiter abstraction

First pass:

p = [] for p in processes: p.Start() for p in processes: pipe_status.append(p.Wait)

This is wrong for two reasons:

Huge Refactoring of Processes and Redirects

Around January, when I started using ASDL, I switch word parsing to use a functional style.

Redirects persisted as "objects". Now they are just ASDL types.

This is the right way to do things.

Thunk abstraction, Process abstraction

Refactoring: Builtins are Functions in builtin.py

The builtins need executor state. They used to be attached to the executor.

But the state they need is DISJOINT.

Now the signature looks like:

def _Wait(argv, job_state):

This is good; it's a mistake in bash for state to be global.


When you have

Dependency injection is the same as functional programming.

Refactoring: State is in state.py

This opens the possibility to serialize the state? For debugging. Some people have asked for a debugger. I'm not going to write one in the near future, time, but it's a good thing to think about.

NOTE: If you've used the bash debugger let me know!

HereDocWriter -- starts a process.

Link to post on shell here docs.

Redirects in Pipelines

BUG Fixed by refactoring

echo 1>&2 | wc -l

zsh differs on this very simple program!

Background Jobs with &, $!, wait, jobs

Three forms of wait

Related to pipelines, so I might as well do this.

wait actually sets ${PIPESTATUS[@]} unlike other shells!

Pipelines can be in the background.

Most interesting part: async processes!

wait builtin

Handling Redirects for Compound Commands uniformly

Factored out the _Dispatch method

zsh bug:

Simple trivia.

Although ${1=foo} is the other one



If you have any questions about the semantics here, levae a comment.