Why Sponsor Oils? | blog | oilshell.org
This post describes the last two Oil releases, and then elaborates on emerging project themes.
The most important one is thinking of the #shell-runtime as a state machine that receives asynchronous messages. This work is in progress, but it's worth describing the motivations, and what we've done so far.
Oil version 0.9.7 - Source tarballs and documentation.
I wrote about the 0.9.5 release in November, in Winter Blog Backlog: Recent Progress.
The work in the the 0.9.6 and and 0.9.7 releases has a variety of motivations:
SIGWINCH, the signal
for terminal window size change. The fixes and testing made me realize that
we should "recast" the #shell-runtime as a state machine.
The next two sections have details and credits. If you're casually following the project, you may want to skip to the last section.
Full changelog: https://www.oilshell.org/release/0.9.6/changelog.html
+= operator. These both work, and many new test cases pass
(spec/append
results).
export foo+=bar (static)export foo${x}+=bar (dynamic)
EISDIR error).osh now supports -l and --login (no-ops for now)fg builtin. Even more importantly, he added
terminal-based pexpect tests to verify the fix!pgen-native parser generator, the fast
version of pgen2. We still need help with this!
I released Oil 0.9.7 two days ago. Let's start with the infrastructure changes. Then we'll look at user-facing changes, which leads into the larger state machine theme.
docker build has
caveats with regard to reproducibility and correct incremental builds. We
should address this in future work.
app-tests CI task, which runs the ble.sh tests with
OSH.
Now let's look at closed issues, which leads into the state machine theme.
| #1077 | Add interactive tests that match other shells (e.g. Ctrl-C is exit code 130) |
| #1072 | Fix vm-baseline benchmark after optimization |
| #1067 | Terminal resize causes wait to exit |
| #1064 | Tab completion does not suggest aliases or functions |
| #743 | $PATH is empty when it's not in the parent's environment, unlike other shells |
| #467 | Ctrl-C in command substitution exits parent shell |
Full changelog: https://www.oilshell.org/release/0.9.7/changelog.html
Let's focus on two of these issues:
wait builtin to return, but it
should be ignored.They don't seem related, or even that interesting. But they led me to reconceptualize the shell runtime as a state machine.
They both relate to signals: Ctrl-C causes SIGINT, and resizing the
terminal causes SIGWINCH. And they reminded me of past bug fixes, like
running trap handlers when the read builtin is interrupted.
I realized I've been playing Whac-A-Mole with this class of bug, which is
bad. So I started to work on the pexpect tests that Brandon added for
the fg bug, expanding the harness and planning a test matrix.
I fixed these bugs, and found more bugs to fix (e.g. in the wait -n variant).
Revelation: We were missing an important way of testing the shell! Now that we
have test/interactive.py, we can make monotonic progress on a
multi-dimensional test matrix. More on this below.
I also tagged older bugs #signal-handling so I can make another pass over them. They should fall in specific cells of the test matrix, and "disappear" once those cells are filled in.
Based on this experience, I sketched an idea for a blog post:
The idea is that the shell interpreter walks the syntax tree and:
fork() to start a process, orread() for the read builtinwaitpid(-1). This is how the kernel tells us that a
process changed state (it finished, or suspend/resume).kill command, ...wait (all jobs), wait -n (next job), and wait %1
(specific job)One part of this is the singleton Waiter abstraction, which has existed for
years. What's new is integrating syscalls and signals into a single state
machine model.
It reminds me of DJB's self-pipe trick.
How do you wait on a child process and a async read() concurrently? By
writing a byte to a "self pipe" in the signal handler, so it reduces to
select().
We're not using the same mechanism, but we also have to unify disparate concurrency styles. (Linux has signalfd, but Oil should be portable to all Unixes.)
I felt dumb for putting this issue off while bugs piled up, even though I hinted at it in the last sentence of this 2019 post.
But then I ran grep SIGWINCH on bash's changelog and found that it has a
history of similar
issues.
They're also playing Whac-A-Mole with signal bugs.
From reading the source of other shells and using strace on them, I don't believe they have a clean runtime model. For example:
diff <(sort left) <(sort nonexistent)
I wrote a comment in test/interactive.py that describes this five dimensional test matrix. It should let us explore a large portion of the state space and prevent regressions.
waitpid(), read(), ...trap 'echo X' SIGWINCHstdin is connected to a terminal, as
opposed to a file. They shouldn't exit in certain circumstances.Again, the idea is to make monotonic progress rather than playing Whac-A-Mole. This will also help the code translate cleanly and automatically to C++.
I wrote about related problems in Technical Issues and Risks (August 2020) > Deferred Issues: What the Interactive Shell Depends On.
Despite recently declaring that the interactive shell is "punted", I'm thinking about all these issues again, and I have a plan for each one:
yield or pipesOne way to clarify this: I'm limiting the scope of the interactive shell to my own usage. I want to punt customizations outside the project -- to the headless shell. In contrast, I very much care about making the Oil language useful for others. That is the core of the project.
I expect that the state machine model will improve Oil, and that future blog posts will make reference to it. I've written a lot about principled and exhaustive parsing (#parsing-shell and #ASDL), but not much about the #shell-runtime. That's because we were missing something!
This post is now too long, so I moved these themes to the next post:
I want to write these posts:
As far as coding, these issues are on my mind:
As usual, these metrics help me keep track of the project. I hope they'll also give the compiler engineer color on what needs to be done.
Let's compare this release with Oil 0.9.4 - User Feedback.
OSH spec tests:
Translation progress:
I forgot to mention that I fixed a mycpp bug related to field inheritance, which led to the big jump in tests passing in C++.
The source code is getting more correct, but not much bigger:
Ditto for the binary: