blog | oilshell.org

Oil 0.9.4 - User Feedback

2021-11-20

This is the latest version of Oil, a Unix shell that's our upgrade path from bash:

Oil version 0.9.4 - Source tarballs and documentation.

To build and run it, follow the instructions in INSTALL.txt. The wiki describes How To Test OSH and Where To Send Feedback.

If you're new to the project, see Why Create a New Shell? and posts tagged #FAQ.

Table of Contents
What's Happened Recently?
Summary of Changes
Prompt Enhancements
The read builtin
Bug: Pipelines Aren't Blocked on Background Processes
The Oil Language
Closed Issues
What's Next?
Links
Appendix: Metrics for the 0.9.4 Release
Spec Tests
Source Code Size
Benchmarks
Native Code Metrics

What's Happened Recently?

Many things have happened in the project, but as usual the blog is behind! Highlights:

I also didn't forget about the "big ideas" I wrote about this summer. I plan to return to those in some form.

However, this is a "workmanlike" release, to get back in the rhythm of things after traveling and moving.

Summary of Changes

For the last couple weeks, I focused on fixing user-reported bugs and addressing feature requests. Thanks to Miles Alan and bb010g for particularly thorough testing.

Prompt Enhancements

To customize your bash prompt, you use a cryptic language in the $PS1 variable. This release implements more parts of that language:

We also ensure that $PS1 is set in interactive shells. (An interactive shell is one that's started without a script argument, and where stdin is connected to a terminal.)

The read builtin

Bug: Pipelines Aren't Blocked on Background Processes

Miles Alan reported an interesting process concurrency bug. It dates back to late 2018 when pipelines started to use the more sophisticated shopt -s lastpipe semantics.

By default, OSH runs the last element of a pipeline in the main shell process. Among other things, this means that the variable binding in echo foo | read myvar persists.

The bug is too subtle to explain here, but it reminds of this canonical example of why process-based concurrency is tricky:

sleep 1 &  # start background process

# The shell has to wait() for 2 processes in this pipeline
my-command | wc -l

But the sleep process might finish before both, in between them, or after both.

In other words, process done notifications via waitpid(-1) are asynchronous, and serialized, so shells need data structures to keep track of it all. This is an important part of the #shell-runtime I haven't written about.

The Oil Language

The feedback above is solidly in the OSH portion of the project, but we use it to improve the Oil language. Some thoughts:

Closed Issues

#1002 Pipeline launched from main process waits for a background process
#1001 Unhandled termios exception
#1000 read builtin: backspace interpreted as ^?
#996 osh throws an `OverflowError` & dumps a stack trace on `(return 2147483648)` or `(exit 2147483648)`
#986 Builtin read not interruptible via signal
#982 Set PS1 in interactive shells
#962 Support \A in PS1
#748 Support \D{} in PS1 and PS0

What's Next?

I want to write these blog posts next:

Oil 0.9.3 - Extended Globs for Nix. Motivated by Nix, I implemented one of the "last" OSH features. This is a good opportunity to restate and explain the principles behind the design and implementation of OSH. I hope to answer many questions on the RFC thread with concrete examples and general principles.

Oil Winter Blog Backlog. As mentioned, the blog is backed up, and I want to catch up. This might overflow into a few parts, like the Summer Blog Backlog did (part 1 and part 2).

Links

Reminder: Please try Oil and send feedback! The best feedback results from trying OSH and Oil on real programs. I just got some great feedback from Arturo Martín-de-Nicolás, which surfaced at least one bug, and reminded me of a gap in the Oil language (issue #1011).

Appendix: Metrics for the 0.9.4 Release

These metrics help me keep track of the project. Let's compare this release with version 0.8.12, which I announced in July.

Spec Tests

The spec test suites for both OSH and Oil continue to expand and turn green. The 0.9.3 work on extended globs resulted in whole suite of tests, because they can appear in many contexts.


Source Code Size

We have ~250 significant lines of code:

And ~600 new lines of physical code:

Benchmarks

I started using new benchmark machines in version 0.9.2, so let's use that as our baseline. It looks like there was no change.

Runtime:

Native Code Metrics

I've been maintaining the C++ translation behind the scenes, trying to make sure we don't dig ourselves into a hole. This comparison looks OK:

But I noticed a regression versus version 0.9.3, released last month:

Looking over the results, I see many 255 exit codes that are supposed to be 0. I believe this is due a change that started truncating exit codes, which fixed a user-reported crash. I suspect it didn't translate to C++ correctly, e.g. because Python has arbitrary size integers and C++ has fixed size integers.

We have ~1000 more lines of native code:

But curiously the binary is slightly smaller: