Why Sponsor Oils? | blog | oilshell.org
I haven't published a blog post in nearly four months, but the Oil project is still alive!
I've been busy coding, trying to push out a polished 0.6.0 release. I thought that would happen soon after I announced that OSH can run thousands of lines of interactive shell programs.
But it turned out that there were a lot of features, fixes, and polish necessary on top of that, and I was also derailed by a performance problem.
I'll review most of that work in this post.
I've been struggling with the fact that making a usable shell is a lot of work. One way to think about it is that bash is a 30-year-old program consisting of 140K lines of C code, and Oil has to replicate most of that functionality [1].
On the positive side, early users have filed high-quality bug reports, which have kept me busy. I need more help like this! Read on for details.
Since announcing the 0.6.pre15 release in February, I've made 7 more pre-releases. Here I mention the general areas changed in each release, and link to the git log for those who want details.
I worked on more aspects of the interactive shell, e.g.
I statically typed the whole OSH front end with MyPy, which is around 10K lines of code. This work had four parts:
.pyi "header files" files for C extension modules.Josh Nelson implemented:
builtin builtin.$PS1 backslash codes.As mentioned in February, the shell is too slow, and I'm eager to fix that problem.
During this release period, I prototyped "mycpp", a program that walks MyPy's
typed AST and emits C++ code. This is one possible way to use types to
speed up the OSH interpreter.
It can translate many of the programs in the mycpp/examples/ dir accurately, and there are significant speedups in most cases.
I haven't yet applied mycpp to Oil's source, but I plan to shortly.
Note: I mentioned OVM2 last fall, but consider it dormant for now. I intend to take the shortest path to speed up the shell, and leveraging MyPy and a C++ compiler should get us there sooner than trying to write my own VM.
I made a pass over the whole codebase, checking that every error message is informative and has location information. These tests try to tickle every error:
Good news: OSH objectively has the most precise error messages of any shell. For example:
# opt=typo
set -o $opt
^~~~
foo.sh:2: 'set' got invalid option 'typo'
I'll show more demos of this with the 0.6.0 release. I decided that the headline for that release will be OSH Has Useful Error Messages.
Bad news: After this release, I noticed a performance regression running
Python's configure. After some debugging, it turns out that there was no
regression. Instead, there had been a bug in the benchmark for months.
What changed? This release fixed the behavior of the $LINENO variable.
Apparently, autoconf generates code to detect whether $LINENO
works. When it didn't work in OSH, the configure script exec'd itself with
/bin/sh, so it ran more quickly.
Fixing $LINENO means that OSH now runs the whole configure script, which
reveals the slowness. :-( I hope that the mycpp work will fix this problem.
{1..10..2}. As usual,
shells disagree on the semantics of this feature, and I made OSH stricter and
saner.set builtin behavior that autoconf scripts rely on.printf builtin, along with printf -v (an odd syntax for
assignment).
${}$(( )) and (( )) [[ ]] and most of [ ]More on that later.
Crestwave and others. Essentially, I backported PEP 475 -- Retry system
calls failing with EINTR to
Python 2.PYTHONPATH=foo myfunc. Thanks to
Michael Greenberg of the smoosh project for pointing this out. We
learned that these bindings might be the feature where POSIX shells disagree
most!unset. This builtin interacts
with shell's dynamic scope rule.Thanks again to the following people, mentioned above:
Crestwave - for testing and bug reports. I still want to run
the brainfuck interpreters in
shell :-)okayzed - for testing, and pointing out a bug with Oil's file descriptor
handling.drwilly for significant work on find and
xargs (not released yet).Please continue to test OSH and file bugs!
Oil's documentation is currently sparse, but I'm not losing track of important points:
oshrc startup file. OSH has a simpler
behavior than other shells, and I expect this to be one of the first issues
you have when starting to use it.Designs on the Wiki:
Developer tips:
I think Oil need more developers, but people are unlikely to become developers unless they're also users.
I've been suggesting that potential developers install OSH (which takes 30 seconds), and then:
oshrc that you would use. My configuration is described
on the wiki: How to Test
OSH. I suspect that
there will be friction, so let me know what happens on #oil-discuss on
oilshell.zulipchat.com.If you're already working with the code, and have problems figuring it out, please ping me on Zulip.
The README.md and Contributing wiki page have helpful tips.
I'm making consistent use of Github's issue tracker.
The #good-first-issue and #help-wanted labels generally up-to-date. Again, feel free to ping me on oilshell.zulipchat.com for ideas.
Other labels:
I went over many topics quickly in this post. If you're interested in more
detail, leave a comment or leave a message on #oil-discuss in
Zulip.
[1] Wikipedia says that the first release of bash was on June 8th, 1989. So its 30th birthday was a few days ago. Happy Birthday bash!
Although I plan to publish the FAQ first, I have drafts of the following blog posts:
mycpp uses explicit types. Somebody told me that they used Shed Skin in
production, which suggested that mycpp was feasible.fork()-friendly, like the Micro
Python garbage collector. If I succeed in
translating Oil to C++, I'll still have to solve the "deallocation
problem".If you have experience implementing garbage collection and Python-like data structures, I'd love to chat!