blog | oilshell.org

OSH 0.6.pre15 Does Not Crave Chaos and Destruction

2019-02-18

As promised two weeks ago, here's a new OSH release:

Please try it both interactively and on batch programs! This post gives further detail on how to test OSH, and what's useful to test.

To build and run it, follow the instructions in INSTALL.txt. Please report bugs on Github.

If you're new to the project, see Why Create a New Shell?.

Table of Contents
Two Colorful Metaphors
Recent Changes
User Contributions And Testing
A New Strict Option
Features That Allowed Dogfooding
Bugs Fixed While Dogfooding
ASDL Schemas Are Now Compiled to MyPy
What To Test and How to Test It
Portability Notes
Extended Globs Rely on GNU libc
Completion Relies on GNU Readline
What's Next?
Appendix: Selected Metrics
Native Code and Bytecode Metrics

Two Colorful Metaphors

The strict-argv feature in this release was motivated by a recent thread on the help-bash mailing list. An excerpt regarding error messages:

... that's just bash's way of lulling newbies into complacency so that their scripts blow up LATER instead of IMMEDIATELY. Obviously a script that APPEARS to work has a better chance of going into production in a broken state.

This maximizes the chances of chaos and destruction. Bash craves these things. The suffering of fools is its lifeblood.

— Greg Wooledge, maintainer of BashFAQ

This suggests some new slogans:

Let me know if you have other ideas :-)

Feedback from the BayLISA talk confirmed that error messages are important, so it's good to know that Oil is on the right track. The release process includes a script that tickles all parse errors and all runtime errors. They're better than bash error messages, but could use another pass of polish. Some location info is missing.


Speaking of metaphors, I learned of the Augean stables in a recent lobste.rs thread. I also got some good ideas for an Oil logo from Rory O'Kane.

Recent Changes

This section summarizes the git changelog for 0.6.pre14 and 0.6.pre15:

User Contributions And Testing

I released OSH twice in the last two weeks due to meaningful contributions and testing:

Thanks to wolverian, RobSykes, tyxieblub (on Github), and tekknolagi (on Reddit) for the bug reports!

In addition, tyxieblub improved bash-compatible prompt support:

A New Strict Option

Inspired by help-bash threads like the one above, I implemented set -o strict-argv.

When it's on, OSH will fail when a command evaluates to an empty argv array, which can happen due to the elision of unquoted, empty words:

osh$ set -o strict-argv
osh$ if $unquoted_empty_var; then echo true; fi
Line 1 of ''
  if $unquoted_empty_var; then echo true; fi
     ^~~~~~~~~~~~~~~~~~~
fatal: Command evaluated to an empty argv array

A future post will describe other OSH-specific options:

I plan to add strict-ALL to opt into all of them in a single line.

I've already written about providing more parse errors. The strict options are about providing more runtime errors.

Features That Allowed Dogfooding

Although I've already announced the 0.6.pre13 release, I hope that these additional details suggest areas to test. OSH needs more testing on real shell scripts!

In addition, I implemented a "devtools" feature that uncovered many bugs:

Bugs Fixed While Dogfooding

These changes also went in the 0.6.pre13 release:

Also:

ASDL Schemas Are Now Compiled to MyPy

I've mentioned a few times that Oil is too big and too slow. This is still an open problem, but I believe that the first step to make it faster is add static types to the code.

Luckily, the MyPy project exists, and has undergone heavy development since I last tried it in 2016. Oil is also more mature, which means I can list every place that it uses metaprogramming/reflection, and generate textual code instead.

In other words, the tradeoff between type checking vs. metaprogramming has changed in the past 2+ years.

This release made progress by compiling ASDL schemas into Python code that passes mypy --strict. We now run MyPy on Travis as well. There's still much more code to annotate with types, but this is a first step.

(Recall that ASDL schemas define the lossless syntax tree, as well as important runtime data structures.)

What To Test and How to Test It

I hope that the long list above gave you a feeling for what OSH can do. It runs real programs, but there are undoubtedly missing features and undiscovered bugs.

You can test it on both interactive and batch programs. I started a How To Test OSH page on the wiki. Summary:

Feedback is welcome in any of these ways:

  1. File a bug on Github
  2. Leave a Reddit comment if you have problems testing.
  3. Or post a message on oilshell.zulipchat.com

Portability Notes

A shell has two main dependencies: libc and readline (a line-editing library). In both cases, OSH has minor GNU dependencies. They should be removed, but for now here are two notes.

Extended Globs Rely on GNU libc

OSH now builds again on OS X, but bash-completion plugins won't work. They use extended globs, and OSH relies on GNU libc to parse that syntax. In contrast, bash has its own implementation of extended globs — which is slow, according to the manual.

OS X and distros like Alpine Linux do not use GNU libc, so extended globs and completion scripts that use them won't work.

The work described in issue 192 should fix this, and it involves some fun computer science. On the other hand, I'm not sure I know how to do it, and it might be OK to do something hacky like bash. Please chime in if you're familiar with the implementation of regular expressions.

The Oil language won't have extended globs, since they're redundant with regexes.

Completion Relies on GNU Readline

Hitting TAB for completion doesn't work on OS X.

I've made some changes to CPython's readline wrapper, which probably broke libedit support on OS X. On the other hand, the readline support in Python 2.7 my Mac seems broken anyway, but I didn't look into it further.

If you have any expertise in this area, let me know.

What's Next?

Appendix: Selected Metrics

Let's compare the November release of O.6.pre8 with the current release. I reviewed metrics for the former in Hollowing Out the Python Interpreter.

65 more spec tests now pass, e.g. for completion builtins and other interactive features:

There was a slight regression on the million lines of shell, due to backtick parsing mentioned above:

But all the new interactive features cost us less than 1000 significant lines of code:

The story is similar when including whitespace and comments (physical lines):

For reference, bash has ~88K significant lines of code and ~124K physical lines of code (line count script).

I'm happy that the OSH source code has stayed small. It gives me confidence that I can make it both faster and smaller without rewriting all the code.

Native Code and Bytecode Metrics

I hope the MyPy / C++ translation will make these metrics obsolete, but here are some minor updates:

There was a slight increase in native code:

along with a bigger increase in binary size:

This was caused by removing linker flags to remove unused code. The flags apparently aren't portable, and this metric isn't very important now. OSH needs to be drastically sped up in some other fashion.

The bytecode size went up nearly 200 KB:

This is due to the compilation of ASDL schemas to MyPy (mentioned above). The pretty-printing methods used in osh -n are expanded to type-specific code. syntax_asdl.pyc is now 200 KB by itself, as opposed to 47 KB when it was called osh_asdl.pyc.