Why Sponsor Oils? | blog | oilshell.org
As promised two weeks ago, here's a new OSH release:
Please try it both interactively and on batch programs! This post gives further detail on how to test OSH, and what's useful to test.
To build and run it, follow the instructions in INSTALL.txt. Please report bugs on Github.
If you're new to the project, see Why Create a New Shell?.
The strict-argv
feature in this release was motivated by a recent
thread on the help-bash mailing list. An excerpt regarding error
messages:
... that's just bash's way of lulling newbies into complacency so that their scripts blow up LATER instead of IMMEDIATELY. Obviously a script that APPEARS to work has a better chance of going into production in a broken state.
This maximizes the chances of chaos and destruction. Bash craves these things. The suffering of fools is its lifeblood.
— Greg Wooledge, maintainer of BashFAQ
This suggests some new slogans:
Let me know if you have other ideas :-)
Feedback from the BayLISA talk confirmed that error messages are important, so it's good to know that Oil is on the right track. The release process includes a script that tickles all parse errors and all runtime errors. They're better than bash error messages, but could use another pass of polish. Some location info is missing.
Speaking of metaphors, I learned of the Augean stables in a recent lobste.rs thread. I also got some good ideas for an Oil logo from Rory O'Kane.
This section summarizes the git changelog for 0.6.pre14 and 0.6.pre15:
I released OSH twice in the last two weeks due to meaningful contributions and testing:
alias
can now contain a pipeline, as well as
several other constructs. Many more spec tests pass.$HOSTNAME
variable.Thanks to wolverian
, RobSykes
, tyxieblub
(on Github), and tekknolagi
(on Reddit) for the bug reports!
In addition, tyxieblub
improved bash-compatible prompt support:
\w
and \W
expand to proper variants of the current working directory.\h
and \H
expand to proper variants of the hostname.Inspired by help-bash threads like the one above, I implemented set -o strict-argv
.
When it's on, OSH will fail when a command evaluates to an empty argv
array, which can happen due to the elision of unquoted, empty words:
osh$ set -o strict-argv osh$ if $unquoted_empty_var; then echo true; fi Line 1 of '' if $unquoted_empty_var; then echo true; fi ^~~~~~~~~~~~~~~~~~~ fatal: Command evaluated to an empty argv array
test -n "$empty_var"
.$(echo x >/dev/null)
— an
unquoted command sub that evaluates to the empty string.A future post will describe other OSH-specific options:
strict-errexit
strict-control-flow
strict-word-eval
I plan to add strict-ALL
to opt into all of them in a single line.
I've already written about providing more parse errors. The strict
options are about providing more runtime errors.
Although I've already announced the 0.6.pre13 release, I hope that these additional details suggest areas to test. OSH needs more testing on real shell scripts!
OSH now runs Python's virtualenv.
I discovered that backticks are parsed differently than $()
, and
virtualenv relies on this odd behavior. I fixed it by adding a lexer
mode and another pass of parsing. (Complete list of lexer
modes).
OSH now runs git-prompt.sh
.
C-escaped string literals within arguments to variable expansions are now
parsed properly. For example, in "${sep:-$'\t'}"
, the TAB literal $'\t'
is an argument to the :-
operator.
The read
builtin now sets variables with dynamic scope.
OSH runs many (but not all) bash-completion plugins.
compgen -W
parses its argument as code, evaluates it, and splits it
with $IFS
.compgen -X <pattern>
properly composes with other flags.In addition, I implemented a "devtools" feature that uncovered many bugs:
OSH_HIJACK_SHEBANG=/usr/local/bin/osh
is set in the
environment, then OSH will run #!/bin/sh
and #!/bin/bash
scripts with
that shell interpreter. This allows deep testing of scripts that invoke
other scripts, without editing code.These changes also went in the 0.6.pre13 release:
okayzed
originally pointed out. I also discovered
that I should use an fnctl()
flag to do this in the kernel, rather than
in user space.pushd
and popd
, including the update of $PWD
.errexit
in Oil's own shell test harnesses.
strict-errexit
flag mentioned above
could be even stricter. Functions and external commands should behave
differently with respect to errors.a[i++]=foo
and backticks, since they
unfortunately require another pass of parsing.
alias
bad, making
two passes of static parsing is also bad. It makes it harder to
generate good error messages (although a solution for this problem is
coming.)Also:
ENOEXEC
when an executable file
doesn't contain native code (ELF) or doesn't start with a shebang line. All
shells handle this error by interpreting the file themselves, as if it were a
shell script.I've mentioned a few times that Oil is too big and too slow. This is still an open problem, but I believe that the first step to make it faster is add static types to the code.
Luckily, the MyPy project exists, and has undergone heavy development since I last tried it in 2016. Oil is also more mature, which means I can list every place that it uses metaprogramming/reflection, and generate textual code instead.
In other words, the tradeoff between type checking vs. metaprogramming has changed in the past 2+ years.
This release made progress by compiling ASDL schemas into Python code that
passes mypy --strict
. We now run MyPy on Travis as well. There's still
much more code to annotate with types, but this is a first step.
(Recall that ASDL schemas define the lossless syntax tree, as well as important runtime data structures.)
I hope that the long list above gave you a feeling for what OSH can do. It runs real programs, but there are undoubtedly missing features and undiscovered bugs.
You can test it on both interactive and batch programs. I started a How To Test OSH page on the wiki. Summary:
bin/osh
behaves like bin/sh
or bin/bash
, which makes it easy to test.~/.bashrc
is ~/.config/oil/oshrc
. You can make an
~/.oshrc
symlink if that becomes annoying to type.dev/osh-release-0.6.pre15
branch. I removed some exotic bash features and simplified the code. The
next post will explain this in detail.OSH_HIJACK_SHEBANG=/usr/local/bin/osh
for deep testing (mentioned
above).--debug-file
flag to show log messages and tracing.Feedback is welcome in any of these ways:
A shell has two main dependencies: libc and readline (a line-editing library). In both cases, OSH has minor GNU dependencies. They should be removed, but for now here are two notes.
OSH now builds again on OS X, but bash-completion plugins won't work. They use extended globs, and OSH relies on GNU libc to parse that syntax. In contrast, bash has its own implementation of extended globs — which is slow, according to the manual.
OS X and distros like Alpine Linux do not use GNU libc, so extended globs and completion scripts that use them won't work.
The work described in issue 192 should fix this, and it involves some fun computer science. On the other hand, I'm not sure I know how to do it, and it might be OK to do something hacky like bash. Please chime in if you're familiar with the implementation of regular expressions.
The Oil language won't have extended globs, since they're redundant with regexes.
Hitting TAB
for completion doesn't work on OS X.
I've made some changes to CPython's readline wrapper, which probably broke
libedit
support on OS X. On the other hand, the readline support in Python
2.7 my Mac seems broken anyway, but I didn't look into it further.
If you have any expertise in this area, let me know.
0.6.0
after:
Let's compare the November release of O.6.pre8 with the current release. I reviewed metrics for the former in Hollowing Out the Python Interpreter.
65 more spec tests now pass, e.g. for completion builtins and other interactive features:
There was a slight regression on the million lines of shell, due to backtick parsing mentioned above:
But all the new interactive features cost us less than 1000 significant lines of code:
The story is similar when including whitespace and comments (physical lines):
For reference, bash has ~88K significant lines of code and ~124K physical lines of code (line count script).
I'm happy that the OSH source code has stayed small. It gives me confidence that I can make it both faster and smaller without rewriting all the code.
I hope the MyPy / C++ translation will make these metrics obsolete, but here are some minor updates:
There was a slight increase in native code:
along with a bigger increase in binary size:
This was caused by removing linker flags to remove unused code. The flags apparently aren't portable, and this metric isn't very important now. OSH needs to be drastically sped up in some other fashion.
The bytecode size went up nearly 200 KB:
This is due to the compilation of ASDL schemas to MyPy
(mentioned above). The pretty-printing methods used in osh -n
are expanded
to type-specific code. syntax_asdl.pyc
is now 200 KB by itself, as
opposed to 47 KB when it was called osh_asdl.pyc
.