Why Sponsor Oils? | blog | oilshell.org
This is the latest version of Oils, a Unix shell. It's our upgrade path from bash to a better language and runtime:
Oils version 0.18.0 - Source tarballs and documentation.
We're moving toward the fast C++ implementation, so there are two tarballs:
INSTALL.txt in oil-*.tar.gz.See README-native.txt in oils-for-unix-*.tar.gz.If you're new to the project, see the Oils 2023 FAQ and posts tagged #FAQ.
We're moving toward retiring the Python tarball: OSH differs by 2 test cases in C++, and YSH differs by 77, down from 157 in the last release.
We're deep in the middle of implementing YSH: particularly functions, procs, and data languages. We're constantly testing and revising the language design.
This release "checkpoints" that work, as well as reflecting progress on every other part of the project.
Another highlight is the new Oils Reference, organized into two tables of contents, and 13 chapters. As YSH stabilizes, we'll document its behavior here.
Let's summarize what we've done, highlighting contributions. As always, we can use more help!
is-main builtin, with feedback from bar-g. (Described below.)joelatschool.The docs still need a lot of renaming from Oil → YSH. Feel free to open bugs on specific docs that are confusing.
Thank you for testing Oils!
Melvin did a big overhaul of job control, and fixed bugs like the lack of notification for sleep 1 &.
I fixed a long-standing escaping bug in shell auto-completion, issue #915.
git-completion.bash, without patches. I'll also show a screencast of this.argv languages — but I now realize that's for YSH. OSH has to be compatible with a somewhat quirky bash API.We register the name oils with GNU readline, not oil.
oil name with their ~/.inputrc.We can now export shell completions as JSON, including the bash completions that OSH runs:
compexport -c 'echo $HO' # completes shell language itself
# e.g. $HOME
compexport -c 'git ch' # runs git's bash plugin for completion
# e.g. checkout, cherry-pick
I claim that this is a missing feature in bash. For example, the git plugin registers itself with this line:
complete -o bashdefault -o default -o nospace -F $wrapper
But programs that want to scrape bash completion can only see the logic in $wrapper, not the three -o options. See Projects Already Doing Something Like Shellac for such programs.
So OSH compexport may be the mechanism for other shells to reuse bash completion!
This work is also still in progress, and only lightly documented. Please join our #shell-gui channel, and help us test it.
It was motivated both by testing our completion logic (escaping), and the headless shell. We want to export shell completions to a GUI. But again, I think other programs can also use it, rather than scraping bash and re-implementing parts of it.
Fixed our own "dogfood" issues:
Respect globbing in word arguments to redirects, matching bash and zsh:
$ tar -x -z < Python* # globbing occurs
# other shells disagree!
Fixed popd bug
Added is-main builtin
The is-main builtin returns 1 (false) if the current file was executed with the source builtin. It's designed to be used like Python:
if __name__ == '__main__': # Python
main(sys.argv)
OSH:
if is-main; then
main "$@"
fi
YSH:
if is-main {
main @ARGV
}
Related: #tools-for-oils > Tree Shaking - Static/Dynamic. Our bundling / tree shaking may work by dynamically walking dependencies, and cat-ting the result. It would have to rewrite source and is-main though.
help builtin to use the reference.
help builtin to C++.Still TODO:
Tuned the pool allocator that Chris Watkins implemented earlier this year. It was very pleasant to work with!
Instead of a single pool for objects under 32 bytes, we have two pools with thresholds of 24 and 48 bytes. In other words, these are members of our MarkSweepHeap:
Pool<682, 24> pool1_;
Pool<341, 48> pool2_;
malloc() header.The 24 bytes comes from the fixed-sized "head" of List<T>, which is the most common type in the program. We should also be able to fit the common Token in 24 bytes.
TODO: Tune our growth factors for Slab<T> so the first one fits exactly in 48 bytes.
Note: we tried other general purpose allocators like tcmalloc and mimalloc, but it's easy to do better on our workloads, measured with uftrace, cachegrind, perf, etc. I like that we have ~800 lines of code for our entire allocator and garbage collector, not 30,000!
Some color on how we arrived at this:
mut+alloc lines are close together:
parse.abuildex.compute-fibSo allocation is no longer a bottleneck, and GC rooting now sticks out as the next thing to optimize. There was also an unrelated slowdown due to more entries in our slow Dict, so we need to write a real Dict.
Some parts of the runtime are still "hilariously unoptimized", but we're making progress.
spec-cpp test delta for OSH is now 2, and for YSH it's 77.osh --tool and ysh --tool, to expose various tools to the command line.
osh --tool cat-em stdlib/math.ysh prints a file from our nascent standard library.--version.Tightening up our metalanguage:
Token instance without a GC header. It was easy to reproduce and fix with ASAN!try/finally.
finally, but C++ doesn't, all usages were bugs!I scrubbed the expression evaluator for consistency.
- and ~ do the same string/number conversions as binary operators.
+ always means numeric addition, and ++ is concatenation, not + as in Python and JavaScript.if and unary not are consistent with and or.More details on issue #1710. There are still a few more behaviors to consider, like tightening up chained comparisons.
YSH now allows shell-style myvar=x at the top level. I had disallowed it in favor of var myvar = 'x', but we now have a Language Design Principle of "not breaking the top level".
That is, these commands all work the same way in YSH, although with stricter error handling and no word splitting:
x=~/src
ls /tmp | wc -l 2>/dev/null
PYTHONPATH=. python3 -c 'print("hi")
What distinguishes YSH is really the compound commands: proc func, if case, and for while, not the top level.
The shell style is easier to type interactively, and is also useful for tilde expansion x=~/src. (YSH expressions don't have an equivalent of tilde expansion, since all strings are single- or double-quoted.)
On the other hand, myvar=x is disallowed in funcs and procs:
$ proc p { myvar=x; echo hi }
^~~~~~~
[ interactive ]:1: Use var/setvar to assign in YSH
Remember that the var foo = {age: 10} style lets you use typed data on the right-hand side.
This change allowed us to remove shopt -s parse_sh_assign, which is good because fewer global options is better.
Thanks to Aidan Olsen and Melvin Walls, all YSH functions and methods are now either:
Aidan:
error builtin...rest args to functions (aka "varargs")proc namespace.CommandParser and generated ExprParser.)source --builtin math.ysh
stdlib/math.ysh in the binarymin() max() abs(). We're testing the design of YSH by writing the "standard library" in YSH.Melvin:
extend()len() glob() split() join() maybe() ...bool() int() str() ...1:n range expression (which we plan to change to 1..n)As mentioned, we're deep in the middle of designing and implementing YSH. Here are Zulip threads that may be good starting points — they link to other threads.
#language-design > Remaining Parsing Changes
#oil-dev > YSH minimal Roadmap is the minimum we should do for YSH:
These 3 milestones are a lot of work, but definitely doable. We'll at least have a small, composable, "algebraically closed" YSH.
#language-design > YSH 2023 lists even more YSH features to implement. One way to think of it is that we're stuffing features from all these language categories together:
It all fits, and is useful! (Paradox: shell encourages polyglot programming, but YSH could make scripting more "monoglot". Reducing language cacophony seems overdue in shell.)
Here's a random sampling of more detailed Zulip threads. They reflect what we're thinking about.
We're nailing down design details as we implement features. I know it's dense, but informed feedback helps!
Decided:
append() is now a method on List, not a free function.append :mylist *.pykebab-case procs, and snake_case vars, etc.BigInt. This will knock off 2 remaining spec-cpp differences.Procs and funcs continue to be a difficult design issue. These threads cover a large chunk of our work in the next few months:
strip(x) and chaining x -> strip() -> upper().mutable=[] pitfall.proc and func.value.IO into functions to start processes, e.g. $(date) and ls | wc -l. This is way of making funcs pure.func and Hay. It's more of a pure evaluator.Also important:
const is "weak sauce" inherited from bash, and should be removed. Static const is better, though perhaps harder to implement in a shell.define at the top-level? This also relates to modules and tree-shaking.j"" prefix, and you can consider a b"" prefix as well. I'm not sure this is worth it.Threads related to modules / namespaces / tree-shaking:
We made progress on all fronts!
This release was delayed by about 2 weeks because I wanted to show Screencasts of an Interactive Shell.
That's the next post. We have concrete demos of things no other shell can do.
In the meantime, feel free to ask questions in the comments!
Some of the work we've done is reflected in these issues. You can also view the full changelog for this release.
| #1722 | Rewrite FUNCNAME BASH_LINENO BASH_SOURCE and fix minor bugs |
| #1716 | Allow shell assignment foo=bar at the top level in YSH |
| #1707 | Polish location info and fix bugs |
| #1703 | Trying to build CPP release from website, and getting errors from ninja-rules-cpp.sh |
| #1695 | [breaking] GNU readline app name is now "oils", not "oil" |
| #1687 | Initialization with rc files section of getting started doc references old filepath |
| #1521 | globs not expanded in redirects, and we use this in `deps/from-binary.sh` |
| #1500 | popd directory stack empty differs from bash |
| #1415 | Arguments to functions like eval_hay() aren't checked |
| #1362 | Enable try/except/else and try/finally errors in mycpp |
| #1163 | replace try/finally with context managers through the codebase |
| #1093 | No notification when 'sleep 1 &' finishes |
| #1011 | Add is-main builtin |
| #915 | oilshell escapes completion candidates |
| #532 | Improve the help builtin and toolchain |
These metrics help me keep track of the project. Let's compare this release with the previous one, version 0.17.0.
We improved OSH, and the 2 extra failures are TODOs on completion and compexport:
Translating the help builtin the C++ tarball very close to parity:
YSH got a lot of new behavior:
And the C++ tarball is catching up rapidly:
When we write our own JSON library, the delta should be almost zero.
The parser is faster due to the new pool allocator:
Parser memory usage increased slightly, which is a little surprising. My memory is that the pool allocator decreased memory usage, so this could have been something else:
parse.configure-coreutils
1.83 M objects comprising 62.1 MB, max RSS 68.9 MBparse.configure-coreutils 1.83 M objects comprising 65.0 MB, max RSS 69.3 MBA few more objects allocated at runtime, partially due to redirect args supporting globs (mentioned above):
configureconfigureOur fib benchmark got slower, and I tracked down the cause of it. It's due to translating more functions to YSH, and our poor Dict implementation! (Update: Melvin just brought this back down with a simple change.)
fib takes 61.6 million irefs, mut+alloc+free+gcfib takes 65.4 million irefs, mut+alloc+free+gcWe still have to catch up with bash here:
configureconfigure
configure. We still have to catch up with bash.OSH got a little bigger (and we still need to add YSH):
Translating YSH led to more C++ code in the tarball:
And more executable code:
Optimizing the GC rooting should bring the binary size down a bit.