Why Sponsor Oil? | blog | oilshell.org

Oil 0.11.0 - Big Features and Project Changes

2022-07-28

I released Oil 0.11.0 last month, with the biggest collection of features in years. The top feature was Hay, an extension to Oil that lets you use Ruby-like blocks to create internal DSLs. This is what I've wanted Oil to be all along — a language to build a simpler cloud!

It got good feedback on lobste.rs. So I wanted to circulate it more widely, and help people try it. But I immediately switched gears to translating Oil to garbage collected C++, working closely with our new "hired contributor" Jesse.

That was another solid month of work, contained in the subsequent Oil 0.12.0 release. So we have two big releases to talk about. This is an announcement for the first, but the second is also worth writing about later.

Oil version 0.12.0 - Source tarballs and documentation.

Table of Contents
Project Recap and Update
Please Sponsor Oil
Summary of Changes
Big Feature: Hay Ain't YAML
More Features
Magic Phrase: shopt --set oil:upgrade
Doc on Upgrading to Oil
Polishing the Oil Language
Under the Hood
New Contributor Docs
Conclusion
We Should Collaborate More
Acknowledgements
Details
Closed Issues
Commit Log
Appendix: Metrics for the 0.11.0 Release
Spec Tests
Source Code Size
Benchmarks
Native Code Metrics

Project Recap and Update

All this work has meant that blogging has fallen by the wayside. So here's a quick recap of where we are.

First, the project's scope has increased and outgrown me. I wrote about this in March: Oil Is Being Implemented "Middle Out" .

As of April, we have an NLnet grant to pay for specialized help, particularly in Europe.

And we now have one paid contributor (Jesse), but he is not European! And I've talked with several others who are interested, skilled, and available.

As a result, the C++ code is in better shape, and easier to work with. I greatly improved the build, translation, test, and CI infrastructure.

I've also reached out to authors of other alternative shells to "join forces".

Please Sponsor Oil

But we need your help. NLnet prefers to pay engineers in Europe, which is understandable, because it's ultimately funded by European taxpayers! But we have interested programmers from the United States, Canada, Australia, Russia, and Brazil.

This means that we don't have money to pay them, and even without the geographical restriction, the grant wouldn't be enough.

The other source of funding we have is Github sponsors. So please sponsor Oil if you're excited about the project.

If you can't for some reason, consider telling your friends about Oil!

The progress I described means that this money will be converted into working, open-source code. That wasn't obvious three months ago, but it's clearly true now.


We have 5 sponsors at the moment, who I really appreciate, but it's not enough!

As of this week, the Oil project is $5000 in debt to me personally. Although it's not the top funding priority, I hope to recoup this somehow — Oil is supposed to be an open source project, not a charity project funded by my personal savings :-)

(Note that because it's open source, most contributors aren't paid. The funding is for specialized skills, i.e. Compiler Engineer Job — analogous to how both Python and Rust were funded for long periods. I've learned the hard way that designing and implementing a language requires sustained effort among many people.)

Summary of Changes

Now on to the features in this release.

Big Feature: Hay Ain't YAML

This feature is novel and deserves good documentation. Here is a start:

Hay - Custom Languages for Unix Systems

To repeat the slogans:

Again, this gets at the core of what I want the Oil language to be! I also wrote this wiki page to help users understand the space of possible solutions:

The lobste.rs thread I mentioned also has substantive comments and viewpoints.

Please play with it and send feedback on Zulip or Github.

More Features

But that's not all!

(1) Oil now has a for loop variant for typed data.

(2) Dictionaries now retain the insertion order of pairs, as recent versions of Python do. This makes JSON serialization more predictable.

(3) For readability, the new fopen builtin lets you attach redirects to the front of a block:

fopen >out.txt {
  echo one
  echo two
}

I added a section in Oil vs. Shell Idioms about this.

(4) The shopt builtin now respect groups like oil:all.

shopt --set strict:all {
  my-func  # run with more errors
}
other-func  # fewer errors

Magic Phrase: shopt --set oil:upgrade

Oil's slogan is that "it's our upgrade path from bash". Prior to this release, the way to upgrade was with the oil:basic option group. It lets you use new features, but doesn't require you to fix strict errors and remove discouraged idioms. (If you want that use, strict:all or run bin/oil.)

Thanks to feedback from ca2013, I renamed oil:basic to oil:upgrade.

So now shopt --set oil:upgrade will be one of our primary memes, which makes a lot more sense.

I'm surprised that this "naming mistake" persisted for so long! Again, if you're interested in a new shell language, please try Oil and send feedback. I know the documentation needs improvement, and we're making some strides there. (Again, sponsorship helps.)

Doc on Upgrading to Oil

I also wrote this doc with the help of ca2013:

What Breaks When You Upgrade to Oil

Surprisingly little breaks! I feel good about this. Not only is OSH the most bash-compatible shell by a mile — it won't be that hard to upgrade to Oil either.

You'll get better error handling; you won't have to quote everything; you'll get Python-like expressions on typed data, as well as Ruby-like blocks. But everything else will mostly work.

(Related comment: Evolution or Clean Slate? Choose both.)

Polishing the Oil Language

More changes:

(5) I changed the recommended syntax for the case statement.

My thought was that when using the Oil language, you should balance parentheses and quote constant strings:

case $name {
  # left paren is optional in shell, but required in Oil
  (*.py)
    echo 'Python'
    ;;
  ('README')  # quotes required to avoid confusion with variable
    echo 'README'
    ;;
}

The reasoning is that parens introduce expressions in Oil, like if (x > 0). But case statements already use parens to means something different, and I don't want to confuse (x) and ('x'), regardless of context.

On second thought, I think we can do even better, removing the two meanings of (), with something like this:

case $name {
  | *.py >
    echo Python
  | README >
    echo README
}

This should make it easier to implement syntax highlighting for Oil, and I think it's nice to get rid of shell's unusual ;; terminator.

Thanks to Kazuma Arino for starting a VSCode Extension for Oil with syntax highlighting!

I'm reluctant to change syntax this late in the game, but shell's case syntax will still work with bin/osh, and I don't want to live with an inconsistent case statement in bin/oil. So a future release should implement this new idea. (As always, feedback is welcome.)

(6) I worked out many tricky design issues regarding =. This had been in flux for awhile.

Package grep {
  version = '3.1'                 # no var or const
  maintainers = ['alice', 'bob']  # typed data on RHS
}

(7) The proc keyword is no longer present in bin/osh. It must be enabled with the parse_proc option, which is part of bin/oil.

(8) parse_dparen is off in Oil and disallows the bash construct (( .

(9) Bug fix: data is allowed as a variable name. Thanks to Kelly Brazil for reporting this.

Under the Hood

We're making it easier to contribute to Oil:

(1) The master branch is now always green (passes a large test suite).

How is this implemented?

So this implements the NRSROSE rule (merge after testing, not before). We do it with a simple shell script on Github Actions, which I'm pretty happy with!

(2) Thanks for Geoffrey Huntley, we now have Gitpod support.

(3) Reducing friction


Future blog posts should cover two parts of our infrastructure, which directly relate to the new Hay feature:

  1. Enhancing Soil, our multi-cloud CI system, which currently uses YAML and shell!
  2. We're using Ninja with Python, shell, and C++. Using GNU Make was a mistake.

Related: Comments About Build Systems and CI Services (April 2021).

New Contributor Docs

In addition to the Hay and upgrade breakage docs, there are new wiki pages to help contributors:

Conclusion

I'm happy with the new features in this release, and excited about having help. This has enabled rapid progress on translating Oil to C++. (Teaser: the C++ translation is passing ~1680 of 1789 spec tests, up from 1495 in May.)

But even though Hay is a long time coming, it feels like just the beginning! I think we need more help on the Oil language, not just with C++ translation.

We Should Collaborate More

As one example, I had a nice chat last month with Alex of Shell++, an impressive alternative shell (which was started in 2016, simultaneously with Oil!)

I wanted to make the case that Oil is a good "Shell++", and he agreed! Oil's strategy of a smooth upgrade path is the best way to "replace" bash.

I also learned several things from transcribing Shell++ example code to Oil. It revealed holes in the Oil language, and vice versa.

So I'd like to invite the authors of other alternative shells to collaborate on Oil. We had some conversations way back in 2017 and 2019, but we should renew them. (We can use Zulip or another tool.)

Oil is more mature now, and I believe it has a solid foundation for more growth.

To pick some concrete examples, I think that both the ABS language and the Koi language are extremely similar to Oil and Shell++, and separate efforts are something of a wasted opportunity!


Reminder: If you like the changes in this release, and the direction of the project, please sponsor Oil! We have real people to pay now.

As always, let me know what you think about any of this in the comments.

Acknowledgements

Again, thanks to ca2013, Kazuma Arino, and Kelly Brazil for feedback and help.

Thanks to Timothy Sample and Danilo Spinella for great discussions on Hay.

Thanks to Jesse Hughes and Chris Watkins for help on Oil 0.12.0, which I want to write about next.

Details

Here is more detail on the work above.

Closed Issues

#1149 Disable proc keyword when no options are set
#1140 allow temp bindings even in bin/oil (PYTHONPATH=. foo.py)
#1139 Better error message: typed args vs. parse_equals
#1137 Rename oil:basic -> oil:upgrade
#1135 for loop enhancements: over typed data, and with index / key / value
#1134 shopt --unset parse_bare_word for case and for
#1132 Disallow shell style assignment in Oil
#1131 'data' should be allowed in expressions (soft keyword or remove it altogether)
#1114 Require left paren in case clauses - case $x in ( foo )
#1104 flaky test/stateful cases
#1099 shopt block syntax should accept oil:all
#1021 Dicts should retain insertion order (add spec tests)
#1016 (( bash construct conflicts with Oil on occasion, may need to be ( (
#863 fopen builtin
#745 continuous build: merge to master on green

Commit Log

These are commits from other contributors. You can also view the full changelog.

ec7ccb32 Geoffrey Huntley Enable software development from any device via https://gitpod.io (#1156)
c40c8950 Justin Hong [build] Fix macOS build from source failing at make step (#1162)

Appendix: Metrics for the 0.11.0 Release

These metrics help me keep track of the project. Let's compare this release with the last one announced, version 0.10.0.

Spec Tests

The spec test suites for both OSH and Oil continue to expand and turn green.


The significant expansion in Oil is due to Hay!

Source Code Size

Despite the new features, the code is still compact. Significant lines:

Physical lines:

Benchmarks

The oil-native parser got a bit slower:

Hm, this is the first change in quite awhile. It may be due to the for loop and case statement changes.

This test of the slow Python build uses wall time and is thus very noisy:

Native Code Metrics

As mentioned, we've made good progress in the last month:

This nice reduction is due a code cleanup:

We now match constant strings like "parse_proc" with a plain C++ function, rather than using re2c.

Binary size:

This is also a nice reduction, due in part to the same change. We also changed compiler and linker flags to remove unused symbols. As mentioned, I want to write about the new Ninja build.