Why Sponsor Oils? | blog | oilshell.org
This is the second Winter Blog Backlog post. It answers frequent questions about the project from a few angles.
(You may want to read the first post for context on this series.)
This blog has more readers than ever, and some of you wonder why Oil isn't done -- after more than 5 years! I wonder that myself, and I'm glad people want to use Oil. I want to use it too.
Here's one answer, and a sketch of a post I haven't written.
... then you have to do something different.
Programmers often complain that languages like shell, PHP, JavaScript, R, and C++ are messily evolved. They're inconsistent and have surprising pitfalls.
So Oil has a slightly unusual approach:
The first part has been successful. This blog generates a lot of feedback, and in response I fix bugs and change the language.
But I will likely need help with the translation, and I don't take that for granted. I hinted at expanding the project at the end of the last post, but I'm not sure what will happen.
The sections How OSH is Designed and How OSH is Implemented are also relevant -- they describe exactly what Oil does differently.
(Aside: I wrote PHP for the first time this year and like many aspects of it. I also regularly use JavaScript, R, and C++ -- all in the Oil project! The most useful languages often have the most pitfalls.)
Here's another way of explaining why Oil is a long project.
(1) Don't break the web is a slogan that appears in this informative history of JavaScript:
They also ask a provocative question:
Did the JavaScript committee spend 10 years doing the wrong thing?
That is, from around 1999 to 2008, the ECMAScript 4 committee tried to change JavaScript without respecting its past. This effort made little progress, and instead we ended up with the compatible upgrades ECMAScript 5, ES6, and so on.
There's an obvious analogy to shell and Oil here. Unlike many new shells, Oil runs existing shell scripts.
Here are two more classic debates in the foundations of software:
(2) Don't Break Windows apps. The Win32 API is stable and valuable.
(3) Don't Break Linux apps. The Linux syscall ABI is stable (which isn't true of other Unixes!). Container formats like Docker wouldn't work without this property.
What these links suggest is that continuity and stability are harder in the short term, but valuable in the long term. I think many experienced programmers understand this, but newer ones often don't. (There's such a thing as "motion without progress".)
So Oil is taking a long time because it has to recapitulate 50 years of shell history (Thompson shell, Bourne shell, Korn shell, and bash).
(These three links also strongly relate to the idea of narrow waists in computing, which I wrote about in posts tagged #software-architecture. I'll return to this subject in this blog series.)
On a related note, readers and contributors are often impressed by my stamina for the project. Here's sketch of another post, and second short answer.
This is a framing that came up during the first #oil-dev walkthrough meeting.
Oil basically started as a parsing experiment which succeeded in a few months. And then I kept going, and things kept working!
In the past, when I've worked on personal projects, there's been a "natural failure" point -- after a month, a year, or 4 years. I run into a problem I don't know how to solve, or something that seems like too much work.
But I broke down the Oil project in a way so that this has never happened!
Oil has had failures along the way, and that was the intention of the draft Five Years of Oil Milestones (And Some Wrong Turns), which I mentioned in the Summer Blog Backlog. But I'm deferring that post in favor of the summary below. Let's save the retrospective when Oil is more "finished".
(1) The Parsing Experiment. Can shell be statically parsed in a single pass? I started this experiment in April 2016, and by October 2016 I had written several posts tagged #parsing-shell.
It worked!
Moreover, the lexing and parsing model has not only held up to 5 years of bug fixes, but it's allowed us to morph shell into a better language. In the last post, I said the Oil 0.9.5 should be subtitled The Triumph of Lexer Modes.
(2) The Interpreter Experiment. Hold on -- can statically-parsed shell run real shell scripts? You don't know if your parser is good enough until you do that. It could be creating an AST that's nonsense.
The answer is yes!
configure
script.(Aside: we achieved that milestone 21 months after the start of the project, which feels reasonable. What doesn't feel reasonable is that it's been almost 4 years since then!)
(3) The Language Evolution Experiment. Can we design a good language on top of shell and bash? Can OSH be evolved into Oil?
Yes! I published A Tour of the Oil Language earlier this year. It describes Oil as a clean-slate, legacy-free language. And the code listings actually run and work.
The feedback has been positive, and it's driven many changes to the language. Thanks to all the readers on /r/oilshell, /r/ProgrammingLanguages, and lobste.rs who asked questions and gave feedback.
For more color on the language design, see the last post and posts tagged #oil-language. The latest success is that commands now take typed arguments, including blocks.
(4) Can an Interpreter Be Written in a High-Level Language like Python?
This experiment is ongoing, but all signs point to "yes".
In January 2020, I published Oil's Parser is 160x to 200x Faster Than It Was 2 Years Ago. It showed that our high level code can be translated to C++ that's as fast as existing shells, and even faster in some cases.
This process is taking longer than I expected. One reason for that is that I spent more time in a debugger than I wanted to. But note that Oil is simple and conservative compared to amazing projects like PyPy. With enough hands, it can easily be translated and optimized.
So three experiments succeeded, and the translation experiment can succeed. But that doesn't mean they worked on the first try. Here's a summary of a few mistakes.
(1) Wrong Turn: Translating Shell to Oil. I prototyped an automatic translator from OSH to Oil. This was impressive, but ultimately a dead end.
The requirement for automatic translation constrained and complicated the Oil language. For example, we had 4 or 5 assignment keywords to accomodate legacy semantics.
So the migration path now is based on a shared runtime, and global, scoped
options via the shopt builtin (doc in
progress).
(2) Wrong Turn: Cobbling together a Python Interpreter. The OPy subproject was a Python frontend and compiler intended to speed up Oil. I wrote a few blog posts about it in early 2017.
The speed was nowhere near good enough, which in retrospect should have been obvious! But I learned a lot by inadvertently repeating a tiny fraction of what PyPy did.
So our strategy is now based on mycpp, a more promising experiment that needs a rewrite.
(3) Mild Regret: Spending too much time trying to bootstrap. I still think that "lifting" some of our metalanguages to the user level is a good idea, but it was much too early.
In summary, the project could have failed many times, but didn't. That's surprising and motivating!
Here's a second answer to "Why Do You Keep Working On It?".
I made a big list of alternative shells to survey the landscape, and none of them is what I want, or what I think the open source ecosystem needs.
In recent discussions with the Nix team, this became apparent. They have a boatload of shell that the maintainers generally don't understand. The only options are to use bash forever, or migrate to OSH.
The upcoming posts on #software-architecture will also explain how Oil is fundamentally different than other shells.
This post answered common questions about the project in a few different ways.
Why isn't it done?
Why do you keep working on it?
Let me know if you have more questions.
I should add that when I get tired of working on Oil, I take a break. I was traveling a lot this year, and took more than a month off at least twice. (This explains the minor lulls in the release history.)
But I'm also humbled by the amount of work it's taken. It feels like time to expand the project to more contributors, which again I brainstormed about in the last post. If you know a qualified person who'd like to get paid for this kind of work, let me know.
I have three more drafts in this series: