blog | oilshell.org
Back in January, I wrote four posts tagged #comments. They were intended to introduce some big ideas quickly, in the form of a dialogue. I wanted to talk about distributed systems, #software-architecture, dev tools, and their relation to Unix shell.
The last post I published was Unix Shell: Philosophy, Design, and FAQs, and the last section previewed the rest of the series.
I didn't finish the series, but recent threads on lobste.rs and Hacker News address the same topics from a different angle. In this post, I walk through these more recent comments. They're concerned with:
I'm not sure how coherent this will be, but this angle is more concrete than the previous one.
This is a fantastic post about the design of modern CI services and the bad engineering practices they encourage.
A summary of my long comment:
Another point is that it's important to be able to debug the continuous build locally. Specifying build logic in an open source shell language is one natural way to do that.
My instinct was to implement our continuous build as a shell script, meaning that we avoid vendor-specific features.
This turned out to be a good choice, because Travis CI was acquired and is in the process of being shut down!
This comment goes into more detail on Toil. It also mentions that gg is also a multi-cloud system, which I discuss further below.
Not only do we have a problem with YAML, we also have the "shell in YAML" anti-pattern.
I think it’s going to be the year (and decade) of shell scripts written in YAML … Github Actions, Gitlab runners, Kubernetes config, sourcehut, etc. :)
My Comment on
Oil is very similar to existing purpose-built configuration languages, including Tcl, SDLang, etc.
My comment on
TODO: I've made at least 5 comments about Oil's configuration dialect, and it would be nice to collect them all in one place. But it would be even better to finish implementing it!
My comment on the Modern CI critique (on Hacker News this time).
Summary: I want to add the "missing declarative" part to shell. This will let users specify graphs naturally. But we also need metaprogramming, or a staged execution model, akin to:
Related: Python is NOT staged, but RPython is.
This section has various opinions on build systems. It's based on concrete experiences, but your use cases may differ.
I liked this blog post, so I'm saving it here. It describes a build system for boot images of embedded systems (deployed within a private company). It uses Ninja and OSTree together in a clever way.
I wrote several GNU Makefiles from scratch for Oil. This turned out badly, and I've concluded that I should have used Ninja all along. These two comments go into detail:
My comment on
My comment on
Ninja naturally invokes shell functions, and it can be generated by a Python script. I'd like to write those shell functions in the Oil language and write a Ninja generator in Oil :-)
The CI critique above mentions Bazel several times (because it has a nice DAG model, among other things). After a decade using it (and a small amount of time working on it), here are two problems with it:
This post introduced many ideas, and at first I thought they weren't that coherent. But now I see a clear connection, and even an implementation strategy:
Does that make sense? Let me know in the comments.
I'd love some help pursuing these ideas. There's too much to do on the Oil project, and I really should get back to work on the garbage collector (and the engine for an interactive shell).
But these ideas arise out of a concrete need, so they'll have to be addressed eventually. As a teaser, here's another post that shows the close relationship between shell and distributed systems: