Why Sponsor Oils? | blog | oilshell.org

After 8 Years, Oils Is Still Small and Flexible

Let's do something hard, and go all the way back to the first post on the project:

• What Oil Looks Like, and a Plan for This Blog (2016)

Let's see how much code we've added, and let's see if the ideas made sense. There's no better test than reading and evaluating what you wrote years ago :-)

Our Code in 2016, versus 2024

In 2016, I showed this summary of the code:

PYTHON SKETCH
...
  1044 sketch/word_parse.py
  1299 sketch/cmd_parse.py
 10315 total

SHELL TESTS
...

This was 6 months into the project, and we had 10 K lines of Python code, and many tests.

That report evolved into the ones I publish on the release quality page:

• line-counts/overview.html - Physical lines by component: wc -l
• line-counts/cloc-report.txt - Significant lines of code (non-whitespace, non-comment)
• line-counts/oils-cpp.txt - Lines in the oils-for-unix tarball.
  • The C++ tarball is 2x - 50x faster than Python, and I started recommending it in March.
  • It's pure C++ generated from DSLs, which is the middle-out style.

Comparison Table

Let's arrange these numbers in columns:

I like this! We have 64 K physical lines / 35 K significant lines in the major components of the project: OSH, YSH, J8 Notation, and the C++ runtime.

All of Oils — including YSH and J8 Notation — has less source code than bash (~142 K lines).

This is despite the fact that YSH has "real" data structures, garbage collection, and more. (The next post will emphasize this.)

And it's not just our source code that's smaller than bash, but our generated code is too. This matters because we read, debug, and profile it.

Project is Big, Code is Small

So the last post showed that the Oils project is big, but now we see that its source code is small.

The appendix links to selected source files, which may give you a feeling for why this is.

(Caveat: I'm counting only Python and C++ code, which is ~7 out of the 13 parts. I'd like to join and fully automate the 3 line count reports, to account for all 13.)

Review

What's Changed in 8 Years?

The table of line counts suggests how the project has changed.

• YSH now exists.

  • I started it in 2019, and it's currently our main focus. It's the "non-negotiable" part of Oils.

• J8 Notation now exists.

  • It upgrades ad hoc text formats into something more principled. We learned that upgrading shell code is not enough.
  • The first design was done in 2020, and the second in 2023.

• We have 8 years of features and functionality, but not 8 years of code.

  • If we'd kept the pace of 10K physical lines in 6 months, then we'd have 160 K lines. But we now have 64 K lines of source code.

• Why is the code short? Because mycpp now exists.

  • It translates a subset of typed Python to readable C++. This means we can ship less than 9 K lines of hand-written C++ code (numbers in red).
  • mycpp was started in 2019. (It needs a rewrite, but this isn't critical.)

The First Post Was An Apology for Python

It's funny to me that the first post can be read as an apology for showing Python code, not C++:

I actually started writing it in C++. But after getting to 3K lines of code in the spring, it began to feel onerous.

I also hinted at what was to come:

Or even better than porting is to use Python as a metaprogramming language for C++.

After some diversions and missteps, this largely came true. We now have a nice situation:

1. We write little C++ by hand, preferring to write DSLs and typed Python instead.
2. We generate both Python and C++ from the DSLs, giving two complete implementations.
3. The C++ tarball has the speed of native code, and no dependencies (e.g. on the Python interpreter).

This is what I call the middle-out style. But it certainly took a long time to get here.

Was the Middle-Out Style Worth It?

I think so, but it's hard to argue that in a short space. For now, I'll abbreviate the argument with some slogans:

• Speed is Hard, and C is Fast - one reason it took a long time.
• Comments on Making Software Over Time - a separate blog post with quotes.
  • I knew Oils would be large, and I wanted to avoid typical software rot.
  • We're 8 years in, and I think we're still in good shape.

Benefits of the Middle-Out Style:

• Two Implementations Is Like a Spec
  • Having two complete implementations enforces discipline.
• Metalanguages Provide Leverage for Optimization
  • Melvin's current work on mycpp is taking advantage of this!
• The Metalanguage Influences the Language
  • For example, I argue that the design of C influenced the design of shell.

A slight surprise:

• Pleasant and Reliable Refactoring
  • Typed Python with Zephyr ASDL is a bit like OCaml or Java, in that you can do aggressive global refactoring. This makes the codebase clean and consistent!
  • We're still taking advantage of this. The command.Redirect refactoring in Oils 0.23.0 made the interpreter faster.

Conclusion

Oils is a big project, with 8 years of functionality, but it's a small codebase. And that was always the goal!

What's next? I extracted two posts from this one:

• Garbage Collection Makes YSH Different - a table of languages
• Comments on Making Software Over Time - some opinions and feelings

This was the original plan for the series:

1. What Oils Looks Like in 2024
   • The project is big, and we can use help
2. After 8 Years, Oils Is Still Small and Flexible
3. Missing Retrospectives on Oils
4. Oils - Grand Ideas and Fiddly Details
5. Oils 0.23.0 - User Feedback, Bug Bounty, and Writing YSH Code

Appendix: The Densest Source Files

Why is our code short? I publish selected source files with every release, and they may give you a feel for this:

• _gen/frontend/id_kind.asdl_c.h - generated

  • 402 integers that denote all syntax elements in Oils.

• _gen/_tmp/match.re2c-input.h - 1277 lines, generated

  • The Oils lexer is a big mapping from (lexer mode, regex) → ID.
  • Compared with bash, it saves at least 10,000 lines of hand-written C code. Dan Kegel liked this on Mastodon!

• frontend/syntax.asdl - 653 lines

  • The lossless syntax tree expresses all Oils syntax with algebraic data types.

• ysh/grammar.pgen2 - 538 lines

  • The YSH grammar is derived from Python's grammar.

• core/value.asdl - 174 lines

  • OSH and YSH share a common data model. Our equivalent of PyObject* is value_t.

Let me know if you need help reading these files! Together, they form a concise description of the many interleaved languages in Oils. They're a big part of what I think of as the executable spec.