Home

Post-Release Details

Summary

There are two ways of looking at OSH:

1. OSH is a vehicle for Oil. I'm implementing OSH to make sure I understand the execution model of bash, and to make sure that Oil subsumes all its features.
2. OSH is production-quality bash clone. It's better because it's statically parsed, has better error messages, and has a better interactive completion API.

I started with the first view. While I've been working on [OSH][osh-language] for the last six months, I've been entertaining the second view.

But not only am I eager to work on Oil, I think it's more interesting for users right now. The slogan I've been using in my head is:

• I'm respecting the past, not calcifying it.

I already did a draft of Oil-Quick reference and I fixed up the osh2oil tests!

(3) New Roadmap; The Top Priority. I want to make a prototype of the Oil

More specifically:

• I will write a lexer, parser, and LST for Oil. This shouldn't be hard, since I have a clear picture of the Oil language, and I've already implemented a lexer, parser, and LST for OSH.
• Oil will somehow use the same runtime and execution model as OSH. I'm not sure exactly how this will happen, so that's a risk.
• The OSH-to-Oil translator will be improved.

To be specific, I'll implement the part of the Oil language that's analogous to [bash][]. Oil will have other features like structured data types and declarative constructs, but I may not get to those by the end of the year.

I need to come up with a name for this subset of Oil, perhaps "Oil B". (B is for bash.) Suggestions are welcome.

(4) Elevator Pitch for Oil. I haven't been doing a great job of

Notice that this is entirely about shell programming. It leaves out the interactive shell.

On the one hand, there are probably 10x or 100x more interactive shell users than shell programmers. On the other hand, I think it's better to completely nail a problem for a small group of users. There's also less competition in this space. For example, [zsh][], [fish][], and [xonsh][] are interactive shells with happy users. But I don't know of large shell scripts written in non-bash languages.

Let me try it out here:

Suppose you've grown a large bash script. You're having problems maintaining it, and you want to port it to a new language.

Oil is the language you can automatically translate it to. It's more readable and maintainable, and you'll get proper data types like numbers, arrays, and hash tables.

Even shorter one that I typed on reddit:

Basically OSH is a bash clone, and Oil is a new shell language that you can automatically translate bash/OSH programs to. It's more readable, maintainable, and has proper data types. The idea is to give you an easy upgrade path from bash.

I think that a fair number of people can relate to this situation. I can elaborate on it, but I want it to also stand alone. Let me know what you think.

(5) Conversations with Friends. In working out the rationale for Oil, I've

• Shell is a language for specifying distributed architecture. Processes and ports. Problem: this is still far away.

• I like to type fast. There is some truth to this. But:

  • I want to set the foundation with a solid language first
  • I don't have the expertise to make a good interactive shell.
  • I don't want ot deal with bike shedding discussions now

New Posts

I've been working on it for 15-16 months now. Time to look back.

• Project Retrospective: What did I do?

  • I didn't write a production-quality shell! It's too big and too slow. But I "de-risked" the whole thing.
  • I do? I made a bash-compatible shell. So I won't.
  • designed Oil language. Implemented a converter, but I didn't implement it.
  • Am I where I thought I would be? Honestly I feel a little burnt out on OSH. I want to work on Oil instead.
  • I haven't really worked on Oil in 6 months.
  • Or working on OPy would be fun, but Oil makes sense.

• What did I learn?

  • I should have spent a LOT LESS TIME thinking about rewriting the project in C++. I finally admitted that will never happen. It's too much work, or I write C++ too slowly to make this feasible.
  • I failed to attract meaningful contributions. Part of this is that the project is immature and doesn't really have users. The other part is that my standards are high. I have more of the Linus Torvalds theory of contribution -- make something work, people will be incentivized to contribute. Not going to make a "community" out of it yet. Guido too: Guido said he appreciated the community when they did the work of maintaining his mailing list!
  • research for Boil -- mostly good. My Makefile is buggy!
  • quick-ref has output DIRECTORY and it has file

• De-risk OVM -- pretty much did that

• De-risk OPy: need to do this.

• List of Possible Directions

  • In particular, the interactive direction is a separate bag of work. However, I've laid the common infrastructure for it. I'm not that excited about tackling it.

• Refocusing the project

  • "Oil language the you use when you want to upgrade to X"

• Why? I need to sell Oil again.

  • Forward looking stuff.
  • said to Ari: Because Shell is a language for describing software architecture.
  • said to Ari: Because I like to type fast.
  • look for quotes: once a shell script reaches X lines, you should rewrite it in another language.
  • TODO: search for X.

• blog generated more excitement than code??

• Goal: see below

• Architecture Odds and Ends -- mainly runtime vs.

• the language of startups

  • I've never worked for a startup, and I don't plan to. But I've lived in San Francisco for 15 years and worked in the tech industry, so many, even most of my friends and ex-coworkers have. I spent time on HN -- it's the best programmer forum (e.g. see the extensive comments on Lossless Syntax Tree post) -- I tend to ignore all the startup gossip.
  • so But I think the formulation of "make something people want" applies to open source.
  • elevator pitch -- one-sentence description. The problem has been that Oil is a big project. There are lots of things here. I dumped everything in this old blog post. I don't think people got the idea.
  • Did I product-market fit?

Sorry if this terminology is obnoxious. But I think it is accurate.

• BLOG found product-market fit! There is a hunger for people to understand new programming technques. Shell: The Good Parts.
  • Parsing, etc.
  • TODO: Maybe I should e-mail more people like Eli Bendersky? For related links?

General Directions

• OSH Language

  • better error reporting -- errexit diagnostics
  • dev tools:
  • better xtrace
  • state dump
  • coverage
  • run more shit: debootstrap, etc.

• OSH Interactive

  • completion

• Oil

  • fix osh2toil
  • Start writing Oil parser
  • figure out how you can use the same Shell Engine!
  • Later
  • Make engine -- no point because not compatible?
  • Awk engine -- no point because not compatible?
  • find engine (parser for GNU find?)
  • xargs engine (-P)

  • OCRE parser

  • probably need a new test framework to test that the conversion and the original both have the same output!! and exit code. Same execution model.

  • spec tests can do this!!!
  • or you could test ASTs? But I don't think that's the same thing.

• next: for Oil design, consult vim and textmate syntax highlighting.

  • vim is a weird command language, and textmate is a weird JSON language. both of them are bad.

• OPy

  • make everything faster
  • strip down the interpreter -- is there any point to this right now?

• not sure:

  • Make parser. Can you borrow from Mozilla pymake? Try it on your own Makefile. Does it parse pattern rules?
  • no point because I dont' want to convert other people's stuff?
  • well Linux kernel might be interesting. But LLVM is more important, and it uses CMake!

Goal

Oil by end of year! 5 months left!

But then I still won't have anything that concrete? Nah I think it being slow is OK. Nobody complained that it was too big and too slow!

Change parser to use exceptions, Fix parse error messages

TO START: oil-quick-ref!

• Next year:
  • start cutting down OVM and make a continuous build service with metrics
  • use the Oil language to cut down OVM.

Architecture

• making args.py WORK ON CONST WORDS WOULD BE VERY USEFUL

• vectorized VarOps

• completion and interactivity - second pass

  • sharing args.py logic with completion logic.
  • complete 'set -o foo'

• [ and [[ should share parser

  • same as core/args.py

• builtins

  • "trap" for signal handling
  • This affects interpreter architecture
  • and I guess it also uses "eval", etc.
  • ulimit -- syscall

• optimization:

  • fork_external can affect subshell
  • you don't need to fork in a pipeline!
  • check other shells
  • bin/child_count