Why Sponsor Oils? | blog | oilshell.org

Appendix: The Journey of Extended Globs

The last post was Winter Blog Backlog: Recent Progress. It mentioned that I drafted a post called How OSH is Designed, which used extended globs as an example to illustrate a few design principles.

I scrapped that draft in favor of the summary, which more accurately conveys the truth.

But I did a bunch of work to dig up the history of the feature in the blog, which I'm publishing here. I don't expect that many readers to be interested in this, but I like it because it helps me keep track of the project.

Context

I ran this command to dig up old posts:

oilshell.org$ grep 'extended.glob' blog/*/*/[^_]*.md

It turned up about a dozen, which I've copied below. Summary: OSH started parsing extended globs in 2017, and I partially implemented them in 2018 order to run bash autocompletion scripts.

I put off finishing the feature, since it's uniquely difficult, and wasn't necessary to run thousands of lines of real distro shell scripts.

But Nix motivated me to finish the feature, and it turned out well!

Posts Mentioning Extended Globs (2016-2020)

• 2016-10 - What Oil Looks Like, and a Plan For This Blog
  • The very first post has spec tests for extended globs! The file has just 24 lines. I was aware of the feature, but had never used it myself.
• 2016-11 - Roadmap #3 Continued
  • I mention that extended globs probably require another lexer mode. Yes!
• 2017-03 - Measuring Progress with Tests
  • I mentioned extended globs as an unparsed language feature.(There's a typo set -o; it should have been write shopt -s extglob.)
• 2017-06 - How I Plan to Use Tests: Transforming OSH
  • I'm still following this philosophy. For example, see the C++ spec test results in the last release announcement.
• 2017-10 - Status Update: Parser Correctness and Performance
  • I also expanded the corpus of testdata. The results showed that I should implement extended globs.
  • Possible blog post: A Critique of the Extended Glob Syntax.
• 2017-11 - OSH 0.2 - Parsing One Million Lines of Shell
  • Parse the extended glob syntax. (Only parsed, not executed.)
• 2018-10 - Running Bash Completion Scripts with OSH
  • Completion scripts often look like their own dialect of bash. Arrays, associative arrays, extended globs, eval, and dynamic scope are common.
• 2019-02 - How to Parse Shell Like a Programming Language
  • Extended globs have their own lexer mode and are a recursive sublanguage. (This was true before 2019, but it took me awhile to write the blog post.)
• 2019-02 - OSH 0.6.pre15 Does Not Crave Chaos and Destruction
  • I mentioned the portability issue here: Extended Globs Rely on GNU libc.
• 2020-05 - Oil 0.8.pre5: Progress in C++
  • Closed issue 758 regarding quoting of words containing extended globs. I actually revisited this test case during the 0.9.3 release. Thanks to Koichi Nakashima for reporting it!
• 2020-07 - Regular Languages, Part 2: Ideas and Questions
  • I mention that one motivation for Oil implementing its own pattern matching engine is extended globs (and shopt -s globstar with **, which we don't yet have).
• 2020-12 - Three Great Videos About Regex Derivatives
  • I mentioned again that extended globs give globs the power of regular expressions. But I didn't realize that they're more expressive in one way: with negation of entire patterns! (This is more expressive for programmers, but not in the mathematical sense of being able to express more subsets of strings!)