Why Sponsor Oils? | blog | oilshell.org

Oils 0.24.0 - Closures, Objects, and Namespaces

2025-01-04

This is a delayed announcement of the November release of:

Oils version 0.24.0 - Source tarballs and documentation.

(The most recent release is on our new oils.pub domain. More about that later.)

Why was it delayed? As I was writing it, it felt too dense, so I wrote a friendly introduction to the ideas introduced:

That is essential background for this announcement. (The reasons are likely not what you think!)

Table of Contents
Intro
Highlights
The Big Picture
Recap of Motivation
Contributors
Build and Packaging Improvements
Docs Updated
help builtin
Breaking Changes in YSH
Top Feedback: Env vars moved to ENV
YSH "here word"
Range operator 1 .. 3 replaced
args.ysh takes Type objects, not Strings
procs can be locals
eval() and evalExpr() moved
Blocks and Control Flow
Deprecations
fopen builtin -> redir
More YSH Changes
Operators, Integers
Added test --true and test --false
Misc Bugs and Fixes
Builtin Functions and Methods
Closures
Background / Definitions
Command values Are Closures
But NOT Block Args to Builtins
Expr values are Closures
TODO on Closures
Objects
Obj API
__invoke__ - Objects can be invoked like procs
Type expressions like List[Str]
Namespaces - "I learned Python with the dir() function"
Ongoing Reorganization
__builtins__ object
__defaults__ object, consulted after ENV
keys() values() get() are Free Functions
Modules
OSH Compatibility
Interactive Shell
What's Next?
Appendix: Selected Closed Issues
Appendix: Metrics for the 0.24.0 Release

Intro

Highlights

What's in this release? Important stuff first:

  1. Build and packaging improvements - thank you for the feedback!
  2. Docs updated
  3. Many YSH tweaks, based on using it. We are polishing the language.

The themes in the title:

  1. Closures
  2. Objects - similar to JavaScript and Lua
  3. Namespaces

We'll also see:

  1. OSH compatibility improvements
  2. Interactive Shell

The Big Picture

With the addition of closures, objects, and namespaces, YSH is now closer to Python and JavaScript than it is to Awk.

Python and JavaScript have all those features, but shell and Awk have none of them!

YSH is also gaining reflective control over the interpreter. We're making a programmable programming language, which supports DSLs like Awk and Hay.

Ruby seems to do better at this than Python or JavaScript, and so far our APIs compare well with Ruby. I welcome Ruby users to challenge us!

With these features, I also feel like YSH is more complete. The remaining big feature is Hay: declarative and programmable configuration. So:

But I don't anticipate big new features in the YSH language. The main change will be an overhaul of Hay!

Recap of Motivation

As mentioned, the objects post is essential background, and an appendix described the motivation for closures. If you want to read more, here are some rough Zulip threads (login required):

Contributors

Thank you to all contributors!

This list is incomplete — feel free to ping me if I left something out!

Build and Packaging Improvements

I did a bunch of work on issue #2080, with great feedback from Void Linux (meator) and Fedora (tkb-github).

Docs Updated

This release has tons of changes, and we're keeping the docs updated.

I realize that we need more blog posts to explain these features in a friendly way. For example, we have a nice design for string literals that I haven't gotten around to highlighting.

But we're testing YSH ourselves first. Feel free to join Zulip if you want to be a part of that process! Feedback and questions are welcome.

help builtin

We're also updating the Oils Error Catalog, With Hints.

The shell sometimes display codes like OILS-ERR-12, to lead you to more detail. Googling OILS-ERR-12 now finds these details, or you can use help to print a direct link:

ysh-0.25.0$ help OILS-ERR-12

     https://oils.pub/release/0.25.0/doc/error-catalog.html#oils-err-12

Breaking Changes in YSH

As usual, the breaking changes are in YSH only. OSH is very stable because most changes are bug fixes in bash features.

Top Feedback: Env vars moved to ENV

This is the most noticeable breaking change. Albin Otterhall and others ran into it, discussed on Zulip.

I mentioned the ENV object in the objects post. I think of it as a new namespace, and it also uses an Obj as a stack of dictionaries.

Feature OSH / POSIX Shell Breaking YSH Change
Read Env Var 
echo $PYTHONPATH

x=$PYTHONPATH

echo $[ENV.PYTHONPATH]

var x = ENV.PYTHONPATH
Permanently Set Env Var 
export PYTHONPATH=.

setglobal ENV.PYTHONPATH = '.'
Temporarily Set Env Var 
PYTHONPATH=foo ./foo.py
 		 
PYTHONPATH=foo ./foo.py
(unchanged)

You may notice that setglobal is a bit verbose, and I agree.

But it's more explicit, and doesn't introduce new rules into the language. Feedback on this is still welcome.

You can also write setenv or sh-env in pure YSH, and Julian has done something like this.

YSH "here word"

This code now behaves as you expect:

read <<< '''
1
2
3
'''

It has three lines, not 4! Adding an extra \n was inherited from bash and mksh, and doesn't make sense in YSH. This is technically a breaking change.

Range operator 1 .. 3 replaced

There are now two operators that are more explicit:

Syntax Values Name
1 ..< 3 1 2 half-open range
1 ..= 3 1 2 3 closed range

This was implemented by Aidan, and motivated by feedback from bar-g.

Using the old .. operator will suggest ..< or ..=.

args.ysh takes Type objects, not Strings

I mentioned this in the post on objects. The API now looks like:

parser (&spec)
  flag -c --count (Int)         # no quotes around 'Int'
  flag -c --source (List[Int])  # parameterized type object
}

procs can be locals

Procs are now defined in the local scope. So this is valid:

proc p {
  proc inner {
    echo hi
  }
  pp (inner)  # inspect the value
}

I also removed shopt -s redefine_proc_func. It inhibits metaprogramming, and is no longer needed now that we have modules with namespaces!

eval() and evalExpr() moved

They're now methods on the io Object:

Why are they both methods on io? I updated the Oils reference with examples of expressions that have effects:

var e1 = ^[ myplace->setValue(42) ]  # memory operation
var e2 = ^[ $(echo 42 > hi) ]        # I/O operation

Blocks and Control Flow

You can now break continue return out of a loop when you're inside a block.

This is technically a breaking change: we used to break from the block, not the surrounding loop. This was issue 2039.

Deprecations

fopen builtin -> redir

I renamed the fopen builtin to redir, based on this use case:

redir 2>&1 {
  call io->eval(b)
} | wc -l

fopen is retained for backward compatibility, but will be removed eventually. (I think Samuel mentioned once that fopen is not the best name.)

Note: Right now, you can't write

call io->eval (b) 2>&1

Instead, you have to use the redir { } block. This is a known parsing issue: #language-design > Parsing issue with commands that end with expressions

More YSH Changes

Operators, Integers

Int Conversion

In arithmetic ops like x + y, YSH has always converted strings to integers:

var x = '42'      # string, not integer
var sum = s + 1   # integer 43

To be consistent, we now do the same thing for List indices:

var s = mylist[x]         # get value at index 42
setvar mylist[x] = '9'    # set value at index 42
setvar mylist[x] += 5     # increment value at index 42

And for the operands to Slice and Range. That is, a[x:y] and x ..< y now have the same rule that mylist[x] and x + y do.

This came up a few times when writing YSH: #language-design > Lessons learned writing YSH code

Int Overflow

This is something that other shells don't do! They silently overflow, which means that their behavior depends on the underlying C compiler and platform. We still have more work to do here, but the plan is for all integer ops in YSH to be well-defined.

Added test --true and test --false

This is a nicer way to combine commands and expressions in conditionals.

if test --file $name && test --true $[myfunc(name)] {
  echo yes
}

This feature was based on feedback from Will Clardy and Julian Brown.

Aidan also added hints that detect when you use || instead of or, or && instead of and. (They use a simple "over-lexing" strategy.)

Note: some of our most common feedback shows that the distinction between YSH commands vs. expressions is not always natural. That is, mixing shell and Python/JS is natural for some people, but not for others. We'll continue to work on this issue.

Misc Bugs and Fixes

Builtin Functions and Methods

Improvements by Aidan:

New vm object:

Now we can pretty print the globals in both OSH and YSH:

osh$ = dict(vm.getFrame(0))

Try it! This is part of

Now let's talk about the themes in the title: Closures, Objects, and Namespaces.

Closures

Why does YSH need closures?

  1. I mentioned the Hay example from Aidan in the appendix to Why Should a Unix Shell have Objects?

We ran into more use cases:

  1. Unevaluated string templates should be closures.
  2. Expression arguments to procs should be closures.

Background / Definitions

The next sections might be cleaer if I clarify that there are many ways of talking about the same thing:

Now let's see what's changed.

Command values Are Closures

Procs can take block arguments, which are denoted by { }, and they are of type Command:

var x = 42
myproc {
  echo $x   # x refers to the variable above
}

Reminder: this is how you write a block expression that's not an argument:

var x = 42
var myblock = ^(echo $x)  # whenever this block is evaluated,
                          # x refers to the variable above

(The ^(echo $x) syntax is similar to $(echo $x).)

But NOT Block Args to Builtins

This is perhaps a bit confusing:

So they are not closures. This is because we want to be able to reference variables created in the block later:

cd /tmp {
  var listing = $(ls -x -y -z)
}
echo $listing  # should refer to the variable in the block

I also think of cd like an "inline proc", in that invoking it doesn't push and pop a new stack frame. There may be a way to resolve this inconsistency, or we can just live with it. Again, feedback is welcome.

Expr values are Closures

Similarly, expressions are closures:

var x = 42

var e1 = ^[x + 1]   # value of type Expr

var e2 = ^"x = $x"  # another value of type Expr

p [x + 1]           # another one, equivalent to:
p (^[x + 1])

Another related design note:

TODO on Closures

What still needs to be done?

For example:

for x in a b c {
  myproc { echo $x }            # x should be captured!
  when [size > x] { echo big }  # ditto
}

It's worth mentioning that the material on closures in Crafting Interpreters was very helpful. This book helped us with garbage collection, hash tables (e.g. deletion/tombstones), and closures!

Objects

Now let's talk about objects. Objects and closures are both ways of bundling code and data.

Languages like Python, JavaScript, Lua, and Ruby all have both objects and closures.

Obj API

I showed the new API in the objects post:

var obj = Obj.new({x: 42}, null)
var mydict = first(obj)
var parent = rest(obj)

I would like this shorter API:

var obj = Obj({x: 42}, null)   # no .new

But that requires the special __call__ method, which we don't have yet.

__invoke__ - Objects can be invoked like procs

You can now invoke objects with the same syntax as procs:

my-object arg1 arg2

You do this by giving them an __invoke__ meta-method. Docs:

At first, this was motivated by the use case of generating procs dynamically, which Julian asked about. We had solutions based on:

  1. eval $mystr
  2. parseCommand() and then io->eval()

And then I decided to experiment with invokable objects. It then played a crucial rule in the implementation of modules:

my-module my-proc

So it's here to stay. I anticipate many more uses of it:

Type expressions like List[Str]

I created type objects like List and Dict, and defined the [ operator on them.

So now List[Str] and Dict[Str, Int] evaluate to singleton objects. This was for the args.ysh use case, mentioned above, and discussed in the objects post.

Namespaces - "I learned Python with the dir() function"

I use this slogan to explain the motivation.

I want users to be able to discover shell by typing — by interacting with the interpreter. Not by reading the manual!

Let's see what changed.

Ongoing Reorganization

Breaking change: I added shopt --set no_init_globals, which means that YSH doesn't initialize certain globals, like SHELLOPTS. This is part of organizing globals into namespaces, which is still ongoing. Feedback is welcome.

__builtins__ object

We moved functions like len() and types like Float to a __builtins__ object. It serves the same purpose as __builtins__ in Python.

Example:

ysh$ = len
<BuiltinFunc 0x7fa1e1842f50>

ysh$ = __builtins__
(Obj)   <Obj 0x7fa1e1970d20>

ysh$ = __builtins__.len'
<BuiltinFunc 0x7fa1e1842f50>

In YSH, a typical variable lookup now has three steps:

  1. Look in locals
  2. Look in globals
  3. Look in __builtins__

So builtins no longer pollute the global namespace.

__defaults__ object, consulted after ENV

For example, we have __defaults__.PATH and __defaults__.PS1.

keys() values() get() are Free Functions

We used to have d => keys(), but now it's just keys(d).

Why? Method calls are now obj.method(), not obj => method(). And this causes a conflict for Dict, which supports mydict.attr.

The => syntax is for function chaining, though it's still allowed for method calls.

Modules

I demonstrated in the objects post. We did this because we use it in the YSH standard library!

Python-like modules are nice and convenient! (Both JavaScript and Lua lacked modules for a long time, and later added them.)

OSH Compatibility

Interactive Shell

When $PS1 is not set, this is the default prompt:

ysh-0.23.0$

When it is, we want OSH versus YSH to look like this:

currentdir$
ysh currentdir$

What's Next?

A couple days ago, I announced that we're (finally) moving to the oils.pub domain. This is actually the last post on oilshell.org! I put it here because the 0.24.0 tarball is also published on this domain.

In that post, I gave a sense for what's in Oils 0.25.0, which is already released: bash compatibility, and "under the hood" improvements to our metalanguages.

I published a skeleton for a Vim syntax plugin:

It needs to be fleshed out, and I want to make it easy to write syntax highlighters for SublimeText, TreeSitter, Helix, and more.

So I expect that the experience of finishing the Vim plugin will feed back into the YSH language design! We can make the syntax simpler, mainly by disallowing legacy shell syntax:

Here's some brainstorming for the rest of 2025:

Let me know what you think in the comments. Happy new year!

Appendix: Selected Closed Issues

Some of these issues are mentioned above, and some are not.

#2118 strict_errexit message missing code location
#2114 printf errors can cause status 1, rather than being fatal
#2110 bug in old version of dash shell causes _build/oils.sh to start too many compilers in parallel
#2108 Ctrl-C causes Interrupted system call
#2107 printf crashes with ValueError when integers are large
#2104 Crash with setvar on out-of-bounds list index
#2096 ysh breaking: Replace 1 .. 5 range syntax with 1 ..< 5 half open and 1 ..= 5 closed range
#2094 allow && || in YSH conditions and add test --true --false
#2080 install script may not match what distros want - Void Linux, stripped binary, binary location when cross-compiling, etc.
#2078 Crash with dict literal
#2074 members of context managers are uninitialized and rooted
#2055 Trap does not check for the first argument being an unsigned integer
#2039 executing blocks that contain return/break/continue/error is inconsistent with eval on strings

Appendix: Metrics for the 0.24.0 Release

These metrics help me keep track of the project. Let's compare this release with the previous one, version 0.23.0.

Docs

Spec Tests

OSH continues to make progress, with 20 more tests passing:

Everything works in fast C++, even though we write typed Python:

YSH made more progress, with 87 more tests passing:

Likewise, everything still works in C++:

Benchmarks

I don't recall why the parser got faster:

Oils is generally getting faster, which is good!

Not much change in parser memory usage:

We got faster on a compute-bound workload:

I think this improvement was due to removing a duplicate hash lookup in Python, e.g. if x in dict: foo = d[x]. (These release notes aren't always complete, and sometimes the benchmarks remind me of improvements we made!)

No change on a I/O bound workload:

Again, our measurements have noise when comparing OSH to bash:

But it's a good sign that, compared with a couple releases ago, our worst numbers are getting closer to bash.

Code Size

YSH has the biggest delta in lines of code, but it's still small:

And generated C++:

And compiled binary size: