Hay - Custom Languages for Unix Systems

Hay lets you use the syntax of the Oil shell to declare data and interleaved code. It allows the shell to better serve its role as essential glue. For example, these systems all combine Unix processes in various ways:

Slogans:

This doc describes how to use Hay, with motivating examples.

As of 2022, this is a new feature of Oil, and it needs user feedback. Nothing is set in stone, so you can influence the language and its features!

Table of Contents
Example
Understanding Hay
Analogies
Prior Art
Comparison
Overview
Two Kinds of Nodes, and Three Kinds of Evaluation
Two Stages of Evaluation
Result Schema
Three Ways to Invoke Hay
Inline Hay Has No Restrictions
In Separate Files
In A Block
Security Model: Restricted != Sandboxed
Reference
Shell Builtins
Functions
Options
Usage: Interleaving Hay and Oil
Conditionals
Iteration
Remove Duplication with proc
More Usage Patterns
Using Oil for the Second Stage
Using Python for the Second Stage
Locating Errors in the Original .hay File
Debian .d Dirs
Parallel Loading
Style
Attributes vs. Procs
Attributes vs. Flags
Dicts vs. Blocks
Oil vs. Shell
Future Work
Links

Example

Hay could be used to configure a hypothetical Linux package manager:

# cpython.hay -- A package definition

hay define Package/TASK  # define a tree of Hay node types

Package cpython {        # a node with attributes, and children

  version = '3.9'
  url = 'https://python.org'

  TASK build {           # a child node, with Oil code
    ./configure
    make
  }
}

This program evaluates to a JSON tree, which you can consume from programs in any language, including Oil:

{ "type": "Package",
  "args": [ "cpython" ],
  "attrs": { "version": "3.9", "url": "https://python.org" },
  "children": [
     { "type": "TASK",
       "args": [ "build" ],
       "code_str": "  ./configure\n  make\n"
     }
  ]
}

That is, a package manager can use the attributes to create a build environment, then execute shell code within it. This is a staged evaluation model.

Understanding Hay

A goal of Hay is to restore the simplicity of Unix to distributed systems. It's all just code and data!

This means that it's a bit abstract, so here are a few ways of understanding it.

Analogies

The relation between Hay and Oil is like the relationship between these pairs of languages:

Each of these is 70's-style macro programming — a stringly-typed language generating another stringly-typed language, with all the associated problems.

In contrast, Hay and Oil are really the same language, with the same syntax, and the same Python- and JavaScript-like dynamic types. Hay is just Oil that builds up data instead of executing commands.

(Counterpoint: Ninja is intended for code generation, and it makes sense for Oil to generate simple languages.)

Prior Art

See the Survey of Config Languages on the wiki, which puts them in these categories:

  1. Languages for String Data
  2. Languages for Typed Data
  3. Programmable String-ish Languages
  4. Programmable Typed Data
  5. Internal DSLs in General Purpose Languages

Excerpts:

YAML is a data format that is (surprisingly) the de-facto control plane language for the cloud. It's an approximate superset of JSON.

UCL (universal config language) and HCL (HashiCorp config language) are influenced by the Nginx config file syntax. If you can read any of these languages, you can read Hay.

Nix has a functional language to configure Linux distros. In contrast, Hay is multi-paradigm and imperative.

The Starlark language is a dialect of Python used by the Bazel build system. It uses imperative code to specify build graph variants, and you can use this same pattern in Hay. That is, if statements, for loops, and functions are useful in Starlark and Hay.

Ruby's use of first-class blocks inspired Oil. They're used in systems like Vagrant (VM dev environments) and Rake (a build system).

In Lisp, code and data are expressed with the same syntax, and can be interleaved. G-Expressions in Guix use a staged evaluation model, like Hay.

Comparison

The biggest difference between Hay and UCL / HCL is that it's embedded in a shell. In other words, Hay languages are internal DSLs, while those languages are external.

This means:

  1. You can interleave shell code with Hay data. We'll discuss the many uses of this below.
  2. Hay isn't a library you embed in another program. Instead, you use Unix-style process-based composition.

The sections below elaborate on these points.

Overview

Hay nodes have a regular structure:

Two Kinds of Nodes, and Three Kinds of Evaluation

There are two kinds of node with this structure.

(1) SHELL nodes contain unevaluated code, and their type is ALL CAPS. The code is turned into a string that can be executed elsewhere.

TASK build {
  ./configure
  make
}
# =>
# ... {"code_str": "  ./configure\n  make\n"}

(2) Attr nodes contain data, and their type starts with a capital letter. They eagerly evaluate a block in a new stack frame and turn it into an attributes dict.

Package cpython {
  version = '3.9'
}
# =>
# ... {"attrs": {"version": "3.9"}} ...

These blocks have a special rule to allow bare assignments like version = '3.9'. That is, you don't need keywords like const or var.

(3) In contrast to these two types of Hay nodes, Oil builtins that take a block usually evaluate it eagerly:

cd /tmp {  # run in a new directory
  echo $PWD
}

Builtins are spelled with lower case letters, so SHELL and Attr nodes won't be confused with them.

Two Stages of Evaluation

So Hay is designed to be used with a staged evaluation model:

  1. The first stage follows the rules above:
  2. Your app or system controls the second stage. You can invoke Oil again to execute shell inside a VM, inside a Linux container, or on a remote machine.

These two stages conceptually different, but use the same syntax and evaluator! Again, the evaluator runs in a mode where it builds up data rather than executing commands.

Result Schema

Here's a description of the result of Hay evaluation (the first stage).

# The source may be "cpython.hay"
FileResult = (source Str, children List[NodeResult])

NodeResult =
  # package cpython { version = '3.9' }
  Attr (type                Str,
        args                List[Str],
        attrs               Map[Str, Any],
        children            List[NodeResult])

  # TASK build { ./configure; make }
| Shell(type                Str,
        args                List[Str],
        location_str        Str,
        location_start_line Int,
        code_str            Str)

Notes:

Three Ways to Invoke Hay

Inline Hay Has No Restrictions

You can put Hay blocks and normal shell code in the same file. Retrieve the result of Hay evaluation with the _hay() function.

# myscript.oil

hay define Rule

Rule mylib.o {
  inputs = ['mylib.c']

  # not recommended, but allowed
  echo 'hi'
  ls /tmp/$(whoami)
}

echo 'bye'  # other shell code

const result = _hay()
json write (result)

In this case, there are no restrictions on the commands you can run.

In Separate Files

You can put hay definitions in their own file:

# my-config.hay

Rule mylib.o {
  inputs = ['mylib.c']
}

echo 'hi'  # allowed for debugging
# ls /tmp/$(whoami) would fail due to restrictions on hay evaluation

In this case, you can use echo and write, but the interpreted is restricted (see below).

Parse it with parse_hay(), and evaluate it with eval_hay():

# my-evaluator.oil

hay define Rule  # node types for the file
const h = parse_hay('build.hay')
const result = eval_hay(h)

json write (result)
# =>
# {
#   "children": [
#     { "type": "Rule",
#       "args": ["mylib.o"],
#       "attrs": {"inputs": ["mylib.c"]}
#     }
#   ]
# }

In A Block

Instead of creating separate files, you can also use the hay eval builtin:

hay define Rule

hay eval :result {  # assign to the variable 'result'
  Rule mylib.o {
    inputs = ['mylib.c']
  }
}

json write (result)  # same as above

This is mainly for testing and demos.

Security Model: Restricted != Sandboxed

The "restrictions" are not a security boundary! (They could be, but we're not making promises now.)

Even with eval_hay() and hay eval, the config file is evaluated in the same interpreter. But the following restrictions apply:

In summary, Hay evaluation is restricted to prevent basic mistakes, but your code isn't completely separate from the evaluated Hay file.

If you want to evaluate untrusted code, use a separate process, and run it in a container or VM.

Reference

Here is a list of all the mechanisms mentioned.

Shell Builtins

Functions

Options

Hay is parsed and evaluated with option group oil:all, which includes parse_proc and parse_equals.

Usage: Interleaving Hay and Oil

Why would you want to interleave data and code? One reason is to naturally express variants of a configuration. Here are some examples.

Build variants. There are many variants of the Oil binary:

So the Ninja build graph to produce these binaries is shaped similarly, but it varies with compiler and linker flags.

Service variants. A common problem in distributed systems is how to develop and debug services locally.

Do your service dependencies live in the cloud, or are they run locally? What about state? Common variants:

Again, these collections of services are all shaped similarly, but the flags vary based on where binaries are physically running.


This model can be referred to as "graph metaprogramming" or "staged programming". In Oil, it's done with dynamically typed data like integers and dictionaries. In contrast, systems like CMake and autotools are more stringly typed.

The following examples are meant to be "evocative"; they're not based on real code. Again, user feedback can improve them!

Conditionals

Conditionals can go on the inside of a block:

Service auth.example.com {    # node taking a block
  if (variant === 'local') {  # condition
    port = 8001
  } else {
    port = 80
  }
}

Or on the outside:

Service web {               # node
  root = '/home/www'
}

if (variant === 'local') {  # condition
  Service auth-local {      # node
    port = 8001
  }
}

Iteration

Iteration can also go on the inside of a block:

Rule foo.o {   # node
  inputs = []  # populate with all .cc files except one

  # variables ending with _ are "hidden" from block evaluation
  for name_ in *.cc {
    if name_ !== 'skipped.cc' {
      _ append(inputs, name_)
    }
  }
}

Or on the outside:

for name_ in *.cc {                # loop
  Rule $(basename $name_ .cc).o {  # node
    inputs = [name_]
  }
}

Remove Duplication with proc

Procs can wrap blocks:

proc myrule(name) {

  # needed for blocks to use variables higher on the stack
  shopt --set dynamic_scope {

    Rule dbg/$name.o {      # node
      inputs = ["$name.c"]
      flags = ['-O0']
    }

    Rule opt/$name.o {      # node
      inputs = ["$name.c"]
      flags = ['-O2']
    }
    
  }
}

myrule foo  # call proc
myrule bar  # call proc

Or they can be invoked from within blocks:

proc set-port(port_num, :out) {
  setref out = "localhost:$port_num"
}

Service foo {      # node
  set-port 80 :p1  # call proc
  set-port 81 :p2  # call proc
}

More Usage Patterns

Using Oil for the Second Stage

The general pattern is:

./my-evaluator.oil my-config.hay | json read :result

The evaluator does the following:

  1. Sets up the execution context with hay define
  2. Parses my-config.hay with parse_hay()
  3. Evaluates it with eval_hay()
  4. Prints the result as JSON.

Then a separate Oil processes reads this JSON and executes application code.

TODO: Show code example.

Using Python for the Second Stage

In Python, you would:

  1. Use the subprocess module to invoke ./my-evaluator.oil my-config.hay.
  2. Use the json module to parse the result.
  3. Then execute application code using the data.

TODO: Show code example.

Locating Errors in the Original .hay File

The Oil interpreter has 2 flags starting with --location that give you control over error messages.

oil --location-str 'foo.hay' --location-start-line 42 -- stage2.oil

Set them to the values of fields location_str and location_start_line in the result of SHELL node evalution.

Debian .d Dirs

Debian has a pattern of splitting configuration into a directory of concatenated files. It's easier for shell scripts to add to a directory than add to a file.

This can be done with an evaluator that simply enumerates all files:

var results = []
for path in myconfig.d/*.hay {
  const code = parse_hay(path)
  const result = eval(hay)
  _ append(results, result)
}

# Now iterate through results

Parallel Loading

TODO: Example of using xargs -P to spawn processes with parse_hay() and eval_hay(). Then merge the JSON results.

Style

Attributes vs. Procs

Assigning attributes and invoking procs can look similar:

Package grep {
  version = '1.0'  # An attribute?

  version 1.0  # or call proc 'version'?
}

The first style is better for typed data like integers and dictionaries. The latter style isn't useful here, but it could be if version 1.0 created complex Hay nodes.

Attributes vs. Flags

Hay nodes shouldn't take flags or --. Flags are for key-value pairs, and blocks are better for expressing such data.

No:

Package --version 1.0 grep {
  license = 'GPL'
}

Yes:

Package grep {
  version = '1.0'
  license = 'GPL'
}

Dicts vs. Blocks

Superficially, dicts and blocks are similar:

Package grep {
  mydict = {name: 'value'}  # a dict

  mynode foo {              # a node taking a block
    name = 'value'
  }
}

Use dicts in cases where you don't know the names or types up front, like

files = {'README.md': true, '__init__.py': false}

Use blocks when there's a schema. Blocks are also different because:

Oil vs. Shell

Hay files are parsed as Oil, not OSH. That includes SHELL nodes:

TASK build {
  cp @deps /tmp   # Oil splicing syntax
}

If you want to use POSIX shell or bash, use two arguments, the second of which is a multi-line string:

TASK build '''
  cp "${deps[@]}" /tmp
'''

The Oil style gives you static parsing, which catches some errors earlier.

Future Work

Please send feedback about Hay. It will inform and prioritize this work!

Links


Generated on Wed May 3 15:38:09 EDT 2023