Oil Source Code

Build Status

Oil is a new Unix shell. It's our upgrade path from bash! (Why Create a New Unix Shell? / 2019 FAQ)

Its source code lives in git.

It's written in Python, so the code is short and easy to change. But we automatically translate it to C++ with custom tools, to make it fast and small. The deployed executable doesn't depend on Python.

Table of Contents
Contributing
Quick Start on Linux
Docs
Repository Structure
A Collection of Interpreters
DSLs / Code Generators
Native Code
Several Kinds of Tests
Dev Tools and Scripts
Temp Dirs
Build System for End Users
Doc Sources
More info
Python Files Not Translated to C++

Contributing

Quick Start on Linux

Fetch the source code:

git clone https://github.com/oilshell/oil.git  # or your fork
git submodule update --init --recursive        # to get dependencies

Build the Python extension, and run bin/osh (or bin/oil):

bash$ build/dev.sh minimal
...
# Now you should have a libc.so symlink in the repository root directory

bash$ bin/osh
osh$ name=world
osh$ echo "hello $name"
hello world

Try running a shell script you wrote with bin/osh myscript.sh.

This is called the developer build, and is very different from the release tarball. The Contributing page describes this difference in detail.

The release tarballs are linked from the home page. (Developer builds don't work on OS X, so use the release tarballs on OS X.)

Run bin/oil to try the Oil language. Send me feedback about it!

Docs

The Wiki has many developer docs. Feel free to edit them. If you make a major change, let us know on Zulip!

There are also READMEs in some subdirs, like opy/ and mycpp/.

If you're confused, the best thing to do is to ask on Zulip and someone should produce a pointer and/or improve the docs.

Docs for end users are linked from each release page.

Repository Structure

Try this to show a summary of what's in the repo and their line counts:

$ metrics/source-code.sh all

(Other functions in this file may be useful as well.)

A Collection of Interpreters

Oil is naturally structured as a set of mutually recursive parsers and evaluators. These interpreters are specified at a high-level: with regular languages, Zephyr ASDL, and aa statically-typed subset of Python.

bin/              # Main entry points like bin/osh (source in bin/oil.py)
frontend/         # Lexing/Parsing code common to Oil and OSH
osh/              # OSH parsers and evaluators (cmd, word, sh_expr)
oil_lang/         # Oil parser and evaluator
core/             # Other code shared between Oil and OSH
pylib/            # Borrowed from the Python standard library.
tools/            # User-facing tools, e.g. the osh2oil translator

DSLs / Code Generators

Here are the tools that transform that high-level code to efficient code:

asdl/             # ASDL implementation, derived from CPython
pgen2/            # Parser Generator, borrowed from CPython
mycpp/            # Experimental translator from typed Python to C++.
                  # Depends on MyPy.
opy/              # Python compiler in Python (mycpp/ will replace it)
  lib/            # Common code
  compiler2/      # Bytecode compiler
  byterun/        # Metacircular bytecode VM in Python
  gold/           # tests
  byterun/        # Unused bytecode interpreter

Native Code

We have native code to support both the dev build (running under CPython) and the oil-native build (pure C++):

Python-2.7.13/    # CPython is the initial basis for the Oil VM
native/           # Python extension modules, e.g. libc.c
cpp/              # C++ code which complements the mycpp translation

Several Kinds of Tests

Unit tests are named foo_test.py and live next to foo.py.

test/             # Test automation
  gold/           # Gold Test cases
  gold.sh         
  sh_spec.py      # shell spec test framework
  spec.sh         # Types of test runner: spec, unit, gold, wild
  unit.sh         
  wild.sh
testdata/
spec/             # Spec test cases
  bin/            # tools used in many spec tests
  testdata/       # scripts for specific test cases
  errors/         # TODO: migrate these bad shell scripts
types/            # Scripts for running MyPy and PyAnnotate, etc.

Dev Tools and Scripts

We use a lot of automation to improve the dev process. It's largely written in shell, of course!

benchmarks/       # Benchmarks should be run on multiple machines.
metrics/          # Metrics don't change between machines (e.g. code size)
build/            # Build automation
  oil-defs/       # Files that define our slice of CPython.
  dev.sh          # For development builds, running CPython
devtools/         # For Oil developers (not end users)
  release.sh      # The (large) release process.
demo/             # Demonstrations of bash/shell features.  Could be
                  # moved to tests/ if automated.
  old/            # A junk drawer.
web/              # HTML/JS/CSS for tests and tools
lazylex/          # An HTML lexer which doctools/ builds upon.
services/         # Integration with cloud services (e.g. Travis CI)

Temp Dirs

Directories that begin with _ are not stored in git. The dev tools above create and use these dirs.

_bin/             # Native executables are put here
_build/           # Temporary build files
_devbuild/        # Developer build files not deleted upon 'make clean'
  gen/            # Generated Python and C code
_deps/            # build dependencies like re2c
_tmp/             # Test suites and other temp files
  spec/
  wild/
    raw/
    www/
  osh-parser/
  osh-runtime/
  vm-baseline/
  oheap/
  startup/
  ...
_release/         # Source release tarballs are put here
  VERSION/        # Published at oilshell.org/release/$VERSION/
    benchmarks/
    doc/
    metrics/
    test/
      spec.wwz
      wild.wwz
      ...
    web/          # Static files, copy of $REPO_ROOT/web
      table/

Build System for End Users

This is very different than the developer build of Oil.

Makefile
configure
install

Doc Sources

doc/              # A mix of docs
doctools/         # Tools that use lazylex/ to transform Markdown/HTML
README.md         # This page, which is For Oil developers

LICENSE.txt       # For end users
INSTALL.txt

More info

Python Files Not Translated to C++

mycpp/
  mylib.py  # statically typed equivalents of Python's data structures
pylib/      # copied from Python stdlib
core/
  py{error,os,util}.py  # too complicated to translate
*/*_def.py  # abstract definitions
*/*_gen.py  # code generators

Generated on Thu Sep 3 10:28:33 PDT 2020