Rewriting Python's Build System


Python is over 25 years old, and it runs on many different systems, including Windows and Mac. It has the philosophy is "batteries included", which means it has a lot of dependencies: sqlite.

Wide portability and Dependencies imply a complex build system.

In the last two weeks I've learned a lot about Python's build system, and I've decided to rewrite it for Oil.

OR: Rewriting Python's build system

In the last two posts, I described experiments that validated the idea of OVM as a fork of the CPython VM.

These experiments are hacky shell scripts in the cpython-slice repo.

I want to clean them up and add them to the oil repo. As part of the cleanup, I've decided to "orchestrate" the shell scripts with a Makefile, for two reasons:

  1. Incremental builds will be faster in this case. (Sometimes they're not.)
  2. It's an opportunity test my Makefile chops on a realistic problem. I want Oil to subsume [shell and make][], but I know more about shell than about Make.

If I just wanted to ship OSH on Linux, I could keep the shell scripts. But I want to take my time, learn more about Make, and prepare for what I'm calling "Boil" -- the Oil build tool.

Today I'll introduce a few things about the problem. Tomorrow I'll write list some big themes it brings up.

Build Systems Are Partial Evaluation

I forget where I heard this phrasing. I think it was probably from one of the papers about Nix.

In CPython repo are build scripts written in various languages: shell, make, m4/autoconf. The goal is for a user to be able to build a running binary.

Note that this process involves two steps and two computers:

  1. Creating a source tarball, with configure and build scripts, from the source repo. This process isn't trivial because autoconf and automake are code generators.
  2. configure detects properties of the target system.

User Experience

Let's go in reverse order, starting from the user experience.

Oil/OVM uses CPython under the hood, but that shouldn't be apparent to either the end user or the builder/packager.

I want to distribute a tarball with Python and C code, and you should be able to make a binary with the standard procedure:

./configure        # needed for readline
make               # makes an 'oiltool' binary
sudo make install  # optionally install on the system

Should build an executable with:

Why make a .zip file? Mainly because I can.

Developer Experience

In the repo, I will have a two-stage execution:

./tarball.sh build


make _bin/oiltool

This is an update so that the blog doesn't get behind the code. It's a list of things I'm thinking -- if anything catches your eye, leave a comment.

How Does it Differ than Python?


Ths process isn't trivial.

QEMU has a nice configure script written by hand.

What about toybox? I think there is no configure script -- it's Linux-only.

Lua: library, POSIX without autoconf

Python: autotools.

LLVM: CMake. Does that subsume autotools?