Home

Rewriting Python's Build System

Python is over 25 years old, and it runs on many different systems, including Windows and Mac. It has the philosophy is "batteries included", which means it has a lot of dependencies: sqlite.

Wide portability and Dependencies imply a complex build system.

In the last two weeks I've learned a lot about Python's build system, and I've decided to rewrite it for Oil.

OR: Rewriting Python's build system

• Taking ownership of the code.
• Autotools is slow.
• Oil has different requirements. Unix-only, and not a lot of dependencies.

In the last two posts, I described experiments that validated the idea of OVM as a fork of the CPython VM.

These experiments are hacky shell scripts in the cpython-slice repo.

I want to clean them up and add them to the oil repo. As part of the cleanup, I've decided to "orchestrate" the shell scripts with a Makefile, for two reasons:

1. Incremental builds will be faster in this case. (Sometimes they're not.)
2. It's an opportunity test my Makefile chops on a realistic problem. I want Oil to subsume [shell and make][], but I know more about shell than about Make.

If I just wanted to ship OSH on Linux, I could keep the shell scripts. But I want to take my time, learn more about Make, and prepare for what I'm calling "Boil" -- the Oil build tool.

Today I'll introduce a few things about the problem. Tomorrow I'll write list some big themes it brings up.

Build Systems Are Partial Evaluation

I forget where I heard this phrasing. I think it was probably from one of the papers about Nix.

In CPython repo are build scripts written in various languages: shell, make, m4/autoconf. The goal is for a user to be able to build a running binary.

Note that this process involves two steps and two computers:

1. Creating a source tarball, with configure and build scripts, from the source repo. This process isn't trivial because autoconf and automake are code generators.
2. configure detects properties of the target system.

User Experience

Let's go in reverse order, starting from the user experience.

Oil/OVM uses CPython under the hood, but that shouldn't be apparent to either the end user or the builder/packager.

I want to distribute a tarball with Python and C code, and you should be able to make a binary with the standard procedure:

./configure        # needed for readline
make               # makes an 'oiltool' binary
sudo make install  # optionally install on the system

Should build an executable with:

• stripped down native code
• libc.so native exnsion
• Python standard library dependencies
• code for Oil
  • .pyc for execution
  • .py for tracebacks. (Tracebacks generally shouldn't be user-visible. I may turn them into crash reporting at some point.)

Why make a .zip file? Mainly because I can.

Developer Experience

In the repo, I will have a two-stage execution:

./tarball.sh build

Goal:

make _bin/oiltool

• Additional features of the Makefile

  • my pet peeve is BUILD VARIANTS. debug vs. release build.
  • other variants like I did with bwk / toybox:
  • ASAN, and other sanitizers.
  • coverage

  • need to experiment with: PGO. Python makefile already has this.

  • others: objects separate from .c files. Not sure if it matters for Python though?

This is an update so that the blog doesn't get behind the code. It's a list of things I'm thinking -- if anything catches your eye, leave a comment.

How Does it Differ than Python?

• No autotools.
• Statically linked modules

• No distutils

• Why rewrite the Python makefile?

  • Python uses autotools. I'm getting rid of the portability crap for the Unix shell.
  • I need libc; I don't need weird old unixes.
  • I don't have any libraries like pthreads or sqlite.
  • libraries I might need: readline, curses.

Conclusion

Ths process isn't trivial.

QEMU has a nice configure script written by hand.

What about toybox? I think there is no configure script -- it's Linux-only.

Lua: library, POSIX without autoconf

Python: autotools.

LLVM: CMake. Does that subsume autotools?