Home

Rewriting Python's Build System From Scratch

What am I working on? In the last post, I linked to bash scripts that built a slice of the CPython VM.

I'm now using these scripts as the foundation for a rewrite of Python's autotools-based build system. This will be the build system for Oil and OVM. You can think of OVM as the part of Oil that is native code.

Why do this?

1. To support deploying Oil as a single executable file, i.e. an app bundle.
2. Because OVM has different dependencies and system requirements than CPython.
3. To practice writing nontrivial Makefiles from scratch. This is research for combining shell and make.

I think writing good build systems is hard, and this experience is supporting that belief. Read on for details.

bin/oil as an App Bundle

Oil's use of CPython should be an implementation detail. To users and packagers, Oil should look like a C program. Thus, bin/oil will be a single file with two components:

1. Native code. CPython without its parser, plus pylibc, an extension module that wraps libc.
2. Bytecode. I'm precompiling .pyc files to save work at startup. We also need .py files to show code snippets in tracebacks.

This is familiar to me because a decade ago I submitted a patch to Python to allow it to directly execute .zip files (python-dev thread).

I'm taking it further now by:

1. Rather than concatenating a shell script + .zip file, I'm concatenating an ELF file + .zip file. That is, the native code will load the zipped bytecode in the same file.
2. Statically linking the pylibc extension module (but not libc itself.)

Why a single file? I want to be completely divorced from Python packaging. I don't want anything to do with site-packages and so forth, which are known to cause deployment problems.

(Historical note: Phillip J. Eby rewrote the patch to support directories with __main__.py in addition to .zip files. This was available in Python 2.6, but it was relatively unknown until Python 3.5 added the zipapp module, a result of PEP 441.)

Requirements and System Dependencies

Building OVM is a different problem than building Python. This section compares them along three dimensions.

Platform Portability:

• Python is over 25 years old, and portable to many non-Unix systems, including Windows and the classic Mac OS.
• Oil is a Unix shell, so it only needs to run on Unix!

(I suppose it will run on Windows 10 via the confusingly-named Windows Subsystem for Linux. I find it amazing that this Windows kernel feature is marketed as Bash on Windows. A program that is over three decades old is now a hot new feature!)

Dependencies:

• Python has a batteries included philosophy, which means its standard library has many dependencies: libffi for ctypes, sqlite, etc.
• A Unix shell depends only on the oldest and most portable parts of Unix: fork(), exec(), pipe(), read(), write(), etc.

Extensibility with Dynamic Linking:

• A large part of Python's build system is dealing with dynamically-loaded extension modules like _json.so. The Python-2.7.13/setup.py that controls this is ~2300 lines long!
• I know exactly which extension modules I need for Oil, so it doesn't need to be configurable.

This means that Python requires a complex build system, but Oil can have a relatively simple one.

I don't think Oil will need autotools. If you can think of a reason it might, leave a comment. The main issue I see is detecting the readline library, but that can be done with a POSIX shell script written by hand.

Research for Boil

Boil is the name I have in mind for the Make-like build tool in Oil. Its relationship to Make is similar to that of Oil and bash, although not identical.

This is a big topic, so for now I'll list a few problems that I found with Python's Makefile. These problems appear in most Makefiles because it's hard to express the solution in Make.

1. Proper incremental builds when changing CFLAGS. It's annoying to do a make clean between debug and release builds.
2. Support for build variants like AddressSanitizer (ASAN) and code coverage. After shaking bugs out of toybox and bwk with ASAN, I'm now a big fan of it.
   • NOTE: Python's coverage support is broken with respect to shared libraries. You don't get coverage for say _json.so because setup.py didn't pass the right flags.
3. Python duplicates its C header dependencies in its Makefile, rather than using automatic dependency generation, i.e. the gcc -M scheme. I plan to reorganize the code for OVM, and this may be a nuisance.

Status and Conclusion

I'm knee-deep in this work right now. It's not finished by any means, but I'm happy to slow down and examine Make more closely. I revisited the research I did before starting the blog last year.

I would say that shell has bad syntax and good semantics, but Make has both bad syntax and bad semantics. It's not statically parseable, and there are many weird mechanisms like double-colon rules, order-only dependencies, .ONESHELL, .SECONDARYEXPANSION, etc.

Boil is still two steps away. A rough plan is:

1. Release OSH with a Makefile.
2. Release the Oil language.
3. Add Boil to Oil.
4. At some point: Rewrite the Makefile in Boil.

So Make and Boil will a major blog topic, but the shape of the work means that I have to jump from topic to topic. I'll continue to periodically organize posts by topic and highlight them on the blog index.