What am I working on? In the last post, I linked to bash scripts that built a slice of the CPython VM.
I'm now using these scripts as the foundation for a rewrite of Python's autotools-based build system. This will be the build system for Oil and OVM. Roughly speaking, OVM is the part of Oil that is native code.
Why do this?
I think writing good build systems is hard, and this experience is supporting that belief. Read on for details.
bin/oilas an App Bundle
To end users and distro packagers, Oil should look like a C program. The use
of CPython is an implementation detail. Thus,
bin/oil will be a single
file with two components:
.pycfiles to reduce startup time. We also need the
.pysource to show code snippets in tracebacks.
Python app bundles are familiar to me because a decade ago I submitted
a patch to Python to allow it to directly execute
.zip files with
.pyc files (python-dev thread).
I'm taking it further now:
.zipfile, I'm concatenating an ELF file +
.zipfile contains the bytecode.
Why a single file? I want to be completely divorced from Python packaging. I
don't want anything to do with
site-packages and so forth, which are known to
cause deployment problems.
(Historical note: Phillip J. Eby rewrote the patch to support directories with
__main__.py in addition to .zip files. This was available in Python 2.6, but
it was relatively unknown until Python 3.5 added the zipapp module, a
result of PEP 441.)
Building OVM is a different problem than building Python. This section compares them along three dimensions.
(I suppose Oil will run on Windows 10 via the confusingly-named Windows Subsystem for Linux. I find it amazing that this Windows kernel feature is marketed as Bash on Windows. A program that is over three decades old is now a hot new feature!)
Dependencies and System Requirements:
write(), etc. The "standard library" consists of separate programs like coreutils or busybox.
Extensibility with Dynamic Linking:
Python-2.7.13/setup.pyfile that controls this is ~2300 lines long!
In summary, these three differences mean that Python requires a complex build system, but Oil can have a relatively simple one.
I don't think Oil will need autotools. If you can think of a reason it might, leave a comment. The main issue I see is detecting the readline library, but that can be done with a POSIX shell script written by hand.
Boil is the name I have in mind for the Make-like build tool in Oil. Its relationship to Make is similar to that of Oil and bash, although not identical.
This is a big topic, so for now I'll list a few problems that I found with Python's Makefile. These problems appear in most Makefiles because the solutions are hard to express in Make.
CFLAGS. It's annoying to do a
make cleanbetween debug and release builds.
setup.pydidn't pass the right flags.
gcc -Mscheme. I plan to reorganize the code for OVM, and this may be a nuisance.
I'm knee-deep in this work right now. It's not finished by any means, but I'm happy to slow down and examine Make more closely. I revisited the research on Make I did before starting the blog last year.
I would say that shell has bad syntax and good semantics, but Make has both bad
syntax and bad semantics. It's not statically parseable,
and there are many weird mechanisms like double-colon rules, order-only
Boil is still two steps away. A rough plan is:
So Make and Boil will a major blog topic, but the shape of the work means that I have to jump from topic to topic. I'll continue to regularly organize posts by topic and highlight them on the blog index.