What am I working on? In the last post, I linked to bash scripts that built a slice of the CPython VM.
I'm now using these scripts as the foundation for a rewrite of Python's autotools-based build system. This will be the build system for Oil and OVM. You can think of OVM as the part of Oil that is native code.
Why do this?
I think writing good build systems is hard, and this experience is supporting that belief. Read on for details.
bin/oilas an App Bundle
Oil's use of CPython should be an implementation detail. To users and
packagers, Oil should look like a C program. Thus,
bin/oil will be a single
file with two components:
.pycfiles to save work at startup. We also need
.pyfiles to show code snippets in tracebacks.
This is familiar to me because a decade ago I submitted a patch to
Python to allow it to directly execute
I'm taking it further now by:
.zipfile, I'm concatenating an ELF file +
.zipfile. That is, the native code will load the zipped bytecode in the same file.
Why a single file? I want to be completely divorced from Python packaging. I
don't want anything to do with
site-packages and so forth, which are known to
cause deployment problems.
(Historical note: Phillip J. Eby rewrote the patch to support directories with
__main__.py in addition to .zip files. This was available in Python 2.6, but
it was relatively unknown until Python 3.5 added the zipapp module, a
result of PEP 441.)
Building OVM is a different problem than building Python. This section compares them along three dimensions.
(I suppose it will run on Windows 10 via the confusingly-named Windows Subsystem for Linux. I find it amazing that this Windows kernel feature is marketed as Bash on Windows. A program that is over three decades old is now a hot new feature!)
Extensibility with Dynamic Linking:
Python-2.7.13/setup.pythat controls this is ~2300 lines long!
This means that Python requires a complex build system, but Oil can have a relatively simple one.
I don't think Oil will need autotools. If you can think of a reason it might, leave a comment. The main issue I see is detecting the readline library, but that can be done with a POSIX shell script written by hand.
Boil is the name I have in mind for the Make-like build tool in Oil. Its relationship to Make is similar to that of Oil and bash, although not identical.
This is a big topic, so for now I'll list a few problems that I found with Python's Makefile. These problems appear in most Makefiles because it's hard to express the solution in Make.
CFLAGS. It's annoying to do a
make cleanbetween debug and release builds.
setup.pydidn't pass the right flags.
gcc -Mscheme. I plan to reorganize the code for OVM, and this may be a nuisance.
I'm knee-deep in this work right now. It's not finished by any means, but I'm happy to slow down and examine Make more closely. I revisited the research I did before starting the blog last year.
I would say that shell has bad syntax and good semantics, but Make has both bad
syntax and bad semantics. It's not statically parseable,
and there are many weird mechanisms like double-colon rules, order-only
Boil is still two steps away. A rough plan is:
So Make and Boil will a major blog topic, but the shape of the work means that I have to jump from topic to topic. I'll continue to periodically organize posts by topic and highlight them on the blog index.