Why Sponsor Oils? | blog | oilshell.org
What am I working on? In the last post, I linked to bash scripts that built a slice of the CPython VM.
I'm now using these scripts as the foundation for a rewrite of Python's autotools-based build system. This will be the build system for Oil and OVM. Roughly speaking, OVM is the part of Oil that is native code.
Why do this?
I think writing good build systems is hard, and this experience is supporting that belief. Read on for details.
bin/oil
as an App BundleTo end users and distro packagers, Oil should look like a C program. The use
of CPython is an implementation detail. Thus, bin/oil
will be a single
file with two components:
libc
called pylibc..pyc
files to
reduce startup time. We also need the .py
source to show code snippets in
tracebacks.Python app bundles are familiar to me because a decade ago I submitted
a patch to Python to allow it to directly execute .zip
files with
.py
and .pyc
files (python-dev thread).
I'm taking it further now:
.zip
file, I'm
concatenating an ELF file + .zip
file. The .zip
file contains the
bytecode.libc
itself.)Why a single file? I want to be completely divorced from Python packaging. I
don't want anything to do with site-packages
and so forth, which are known to
cause deployment problems.
(Historical note: Phillip J. Eby rewrote the patch to support directories with
__main__.py
in addition to .zip files. This was available in Python 2.6, but
it was relatively unknown until Python 3.5 added the zipapp module, a
result of PEP 441.)
Building OVM is a different problem than building Python. This section compares them along three dimensions.
Platform Portability:
(I suppose Oil will run on Windows 10 via the confusingly-named Windows Subsystem for Linux. I find it amazing that this Windows kernel feature is marketed as Bash on Windows. A program that is over three decades old is now a hot new feature!)
Dependencies and System Requirements:
libffi
for ctypes, sqlite
, etc.fork()
, exec()
, pipe()
, read()
, write()
, etc. The "standard
library" consists of separate programs like coreutils or busybox.Extensibility with Dynamic Linking:
_json.so
. The Python-2.7.13/setup.py
file that
controls this is ~2300 lines long!In summary, these three differences mean that Python requires a complex build system, but Oil can have a relatively simple one.
I don't think Oil will need autotools. If you can think of a reason it might, leave a comment. The main issue I see is detecting the readline library, but that can be done with a POSIX shell script written by hand.
Boil is the name I have in mind for the Make-like build tool in Oil. Its relationship to Make is similar to that of Oil and bash, although not identical.
This is a big topic, so for now I'll list a few problems that I found with Python's Makefile. These problems appear in most Makefiles because the solutions are hard to express in Make.
CFLAGS
. It's annoying to do a
make clean
between debug and release builds._json.so
because setup.py
didn't pass the right flags.gcc -M
scheme.
I plan to reorganize the code for OVM, and this may be a nuisance.I'm knee-deep in this work right now. It's not finished by any means, but I'm happy to slow down and examine Make more closely. I revisited the research on Make I did before starting the blog last year.
I would say that shell has bad syntax and good semantics, but Make has both bad
syntax and bad semantics. It's not statically parseable,
and there are many weird mechanisms like double-colon rules, order-only
dependencies, .ONESHELL
, .SECONDARYEXPANSION
, etc.
Boil is still two steps away. A rough plan is:
So Make and Boil will a major blog topic, but the shape of the work means that I have to jump from topic to topic. I'll continue to regularly organize posts by topic and highlight them on the blog index.