Home

The Long Slog: Running Programs I Didn't Write

Here I start to run real programs.

debootstrap:

• set -e -- I use set -o errexit

Flags Parsing

- this went into a huge flags parsing digression.  TWO flags parsers.
- case that convinced me: set -o pipefail vs. read -t1.0
  - another special case: echo

• unset builtin -- set and unset aren't opposites!
  • exec 4>&1 -- disabled an EBADF error. Honestly I don't quite understand why scripts do this.
• umask -- stubbed out
• trap -- stubbed out
• functions: in_path
  • xtrace might help with this. what is going wrong?
  • this is because of 'export PATH' -- it's erasing the variable!
    • not implemented correctly
    • needs to be dynamic
    • variables initialized from environ block at startup
• set -u is different than set -u -- - very tricky! double dash usually isn't significant, but here it is!
  • partial implementation of xtrace: useful for figuring out what's going on.

toybox

• echo -e -- stubbed out - need to also implement lexstate for $'' too - \n - \t
• patsub

Python configure.

• python configure
  • : is like true, but a special builtin. This seems like copying cruft now, but OK.
  • takes 17 seconds to parse!!!! 16,000 line script. On a core i7. So 1000 lines a second. Abysmally slow!
  • maybe parse line-by-line
  • $- -- special var with flags

SUCCESS! I can now run Python's configure script.

This is sort of cheating since configure scripts are meant to be extremely portable and use none of bash's features. But I'm willing to go with it to get a release out.

Python config.status

• config.status -- Redirect bug! This one was confusing. Gah i need to understand redirects better. Good thing I refactored them. The code is now much easier to follow.

Another candidate: && || bugs

configure scripts use a lot of awk, here docs, and MULTIPLE LEVELS of code genration.

configure.ac is an autoconf/m4 script that generates a configure script, which generates a shell script config.status, which generates Makefile.pre and pyconfig.h. What a huge mess!

TODO: Try to run QEMU's hand-written script?

• 7/1: SKIPPING config.status. Handled redirect errors though.

• REFACTORING and bug cleanup: overhaul of Mem interface to be much smaller. Instead of SetLocals, SetLocalOrGlobal, I have a single SetVar.

  • export -n -- exporting is a SEPARATE BIT

Env variables: 1. At startup, all environment variables are turned into global shell variables with the 'export' flag set. Previously, I was doing things lazily: the $HOME lookup fall back on getenv(). But this doesn't work because there is this eparate bit of export state.

1. you can mutate BOTH THE VALUE AND THE EXPORT BIT (with export, export -n) (dash and mksh don't let you manipulate the export bit)

2. you can export stuff in ANY SCOPE

Other issues:

static vs. dynamic parsing of export. Switched to dynamic because of export $1 in aboriginal. Still struggling.

unset builtin -- this could be parsed statically too. unset a[1] works! Just like delete in Python

Allowed implementing LHS -- ++a[0], etc. Needed the LValue concept for this.

7/1: Going back to descriptor bug.

HARDEST BUG EVER. - reading through generated config.status is really annoying. it uses every weird part of hsell. - I can't log!!! Because stderr is getting redirected. - Opening a separate disk file causes some weird dup2() error I don't understand.

RUNTIME.ASDL and OSH.ASDL need to be closed! Because of my stupid metaprogramming.

Goddamnit.

/dev/urandom still open -- a problem with using Python interpreter.

I don't even really understand what happened still! Gah!

It took me WAY TO LONG to look at /proc/$PID/fd/ of an OSH process! It has symlinks to open files.

Hard-coded descriptors are an anti-pattern! That means the parent process cannot have any open files! But I guess that's what CLO_EXEC means. I learned that the hard way -- through hours of debugging!

next feature:

eval set a b

-- I'm not joining the arguments!

No gratituous incompatibilities. I made this "fixup" decision along time ago, when I had not come to terms with how inelegant shell/bash can be. Now I am in "copying bugs" mode.

Updated Roadmap

1. build: finish configure and install. Linux-only for now. At least two distros! Ubuntu, test on Alpine Linux which should be sufficiently different. GNU make (-j) and bash required?
2. Error messages. Pipefail demo. Which one failed! This is good for showing off.
3. Run a real program. configure. DONE.
4. Spec tests: failures below 100. Almost there. Probably just implement LHS array?
   • a cheap one would be brace expansion! 12 failing tests.

Later, maybe OSH 0.2

1. POSIX build environment. Raspian? BSD? make -j or bash?
2. More polish of error messages.
3. work on debootstrap / toybox
   • toybox: patsub, echo -e
   • debootstrap: not sure what.
4. No particular goal here; make it organic. Might rise about 100 if we implement failing tests.