1 OPy Compiler and Byterun
2 ========================
3
4 The OPy compiler is a Python bytecode compiler written in Python. See
5 [Building Oil with the OPy Bytecode Compiler][oil-with-opy]. It's currently
6 used to translate Python source code in Oil to `.pyc` files.
7
8 The `byterun/` directory is a fork of [byterun][]. It's an experiment for
9 learning what it will take to write a minimal interpreter for Oil. It can
10 currently run all Oil unit tests, but isn't otherwise used.
11
12 [oil-with-opy]: http://www.oilshell.org/blog/2018/03/04.html
13
14 [byterun]: http://aosabook.org/en/500L/a-python-interpreter-written-in-python.html
15
16 ## 2022 Update: OPy Will be "Replaced" By mycpp / Pea
17
18 A bytecode interpreter isn't fast enough to run Oil. We still have the
19 double-interpretation problem.
20
21 ## Getting started
22
23 Do the "Quick Start" in "in https://github.com/oilshell/oil/wiki/Contributing .
24
25 Then build the `py27.grammar` file:
26
27 $ make _build/opy/py27.grammar.pickle
28
29 After Oil is setup, we can try out OPy. Run these commands (and let me know if
30 any of them doesn't work):
31
32 oil$ cd opy
33 opy$ ../bin/opyc run gold/hello_py2.py # basic test of compiler and runtime
34
35 Compile Oil with the OPy compiler:
36
37 $ ./build.sh oil-repo # makes _tmp/repo-with-opy and _tmp/repo-with-cpython
38
39 Run Oil unit tests, compiled with OPy, under **CPython**:
40
41 $ ./test.sh oil-unit
42
43 Run Oil unit tests, compiled with OPy, under **byterun**:
44
45 $ ./test.sh oil-unit-byterun # Run Oil unit tests, compiled with OPy, under CPython
46
47 Gold tests in `gold/` compare the output of CPython vs. byterun:
48
49 $ ./test.sh gold
50
51 Oil spec tests under byterun (slow):
52
53 opy$ ./test.sh spec smoke # like $REPO_ROOT/test/spec.sh smoke
54 opy$ ./test.sh spec all # like $REPO_ROOT/test/spec.sh all
55
56 FYI, they can be run manually like this:
57
58 $ gold/regex_compile.py # run with CPython
59 $ ../bin/opyc run gold/regex_compile.py
60
61 Demo of the speed difference between OSH under CPython and OSH under byterun:
62
63 ./demo.sh osh-byterun-speed
64
65 ## OPy Compiler Regtest
66
67 This uses an old snapshot of the repo in `_regtest/`.
68
69 ./regtest.sh compile
70 ./regtest.sh verify-golden
71
72 ## Notes on Three OPy Builds
73
74 - `$REPO_ROOT/_build/oil/bytecode-opy`: Bytecode for the release binary. Built
75 by `Makefile`.
76 - `$REPO_ROOT/opy/_tmp/repo-with-opy`: The entire repo with OPy. For running
77 Oil unit/spec tests under byterun, etc. Built by `./build.sh oil-repo`.
78 - `$REPO_ROOT/opy/_tmp/regtest`: The snapshot of Python files in `opy/_regtest`
79 are compiled, so we are insensitive to repo changes. Built by `./regtest.sh
80 compile`.
81
82 ## OPy Compiler Divergences from CPython
83
84 ### Lexer
85
86 - I don't remember where exactly, but I ran into a bug lexing the CPython test
87 suite. IIRC, CPython's lexer was more lenient about adjacent tokens without
88 spaces than `tokenize.py`.
89 - `heapq.py` had `-*- coding: latin-1 -*-`, which causes problems. OPy
90 should require `utf-8` source anyway.
91
92 ### Parser
93
94 - I ran into a bug where a file like `d = {}`, without a trailing newline,
95 gives a parse error. Adding the newline fixes it.
96 - print statements aren't allowed; we force Python 3-style `print(x, y,
97 file=sys.stderr)`. I think this is because the parser doesn't know about
98 `__future__` statements, so it can't change the parsing mode on the fly.
99
100 ### Bytecode Compiler
101
102 - I think there are no `LOAD_FAST` bytecodes generated? TODO: Make a bytecode
103 histogram using `opy/misc/inspect_pyc`.
104 - The OPy bytecode is bigger than the CPython bytecode! Why is that?
105