Overview: Translating Oil to C++

category total lines num files
mycpp Translator 4,351 10
mycpp Test Data 2,190 26
Zephyr ASDL 2,911 10
pgen2 Parser Generator 1,717 8
Other Code Generators 2,068 8
Hand-written C++ Code 2,445 27
Old mycpp Runtime 1,873 3
New Garbage-Collected Runtime 3,191 10
Unit tests in C++ 3,839 10
Incremental C++ Build 1,595 6
OSH (and common libraries) 37,401 79
Spec Tests 37,035 143
Gold Tests 2,725 26
Generated C+ Code 77,361 26

Related Documents

The README for oilshell/oil has another overview of the repository.

mycpp Translator

This prototype uses the MyPy frontend to translate statically-typed Python to C++. The generated code calls a small runtime which implements things like List[T], Dict[K, V], and Python's len().

mycpp/const_pass.py                             524
mycpp/cppgen_pass.py                          2,640
mycpp/crash.py                                   35
mycpp/debug_pass.py                             459
mycpp/format_strings.py                         110
mycpp/mycpp_main.py                             355
mycpp/mylib.py                                  160
mycpp/pass_state.py                              54
mycpp/util.py                                    14

4,351 lines in 10 files  

mycpp Test Data

Small Python examples that translate to C++, compile, and run.

mycpp/examples/asdl_generated.py                 62
mycpp/examples/cartesian.py                      75
mycpp/examples/cgi.py                            59
mycpp/examples/classes.py                       141
mycpp/examples/containers.py                    170
mycpp/examples/control_flow.py                  110
mycpp/examples/escape.py                         67
mycpp/examples/fib_iter.py                       54
mycpp/examples/fib_recursive.py                  49
mycpp/examples/files.py                          53
mycpp/examples/__init__.py                        0
mycpp/examples/length.py                        106
mycpp/examples/lexer_main.py                     48
mycpp/examples/loops.py                         177
mycpp/examples/modules.py                        78
mycpp/examples/parse.py                         232
mycpp/examples/pgen2_demo.py                    106
mycpp/examples/scoped_resource.py               176
mycpp/examples/strings.py                        61
mycpp/examples/test_cast.py                      59
mycpp/examples/test_conditional.py               67
mycpp/examples/test_hoist.py                     61
mycpp/examples/test_switch.py                    42
mycpp/examples/tuple_return_value.py             53
mycpp/examples/varargs.py                        84

2,190 lines in 26 files  

Zephyr ASDL

A DSL for algebraic data types, borrowed from Python. Oil is the most strongly typed Bourne shell implementation!

asdl/asdl_.py                                   253
asdl/format.py                                  492
asdl/front_end.py                               488
asdl/gen_cpp.py                                 631
asdl/gen_python.py                              528
asdl/pybase.py                                   63
asdl/runtime.py                                  41
asdl/tool.py                                    281
asdl/visitor.py                                 134

2,911 lines in 10 files  

pgen2 Parser Generator

An LL(1) parser generator used to parse Oil expressions. Also borrowed from CPython.

pgen2/driver.py                                 101
pgen2/grammar.py                                222
pgen2/parse.py                                  209
pgen2/pgen.py                                   428
pgen2/pnode.py                                   29
pgen2/tokenize.py                               579
pgen2/token.py                                  149

1,717 lines in 8 files  

Other Code Generators

In order to make Oil statically typed, we had to abandon Python reflection and use C++ source code generation instead. The lexer, flag definitions, and constants can be easily compiled to C++.

core/optview_gen.py                              94
frontend/consts_gen.py                          563
frontend/flag_gen.py                            482
frontend/lexer_gen.py                           488
frontend/option_gen.py                          115
oil_lang/grammar_gen.py                         226
osh/arith_parse_gen.py                          100

2,068 lines in 8 files  

Hand-written C++ Code

Includes OS bindings. Small C++ files like cpp/osh_arith_parse.{cc,h} correspond to larger Python files like osh/arith_parse.py.

cpp/dumb_alloc.cc                                86
cpp/dumb_alloc.h                                 17
cpp/leaky_core.cc                               269
cpp/leaky_core_error.h                          171
cpp/leaky_core.h                                101
cpp/leaky_core_pyerror.h                         90
cpp/leaky_frontend_flag_spec.cc                 294
cpp/leaky_frontend_flag_spec.h                  136
cpp/leaky_frontend_match.cc                     107
cpp/leaky_frontend_match.h                       66
cpp/leaky_frontend_tdop.cc                       16
cpp/leaky_frontend_tdop.h                        47
cpp/leaky_libc.cc                               142
cpp/leaky_libc.h                                 56
cpp/leaky_osh.cc                                154
cpp/leaky_osh_eval_stubs.h                       49
cpp/leaky_osh.h                                  50
cpp/leaky_pgen2.cc                               15
cpp/leaky_pgen2.h                                53
cpp/leaky_preamble.h                             82
cpp/leaky_pylib.cc                               50
cpp/leaky_pylib.h                                24
cpp/leaky_stdlib.cc                             165
cpp/leaky_stdlib.h                              114
cpp/qsn.h                                        67
cpp/segfault_handler.h                           24

2,445 lines in 27 files  

Old mycpp Runtime

This implementation has no garbage collection; it allocates memory forever.

mycpp/mylib_old.cc                              560
mycpp/mylib_old.h                             1,313

1,873 lines in 3 files  

New Garbage-Collected Runtime

Uses a simple Cheney / semi-space collector.

mycpp/gc_builtins.cc                            448
mycpp/gc_builtins.h                             416
mycpp/gc_heap.cc                                303
mycpp/gc_heap.h                                 557
mycpp/gc_mylib.cc                               217
mycpp/gc_mylib.h                                235
mycpp/gc_types.h                                875
mycpp/leaky_types.cc                            110
mycpp/myerror.h                                  30

3,191 lines in 10 files  

Unit tests in C++

The goal is to make the spec tests pass, but unit tests are helpful too.

cpp/gc_binding_test.cc                           50
cpp/leaky_binding_test.cc                       371
cpp/leaky_flag_spec_test.cc                     134
mycpp/gc_builtins_test.cc                       809
mycpp/gc_heap_test.cc                         1,007
mycpp/gc_mylib_test.cc                          172
mycpp/gc_stress_test.cc                         208
mycpp/leaky_types_test.cc                       191
mycpp/mylib_old_test.cc                         897

3,839 lines in 10 files  

Incremental C++ Build

cpp/NINJA-steps.sh                              260
cpp/NINJA_subgraph.py                           361
mycpp/NINJA-steps.sh                            305
mycpp/NINJA_subgraph.py                         605
NINJA_config.py                                  64

1,595 lines in 6 files  

OSH (and common libraries)

This is the input to the translators, written in statically-typed Python. Note that bash is at least 140K lines of code, and OSH implements a large part of bash and more.

bin/oil.py                                      355
core/alloc.py                                   166
core/ansi.py                                     17
core/completion.py                            1,261
core/comp_ui.py                                 585
core/dev.py                                     598
core/error.py                                   198
core/executor.py                                567
core/main_loop.py                               361
core/optview.py                                  57
core/process.py                               1,510
core/pyerror.py                                  71
core/pyos.py                                    384
core/pyutil.py                                  247
core/runtime.asdl                               194
core/shell_native.py                            535
core/shell.py                                   689
core/state.py                                 2,385
core/ui.py                                      396
core/util.py                                     93
core/vm.py                                      213
frontend/args.py                                673
frontend/builtin_def.py                         164
frontend/consts.py                              342
frontend/flag_def.py                            327
frontend/flag_spec.py                           391
frontend/id_kind_def.py                         638
frontend/lexer_def.py                         1,012
frontend/lexer.py                               330
frontend/location.py                             99
frontend/match.py                               254
frontend/option_def.py                          382
frontend/parse_lib.py                           477
frontend/py_reader.py                            81
frontend/reader.py                              141
frontend/signal_def.py                           45
frontend/syntax_abbrev.py                       144
frontend/syntax.asdl                            582
frontend/typed_args.py                           70
frontend/types.asdl                              41
mycpp/mylib.pyi                                  74
native/fanos.c                                  294
native/fastlex.c                                223
native/fastlex.pyi                               13
native/libc.c                                   402
native/libc.pyi                                  11
native/posix_.pyi                               211
native/yajl.pyi                                   3
osh/arith_parse.py                              184
osh/bool_parse.py                               269
osh/bool_stat.py                                136
osh/braces.py                                   498
osh/builtin_assign.py                           514
osh/builtin_bracket.py                          280
osh/builtin_comp.py                             518
osh/builtin_lib.py                               94
osh/builtin_meta.py                             435
osh/builtin_misc.py                             875
osh/builtin_printf.py                           477
osh/builtin_process2.py                         223
osh/builtin_process.py                          131
osh/builtin_pure.py                             960
osh/builtin_trap.py                             225
osh/cmd_eval.py                               1,859
osh/cmd_parse.py                              2,415
osh/glob_.py                                    498
osh/history.py                                  184
osh/prompt.py                                   300
osh/sh_expr_eval.py                           1,048
osh/split.py                                    307
osh/string_ops.py                               454
osh/tdop.py                                     333
osh/word_compile.py                             257
osh/word_eval.py                              2,299
osh/word_parse.py                             1,838
osh/word_.py                                    821
qsn_/qsn_native.py                               75
qsn_/qsn.py                                     588

37,401 lines in 79 files  

Spec Tests

A comprehensive test suite that compares OSH against other shells. If OSH passes these tests in BOTH Python and C++, it means that the translation works.

spec/alias.test.sh                              536
spec/append.test.sh                             291
spec/arith-context.test.sh                       81
spec/arith.test.sh                              631
spec/array-compat.test.sh                        96
spec/array.test.sh                              637
spec/assign-deferred.test.sh                    105
spec/assign-dialects.test.sh                     59
spec/assign-extended.test.sh                    810
spec/assign.test.sh                             672
spec/assoc.test.sh                              631
spec/assoc-zsh.test.sh                           53
spec/background.test.sh                         182
spec/ble-features.test.sh                       644
spec/ble-idioms.test.sh                         168
spec/blog1.test.sh                               95
spec/blog2.test.sh                               49
spec/blog-other1.test.sh                         55
spec/brace-expansion.test.sh                    441
spec/bugs.test.sh                               169
spec/builtin-bash.test.sh                       346
spec/builtin-bracket.test.sh                    564
spec/builtin-completion.test.sh                 415
spec/builtin-dirs.test.sh                       205
spec/builtin-eval-source.test.sh                249
spec/builtin-getopts.test.sh                    318
spec/builtin-io.test.sh                         751
spec/builtin-printf.test.sh                     888
spec/builtins2.test.sh                          233
spec/builtin-special.test.sh                    111
spec/builtins.test.sh                           420
spec/builtin-times.test.sh                       15
spec/builtin-trap.test.sh                       410
spec/builtin-vars.test.sh                       710
spec/case_.test.sh                              165
spec/command-parsing.test.sh                     63
spec/command-sub.test.sh                        272
spec/command_.test.sh                           192
spec/comments.test.sh                            11
spec/dbracket.test.sh                           436
spec/dparen.test.sh                             202
spec/empty-bodies.test.sh                        24
spec/errexit-oil.test.sh                        755
spec/errexit.test.sh                            393
spec/exit-status.test.sh                        238
spec/explore-parsing.test.sh                     42
spec/extglob-files.test.sh                      391
spec/extglob-match.test.sh                      400
spec/fatal-errors.test.sh                       186
spec/for-expr.test.sh                            92
spec/func-parsing.test.sh                       105
spec/glob.test.sh                               396
spec/hay-isolation.test.sh                      169
spec/hay-meta.test.sh                           200
spec/hay.test.sh                                645
spec/here-doc.test.sh                           365
spec/if_.test.sh                                 56
spec/interactive.test.sh                        216
spec/introspect.test.sh                         237
spec/let.test.sh                                 23
spec/loop.test.sh                               408
spec/nameref.test.sh                            642
spec/nix-idioms.test.sh                         151
spec/nocasematch-match.test.sh                   69
spec/oil-array.test.sh                           38
spec/oil-assign.test.sh                         359
spec/oil-bin.test.sh                             17
spec/oil-blocks.test.sh                         377
spec/oil-bugs.test.sh                            48
spec/oil-builtin-argparse.test.sh                35
spec/oil-builtin-describe.test.sh                38
spec/oil-builtin-error.test.sh                  424
spec/oil-builtin-pp.test.sh                      47
spec/oil-builtin-process.test.sh                 85
spec/oil-builtin-shopt.test.sh                  179
spec/oil-builtins.test.sh                       568
spec/oil-case.test.sh                            27
spec/oil-command-sub.test.sh                    109
spec/oil-demo.test.sh                            59
spec/oil-expr-sub.test.sh                        40
spec/oil-expr.test.sh                           881
spec/oil-for.test.sh                            173
spec/oil-funcs-builtin.test.sh                  200
spec/oil-funcs-external.test.sh                  87
spec/oil-interactive.test.sh                     15
spec/oil-json.test.sh                           121
spec/oil-keywords.test.sh                        37
spec/oil-multiline.test.sh                      179
spec/oil-options-assign.test.sh                 103
spec/oil-options.test.sh                        781
spec/oil-proc.test.sh                           270
spec/oil-regex.test.sh                          689
spec/oil-reserved.test.sh                        46
spec/oil-scope.test.sh                          764
spec/oil-slice-range.test.sh                    162
spec/oil-special-vars.test.sh                    48
spec/oil-string.test.sh                         344
spec/oil-tuple.test.sh                           44
spec/oil-usage.test.sh                           22
spec/oil-user-feedback.test.sh                  209
spec/oil-var-sub.test.sh                         45
spec/oil-with-sh.test.sh                         77
spec/oil-word-eval.test.sh                       81
spec/oil-xtrace.test.sh                         478
spec/osh-only.test.sh                            78
spec/parse-errors.test.sh                       192
spec/pipeline.test.sh                           166
spec/posix.test.sh                              153
spec/process-sub.test.sh                        205
spec/prompt.test.sh                             254
spec/quote.test.sh                              248
spec/redirect.test.sh                           654
spec/regex.test.sh                              344
spec/serialize.test.sh                          229
spec/shell-grammar.test.sh                      205
spec/sh-func.test.sh                            150
spec/sh-options.test.sh                         694
spec/sh-usage.test.sh                            90
spec/smoke.test.sh                              123
spec/strict-options.test.sh                     249
spec/subshell.test.sh                            23
spec/tea-func.test.sh                           288
spec/tilde.test.sh                              172
spec/TODO-deprecate.test.sh                      14
spec/toysh-posix.test.sh                        347
spec/toysh.test.sh                              144
spec/type-compat.test.sh                        107
spec/var-num.test.sh                             39
spec/var-op-bash.test.sh                        309
spec/var-op-len.test.sh                         219
spec/var-op-patsub.test.sh                      291
spec/var-op-slice.test.sh                       315
spec/var-op-strip.test.sh                       357
spec/var-op-test.test.sh                        339
spec/var-ref.test.sh                            485
spec/vars-bash.test.sh                           27
spec/vars-special.test.sh                       456
spec/var-sub-quote.test.sh                      312
spec/var-sub.test.sh                             74
spec/word-eval.test.sh                           62
spec/word-split.test.sh                         387
spec/xtrace.test.sh                             344

37,035 lines in 143 files  

Gold Tests

Another suite that tests shells "from the outside". Instead of making explicit assertions, we verify that OSH behaves like bash.

test/gold/abuild.sh                               9
test/gold/and-or.sh                              27
test/gold/big-here-doc.sh                     1,531
test/gold/case-in-subshell.sh                    37
test/gold/char-class.sh                          33
test/gold/command-sub-2.sh                       12
test/gold/command-sub.sh                         39
test/gold/comments.sh                            22
test/gold/complex-here-docs.sh                  192
test/gold/configure-bug.sh                       13
test/gold/de1.sh                                 24
test/gold/declare.sh                             28
test/gold/dollar-sq.sh                           42
test/gold/echo-e.sh                              36
test/gold/errexit-confusion.sh                  139
test/gold/errexit.sh                            121
test/gold/export.sh                              65
test/gold/glob.sh                                26
test/gold/lineno.sh                              23
test/gold/nix.sh                                 32
test/gold/readlink.sh                            61
test/gold/readonly.sh                            24
test/gold/scope.sh                              126
test/gold/strip-op-char-class.sh                  6
test/gold/word-eval.sh                           57

2,725 lines in 26 files  

Generated C+ Code

mycpp generates the big file _build/cpp/osh_eval.cc. Other programs like Zephyr ASDL and re2c generate other files.

_build/cpp/arg_types.cc                         414
_build/cpp/arg_types.h                          403
_build/cpp/arith_parse.cc                       129
_build/cpp/consts.cc                          1,354
_build/cpp/consts.h                              45
_build/cpp/core_optview.h                       117
_build/cpp/hnode_asdl.cc                         39
_build/cpp/hnode_asdl.h                         149
_build/cpp/id_kind_asdl.cc                      856
_build/cpp/id_kind_asdl.h                       441
_build/cpp/option_asdl.cc                       364
_build/cpp/option_asdl.h                        197
_build/cpp/osh_eval.cc                       33,902
_build/cpp/osh_eval.h                           342
_build/cpp/runtime_asdl.cc                    1,573
_build/cpp/runtime_asdl.h                     1,186
_build/cpp/syntax_asdl.cc                     6,829
_build/cpp/syntax_asdl.h                      4,278
_build/cpp/types_asdl.cc                        107
_build/cpp/types_asdl.h                          88
_devbuild/gen/find_nt.h                          13
_devbuild/gen/grammar_nt.h                       93
_devbuild/gen/id.h                              382
_devbuild/gen/osh-lex.h                      24,024
_devbuild/gen/osh-types.h                        36

77,361 lines in 26 files