Overview: Translating Oil to C++

category total lines num files
mycpp Translator 4,294 10
mycpp Test Data 2,182 25
Zephyr ASDL 2,904 10
pgen2 Parser Generator 1,717 8
Other Code Generators 2,062 8
Hand-written C++ Code 2,370 38
Old mycpp Runtime 1,854 3
New Garbage-Collected Runtime 3,068 7
Unit tests in C++ 3,106 7
OSH (and common libraries) 37,090 77
Generated C+ Code 84,799 26
Spec Tests 35,389 131
Gold Tests 2,725 26

Related Documents

The README for oilshell/oil has another overview of the repository.

mycpp Translator

This prototype uses the MyPy frontend to translate statically-typed Python to C++. The generated code calls a small runtime which implements things like List[T], Dict[K, V], and Python's len().

mycpp/const_pass.py                             526
mycpp/cppgen_pass.py                          2,602
mycpp/crash.py                                   37
mycpp/debug_pass.py                             459
mycpp/format_strings.py                         110
mycpp/mycpp_main.py                             346
mycpp/mylib.py                                  148
mycpp/pass_state.py                              54
mycpp/util.py                                    12

4,294 lines in 10 files  

mycpp Test Data

Small Python examples that translate to C++, compile, and run.

mycpp/examples/asdl_generated.py                 62
mycpp/examples/cartesian.py                      75
mycpp/examples/cgi.py                            59
mycpp/examples/classes.py                       141
mycpp/examples/containers.py                    162
mycpp/examples/control_flow.py                  110
mycpp/examples/escape.py                         67
mycpp/examples/fib_iter.py                       54
mycpp/examples/fib_recursive.py                  49
mycpp/examples/files.py                          53
mycpp/examples/length.py                        106
mycpp/examples/lexer_main.py                     48
mycpp/examples/loops.py                         177
mycpp/examples/modules.py                        78
mycpp/examples/parse.py                         232
mycpp/examples/pgen2_demo.py                    106
mycpp/examples/scoped_resource.py               176
mycpp/examples/strings.py                        61
mycpp/examples/test_cast.py                      59
mycpp/examples/test_conditional.py               67
mycpp/examples/test_hoist.py                     61
mycpp/examples/test_switch.py                    42
mycpp/examples/tuple_return_value.py             53
mycpp/examples/varargs.py                        84

2,182 lines in 25 files  

Zephyr ASDL

A DSL for algebraic data types, borrowed from Python. Oil is the most strongly typed Bourne shell implementation!

asdl/asdl_.py                                   253
asdl/format.py                                  494
asdl/front_end.py                               488
asdl/gen_cpp.py                                 622
asdl/gen_python.py                              529
asdl/pybase.py                                   63
asdl/runtime.py                                  41
asdl/tool.py                                    280
asdl/visitor.py                                 134

2,904 lines in 10 files  

pgen2 Parser Generator

An LL(1) parser generator used to parse Oil expressions. Also borrowed from CPython.

pgen2/driver.py                                 101
pgen2/grammar.py                                222
pgen2/parse.py                                  209
pgen2/pgen.py                                   428
pgen2/pnode.py                                   29
pgen2/tokenize.py                               579
pgen2/token.py                                  149

1,717 lines in 8 files  

Other Code Generators

In order to make Oil statically typed, we had to abandon Python reflection and use C++ source code generation instead. The lexer, flag definitions, and constants can be easily compiled to C++.

core/optview_gen.py                              89
frontend/consts_gen.py                          551
frontend/flag_gen.py                            485
frontend/lexer_gen.py                           495
frontend/option_gen.py                          115
oil_lang/grammar_gen.py                         226
osh/arith_parse_gen.py                          101

2,062 lines in 8 files  

Hand-written C++ Code

Includes OS bindings. Small C++ files like cpp/osh_arith_parse.{cc,h} correspond to larger Python files like osh/arith_parse.py.

cpp/core_error.h                                170
cpp/core_pyerror.h                               74
cpp/core_pyos.cc                                131
cpp/core_pyos.h                                  80
cpp/core_pyutil.cc                               54
cpp/core_pyutil.h                                47
cpp/dumb_alloc.cc                                86
cpp/dumb_alloc.h                                 17
cpp/fcntl_.cc                                    30
cpp/fcntl_.h                                     16
cpp/frontend_flag_spec.cc                       328
cpp/frontend_flag_spec.h                        138
cpp/frontend_match.cc                           124
cpp/frontend_match.h                             67
cpp/frontend_tdop.cc                             16
cpp/frontend_tdop.h                              49
cpp/libc.cc                                     149
cpp/libc.h                                       48
cpp/osh_arith_parse.cc                            9
cpp/osh_arith_parse.h                            22
cpp/osh_bool_stat.cc                             24
cpp/osh_bool_stat.h                              68
cpp/osh_eval_stubs.h                             81
cpp/osh_sh_expr_eval.h                           24
cpp/pgen2_parse.cc                               15
cpp/pgen2_parse.h                                53
cpp/posix.cc                                     26
cpp/posix.h                                     144
cpp/preamble.h                                   87
cpp/pylib_os_path.cc                             35
cpp/pylib_os_path.h                              18
cpp/pylib_path_stat.cc                            9
cpp/pylib_path_stat.h                            20
cpp/qsn_qsn.h                                    53
cpp/signal_.cc                                   14
cpp/signal_.h                                    14
cpp/time_.h                                      30

2,370 lines in 38 files  

Old mycpp Runtime

This implementation has no garbage collection; it allocates memory forever.

mycpp/mylib.cc                                  542
mycpp/mylib.h                                 1,312

1,854 lines in 3 files  

New Garbage-Collected Runtime

Uses a simple Cheney / semi-space collector.

mycpp/gc_heap.cc                                303
mycpp/gc_heap.h                               1,366
mycpp/mylib2.cc                                 216
mycpp/mylib2.h                                  232
mycpp/my_runtime.cc                             526
mycpp/my_runtime.h                              425

3,068 lines in 7 files  

Unit tests in C++

The goal is to make the spec tests pass, but unit tests are helpful too.

cpp/unit_tests.cc                               486
mycpp/gc_heap_test.cc                           931
mycpp/gc_stress_test.cc                         202
mycpp/mylib2_test.cc                            175
mycpp/mylib_test.cc                             505
mycpp/my_runtime_test.cc                        807

3,106 lines in 7 files  

OSH (and common libraries)

This is the input to the translators, written in statically-typed Python. Note that bash is at least 140K lines of code, and OSH implements a large part of bash and more.

bin/oil.py                                      360
core/alloc.py                                   133
core/ansi.py                                     17
core/completion.py                            1,261
core/comp_ui.py                                 585
core/dev.py                                     597
core/error.py                                   198
core/executor.py                                545
core/main_loop.py                               361
core/optview.py                                  57
core/oven.py                                    574
core/process.py                               1,498
core/pyerror.py                                  58
core/pyos.py                                    384
core/pyutil.py                                  247
core/runtime.asdl                               193
core/shell_native.py                            492
core/shell.py                                   695
core/state.py                                 2,179
core/ui.py                                      381
core/util.py                                     93
core/vm.py                                      209
frontend/args.py                                673
frontend/builtin_def.py                         156
frontend/consts.py                              321
frontend/flag_def.py                            322
frontend/flag_spec.py                           391
frontend/id_kind_def.py                         638
frontend/lexer_def.py                         1,012
frontend/lexer.py                               330
frontend/location.py                             99
frontend/match.py                               260
frontend/option_def.py                          393
frontend/parse_lib.py                           458
frontend/py_reader.py                            81
frontend/reader.py                              136
frontend/signal_def.py                           43
frontend/syntax_abbrev.py                       144
frontend/syntax.asdl                            576
frontend/typed_args.py                           70
frontend/types.asdl                              40
mycpp/mylib.pyi                                  72
native/fanos.c                                  294
native/fastlex.c                                239
native/fastlex.pyi                               13
native/libc.c                                   402
native/libc.pyi                                  11
native/posix_.pyi                               211
osh/arith_parse.py                              184
osh/bool_parse.py                               269
osh/bool_stat.py                                136
osh/braces.py                                   498
osh/builtin_assign.py                           514
osh/builtin_bracket.py                          280
osh/builtin_comp.py                             518
osh/builtin_lib.py                               94
osh/builtin_meta.py                             435
osh/builtin_misc.py                             853
osh/builtin_printf.py                           477
osh/builtin_process.py                          555
osh/builtin_pure.py                             673
osh/cmd_eval.py                               1,771
osh/cmd_parse.py                              2,311
osh/glob_.py                                    498
osh/history.py                                  184
osh/prompt.py                                   300
osh/sh_expr_eval.py                           1,047
osh/split.py                                    305
osh/string_ops.py                               454
osh/tdop.py                                     333
osh/word_compile.py                             257
osh/word_eval.py                              2,297
osh/word_parse.py                             1,838
osh/word_.py                                    817
qsn_/qsn_native.py                               75
qsn_/qsn.py                                     615

37,090 lines in 77 files  

Generated C+ Code

mycpp generates the big file _build/cpp/osh_eval.cc. Other programs like Zephyr ASDL and re2c generate other files.

_build/cpp/arg_types.cc                         446
_build/cpp/arg_types.h                          388
_build/cpp/arith_parse.cc                       129
_build/cpp/consts.cc                            899
_build/cpp/consts.h                              43
_build/cpp/core_optview.h                       107
_build/cpp/hnode_asdl.cc                         50
_build/cpp/hnode_asdl.h                         137
_build/cpp/id_kind_asdl.cc                      856
_build/cpp/id_kind_asdl.h                       441
_build/cpp/option_asdl.cc                       348
_build/cpp/option_asdl.h                        189
_build/cpp/osh_eval.cc                       32,994
_build/cpp/osh_eval.h                           337
_build/cpp/runtime_asdl.cc                    2,321
_build/cpp/runtime_asdl.h                     1,176
_build/cpp/syntax_asdl.cc                     9,999
_build/cpp/syntax_asdl.h                      4,221
_build/cpp/types_asdl.cc                         96
_build/cpp/types_asdl.h                          65
_devbuild/gen/find_nt.h                          13
_devbuild/gen/grammar_nt.h                       93
_devbuild/gen/id.h                              382
_devbuild/gen/osh-lex.h                      29,037
_devbuild/gen/osh-types.h                        32

84,799 lines in 26 files  

Spec Tests

A comprehensive test suite that compares OSH against other shells. If OSH passes these tests in BOTH Python and C++, it means that the translation works.

spec/alias.test.sh                              536
spec/append.test.sh                             291
spec/arith-context.test.sh                       81
spec/arith.test.sh                              631
spec/array-compat.test.sh                        96
spec/array.test.sh                              637
spec/assign-deferred.test.sh                    105
spec/assign-dialects.test.sh                     59
spec/assign-extended.test.sh                    810
spec/assign.test.sh                             672
spec/assoc.test.sh                              631
spec/assoc-zsh.test.sh                           53
spec/background.test.sh                         182
spec/ble-features.test.sh                       644
spec/ble-idioms.test.sh                         168
spec/blog1.test.sh                               95
spec/blog2.test.sh                               49
spec/blog-other1.test.sh                         55
spec/brace-expansion.test.sh                    441
spec/bugs.test.sh                               155
spec/builtin-bash.test.sh                       346
spec/builtin-bracket.test.sh                    564
spec/builtin-completion.test.sh                 415
spec/builtin-dirs.test.sh                       205
spec/builtin-eval-source.test.sh                249
spec/builtin-getopts.test.sh                    318
spec/builtin-io.test.sh                         751
spec/builtin-printf.test.sh                     831
spec/builtins2.test.sh                          233
spec/builtin-special.test.sh                    111
spec/builtins.test.sh                           420
spec/builtin-times.test.sh                       15
spec/builtin-trap.test.sh                       410
spec/builtin-vars.test.sh                       710
spec/case_.test.sh                              165
spec/command-parsing.test.sh                     63
spec/command-sub.test.sh                        272
spec/command_.test.sh                           192
spec/comments.test.sh                            11
spec/dbracket.test.sh                           436
spec/dparen.test.sh                             202
spec/empty-bodies.test.sh                        24
spec/errexit-oil.test.sh                        755
spec/errexit.test.sh                            393
spec/exit-status.test.sh                        238
spec/explore-parsing.test.sh                     42
spec/extglob-files.test.sh                      391
spec/extglob-match.test.sh                      400
spec/fatal-errors.test.sh                       186
spec/for-expr.test.sh                            92
spec/func-parsing.test.sh                       105
spec/glob.test.sh                               396
spec/here-doc.test.sh                           365
spec/if_.test.sh                                 56
spec/interactive.test.sh                        216
spec/introspect.test.sh                         237
spec/let.test.sh                                 23
spec/loop.test.sh                               397
spec/nameref.test.sh                            642
spec/nix-idioms.test.sh                         151
spec/nocasematch-match.test.sh                   69
spec/oil-array.test.sh                           86
spec/oil-assign.test.sh                         374
spec/oil-bin.test.sh                             17
spec/oil-blocks.test.sh                         330
spec/oil-bugs.test.sh                            22
spec/oil-builtin-error.test.sh                  423
spec/oil-builtin-pp.test.sh                      46
spec/oil-builtin-process.test.sh                 85
spec/oil-builtins.test.sh                       584
spec/oil-command-sub.test.sh                    108
spec/oil-config.test.sh                         188
spec/oil-demo.test.sh                            59
spec/oil-expr-sub.test.sh                        40
spec/oil-expr.test.sh                           896
spec/oil-funcs-builtin.test.sh                  229
spec/oil-interactive.test.sh                     15
spec/oil-json.test.sh                           119
spec/oil-keywords.test.sh                        37
spec/oil-multiline.test.sh                      179
spec/oil-options.test.sh                        795
spec/oil-proc.test.sh                           252
spec/oil-regex.test.sh                          689
spec/oil-scope.test.sh                          748
spec/oil-slice-range.test.sh                    195
spec/oil-special-vars.test.sh                    48
spec/oil-string.test.sh                         344
spec/oil-tuple.test.sh                           44
spec/oil-user-feedback.test.sh                  209
spec/oil-var-sub.test.sh                         45
spec/oil-with-sh.test.sh                         40
spec/oil-word-eval.test.sh                       81
spec/oil-xtrace.test.sh                         477
spec/osh-only.test.sh                            78
spec/parse-errors.test.sh                       192
spec/pipeline.test.sh                           166
spec/posix.test.sh                              146
spec/process-sub.test.sh                        205
spec/prompt.test.sh                             254
spec/quote.test.sh                              248
spec/redirect.test.sh                           654
spec/regex.test.sh                              344
spec/serialize.test.sh                          229
spec/shell-grammar.test.sh                      205
spec/sh-func.test.sh                            150
spec/sh-options.test.sh                         694
spec/sh-usage.test.sh                            90
spec/smoke.test.sh                              123
spec/strict-options.test.sh                     249
spec/subshell.test.sh                            23
spec/tea-func.test.sh                           288
spec/tilde.test.sh                              172
spec/toysh-posix.test.sh                        347
spec/toysh.test.sh                              144
spec/type-compat.test.sh                         80
spec/var-num.test.sh                             39
spec/var-op-bash.test.sh                        309
spec/var-op-len.test.sh                         219
spec/var-op-patsub.test.sh                      291
spec/var-op-slice.test.sh                       315
spec/var-op-strip.test.sh                       357
spec/var-op-test.test.sh                        339
spec/var-ref.test.sh                            485
spec/vars-bash.test.sh                           27
spec/vars-special.test.sh                       456
spec/var-sub-quote.test.sh                      312
spec/var-sub.test.sh                             74
spec/word-eval.test.sh                           62
spec/word-split.test.sh                         387
spec/xtrace.test.sh                             344

35,389 lines in 131 files  

Gold Tests

Another suite that tests shells "from the outside". Instead of making explicit assertions, we verify that OSH behaves like bash.

test/gold/abuild.sh                               9
test/gold/and-or.sh                              27
test/gold/big-here-doc.sh                     1,531
test/gold/case-in-subshell.sh                    37
test/gold/char-class.sh                          33
test/gold/command-sub-2.sh                       12
test/gold/command-sub.sh                         39
test/gold/comments.sh                            22
test/gold/complex-here-docs.sh                  192
test/gold/configure-bug.sh                       13
test/gold/de1.sh                                 24
test/gold/declare.sh                             28
test/gold/dollar-sq.sh                           42
test/gold/echo-e.sh                              36
test/gold/errexit-confusion.sh                  139
test/gold/errexit.sh                            121
test/gold/export.sh                              65
test/gold/glob.sh                                26
test/gold/lineno.sh                              23
test/gold/nix.sh                                 32
test/gold/readlink.sh                            61
test/gold/readonly.sh                            24
test/gold/scope.sh                              126
test/gold/strip-op-char-class.sh                  6
test/gold/word-eval.sh                           57

2,725 lines in 26 files