Overview: Translating Oil to C++

category total lines num files
mycpp Translator 4,294 10
mycpp Test Data 2,182 25
Zephyr ASDL 2,904 10
pgen2 Parser Generator 1,717 8
Other Code Generators 2,062 8
Hand-written C++ Code 2,290 38
Old mycpp Runtime 1,813 3
New Garbage-Collected Runtime 3,068 7
Unit tests in C++ 3,064 7
OSH (and common libraries) 36,814 77
Generated C+ Code 84,624 26
Spec Tests 34,188 127
Gold Tests 952 20

Related Documents

The README for oilshell/oil has another overview of the repository.

mycpp Translator

This prototype uses the MyPy frontend to translate statically-typed Python to C++. The generated code calls a small runtime which implements things like List[T], Dict[K, V], and Python's len().

mycpp/const_pass.py                             526
mycpp/cppgen_pass.py                          2,602
mycpp/crash.py                                   37
mycpp/debug_pass.py                             459
mycpp/format_strings.py                         110
mycpp/mycpp_main.py                             346
mycpp/mylib.py                                  148
mycpp/pass_state.py                              54
mycpp/util.py                                    12

4,294 lines in 10 files  

mycpp Test Data

Small Python examples that translate to C++, compile, and run.

mycpp/examples/asdl_generated.py                 62
mycpp/examples/cartesian.py                      75
mycpp/examples/cgi.py                            59
mycpp/examples/classes.py                       141
mycpp/examples/containers.py                    162
mycpp/examples/control_flow.py                  110
mycpp/examples/escape.py                         67
mycpp/examples/fib_iter.py                       54
mycpp/examples/fib_recursive.py                  49
mycpp/examples/files.py                          53
mycpp/examples/length.py                        106
mycpp/examples/lexer_main.py                     48
mycpp/examples/loops.py                         177
mycpp/examples/modules.py                        78
mycpp/examples/parse.py                         232
mycpp/examples/pgen2_demo.py                    106
mycpp/examples/scoped_resource.py               176
mycpp/examples/strings.py                        61
mycpp/examples/test_cast.py                      59
mycpp/examples/test_conditional.py               67
mycpp/examples/test_hoist.py                     61
mycpp/examples/test_switch.py                    42
mycpp/examples/tuple_return_value.py             53
mycpp/examples/varargs.py                        84

2,182 lines in 25 files  

Zephyr ASDL

A DSL for algebraic data types, borrowed from Python. Oil is the most strongly typed Bourne shell implementation!

asdl/asdl_.py                                   253
asdl/format.py                                  494
asdl/front_end.py                               488
asdl/gen_cpp.py                                 622
asdl/gen_python.py                              529
asdl/pybase.py                                   63
asdl/runtime.py                                  41
asdl/tool.py                                    280
asdl/visitor.py                                 134

2,904 lines in 10 files  

pgen2 Parser Generator

An LL(1) parser generator used to parse Oil expressions. Also borrowed from CPython.

pgen2/driver.py                                 101
pgen2/grammar.py                                222
pgen2/parse.py                                  209
pgen2/pgen.py                                   428
pgen2/pnode.py                                   29
pgen2/tokenize.py                               579
pgen2/token.py                                  149

1,717 lines in 8 files  

Other Code Generators

In order to make Oil statically typed, we had to abandon Python reflection and use C++ source code generation instead. The lexer, flag definitions, and constants can be easily compiled to C++.

core/optview_gen.py                              89
frontend/consts_gen.py                          551
frontend/flag_gen.py                            485
frontend/lexer_gen.py                           495
frontend/option_gen.py                          115
oil_lang/grammar_gen.py                         226
osh/arith_parse_gen.py                          101

2,062 lines in 8 files  

Hand-written C++ Code

Includes OS bindings. Small C++ files like cpp/osh_arith_parse.{cc,h} correspond to larger Python files like osh/arith_parse.py.

cpp/core_error.h                                157
cpp/core_pyerror.h                               74
cpp/core_pyos.cc                                107
cpp/core_pyos.h                                  79
cpp/core_pyutil.cc                               47
cpp/core_pyutil.h                                49
cpp/dumb_alloc.cc                                88
cpp/dumb_alloc.h                                 17
cpp/fcntl_.cc                                    12
cpp/fcntl_.h                                     30
cpp/frontend_flag_spec.cc                       320
cpp/frontend_flag_spec.h                        133
cpp/frontend_match.cc                           118
cpp/frontend_match.h                             66
cpp/frontend_tdop.cc                             16
cpp/frontend_tdop.h                              49
cpp/libc.cc                                     149
cpp/libc.h                                       48
cpp/osh_arith_parse.cc                            9
cpp/osh_arith_parse.h                            22
cpp/osh_bool_stat.cc                             24
cpp/osh_bool_stat.h                              68
cpp/osh_eval_stubs.h                             81
cpp/osh_sh_expr_eval.h                           24
cpp/pgen2_parse.cc                               15
cpp/pgen2_parse.h                                53
cpp/posix.cc                                     16
cpp/posix.h                                     148
cpp/preamble.h                                   83
cpp/pylib_os_path.cc                             35
cpp/pylib_os_path.h                              18
cpp/pylib_path_stat.cc                            9
cpp/pylib_path_stat.h                            15
cpp/qsn_qsn.h                                    53
cpp/signal_.cc                                   14
cpp/signal_.h                                    14
cpp/time_.h                                      30

2,290 lines in 38 files  

Old mycpp Runtime

This implementation has no garbage collection; it allocates memory forever.

mycpp/mylib.cc                                  535
mycpp/mylib.h                                 1,278

1,813 lines in 3 files  

New Garbage-Collected Runtime

Uses a simple Cheney / semi-space collector.

mycpp/gc_heap.cc                                303
mycpp/gc_heap.h                               1,366
mycpp/mylib2.cc                                 216
mycpp/mylib2.h                                  232
mycpp/my_runtime.cc                             526
mycpp/my_runtime.h                              425

3,068 lines in 7 files  

Unit tests in C++

The goal is to make the spec tests pass, but unit tests are helpful too.

cpp/unit_tests.cc                               444
mycpp/gc_heap_test.cc                           931
mycpp/gc_stress_test.cc                         202
mycpp/mylib2_test.cc                            175
mycpp/mylib_test.cc                             505
mycpp/my_runtime_test.cc                        807

3,064 lines in 7 files  

OSH (and common libraries)

This is the input to the translators, written in statically-typed Python. Note that bash is at least 140K lines of code, and OSH implements a large part of bash and more.

bin/oil.py                                      360
core/alloc.py                                   133
core/ansi.py                                     17
core/completion.py                            1,261
core/comp_ui.py                                 583
core/dev.py                                     597
core/error.py                                   183
core/executor.py                                548
core/main_loop.py                               352
core/optview.py                                  57
core/oven.py                                    573
core/process.py                               1,501
core/pyerror.py                                  58
core/pyos.py                                    296
core/pyutil.py                                  247
core/runtime.asdl                               193
core/shell_native.py                            491
core/shell.py                                   695
core/state.py                                 2,124
core/ui.py                                      381
core/util.py                                     93
core/vm.py                                      209
frontend/args.py                                673
frontend/builtin_def.py                         155
frontend/consts.py                              321
frontend/flag_def.py                            318
frontend/flag_spec.py                           380
frontend/id_kind_def.py                         638
frontend/lexer_def.py                         1,012
frontend/lexer.py                               330
frontend/location.py                             94
frontend/match.py                               257
frontend/option_def.py                          389
frontend/parse_lib.py                           458
frontend/py_reader.py                            81
frontend/reader.py                              136
frontend/signal_def.py                           43
frontend/syntax_abbrev.py                       144
frontend/syntax.asdl                            576
frontend/typed_args.py                           63
frontend/types.asdl                              40
mycpp/mylib.pyi                                  72
native/fanos.c                                  294
native/fastlex.c                                239
native/fastlex.pyi                               13
native/libc.c                                   402
native/libc.pyi                                  11
native/posix_.pyi                               211
osh/arith_parse.py                              184
osh/bool_parse.py                               269
osh/bool_stat.py                                136
osh/braces.py                                   498
osh/builtin_assign.py                           514
osh/builtin_bracket.py                          280
osh/builtin_comp.py                             501
osh/builtin_lib.py                               94
osh/builtin_meta.py                             403
osh/builtin_misc.py                             844
osh/builtin_printf.py                           477
osh/builtin_process.py                          539
osh/builtin_pure.py                             670
osh/cmd_eval.py                               1,772
osh/cmd_parse.py                              2,311
osh/glob_.py                                    498
osh/history.py                                  184
osh/prompt.py                                   300
osh/sh_expr_eval.py                           1,047
osh/split.py                                    305
osh/string_ops.py                               454
osh/tdop.py                                     333
osh/word_compile.py                             257
osh/word_eval.py                              2,297
osh/word_parse.py                             1,838
osh/word_.py                                    817
qsn_/qsn_native.py                               75
qsn_/qsn.py                                     615

36,814 lines in 77 files  

Generated C+ Code

mycpp generates the big file _build/cpp/osh_eval.cc. Other programs like Zephyr ASDL and re2c generate other files.

_build/cpp/arg_types.cc                         441
_build/cpp/arg_types.h                          388
_build/cpp/arith_parse.cc                       129
_build/cpp/consts.cc                            898
_build/cpp/consts.h                              43
_build/cpp/core_optview.h                       107
_build/cpp/hnode_asdl.cc                         50
_build/cpp/hnode_asdl.h                         137
_build/cpp/id_kind_asdl.cc                      856
_build/cpp/id_kind_asdl.h                       441
_build/cpp/option_asdl.cc                       346
_build/cpp/option_asdl.h                        188
_build/cpp/osh_eval.cc                       32,805
_build/cpp/osh_eval.h                           337
_build/cpp/runtime_asdl.cc                    2,321
_build/cpp/runtime_asdl.h                     1,176
_build/cpp/syntax_asdl.cc                     9,999
_build/cpp/syntax_asdl.h                      4,221
_build/cpp/types_asdl.cc                         96
_build/cpp/types_asdl.h                          65
_devbuild/gen/find_nt.h                          13
_devbuild/gen/grammar_nt.h                       93
_devbuild/gen/id.h                              382
_devbuild/gen/osh-lex.h                      29,060
_devbuild/gen/osh-types.h                        32

84,624 lines in 26 files  

Spec Tests

A comprehensive test suite that compares OSH against other shells. If OSH passes these tests in BOTH Python and C++, it means that the translation works.

spec/alias.test.sh                              536
spec/append.test.sh                             291
spec/arith-context.test.sh                       81
spec/arith.test.sh                              631
spec/array-compat.test.sh                        96
spec/array.test.sh                              637
spec/assign-deferred.test.sh                    105
spec/assign-dialects.test.sh                     59
spec/assign-extended.test.sh                    810
spec/assign.test.sh                             671
spec/assoc.test.sh                              631
spec/assoc-zsh.test.sh                           53
spec/background.test.sh                         150
spec/ble-features.test.sh                       631
spec/ble-idioms.test.sh                         159
spec/blog1.test.sh                               95
spec/blog2.test.sh                               49
spec/blog-other1.test.sh                         55
spec/brace-expansion.test.sh                    441
spec/bugs.test.sh                               103
spec/builtin-bash.test.sh                       346
spec/builtin-bracket.test.sh                    564
spec/builtin-completion.test.sh                 399
spec/builtin-dirs.test.sh                       177
spec/builtin-eval-source.test.sh                249
spec/builtin-getopts.test.sh                    318
spec/builtin-io.test.sh                         761
spec/builtin-printf.test.sh                     831
spec/builtins2.test.sh                          233
spec/builtin-special.test.sh                    111
spec/builtins.test.sh                           420
spec/builtin-times.test.sh                       15
spec/builtin-trap.test.sh                       406
spec/builtin-vars.test.sh                       710
spec/case_.test.sh                              165
spec/command-parsing.test.sh                     63
spec/command-sub.test.sh                        272
spec/command_.test.sh                           192
spec/comments.test.sh                            11
spec/dbracket.test.sh                           436
spec/dparen.test.sh                             202
spec/empty-bodies.test.sh                        24
spec/errexit-oil.test.sh                        675
spec/errexit.test.sh                            392
spec/exit-status.test.sh                        238
spec/explore-parsing.test.sh                     42
spec/extglob-files.test.sh                      391
spec/extglob-match.test.sh                      400
spec/for-expr.test.sh                            92
spec/func-parsing.test.sh                       105
spec/glob.test.sh                               396
spec/here-doc.test.sh                           365
spec/if_.test.sh                                 56
spec/interactive.test.sh                        216
spec/introspect.test.sh                         237
spec/let.test.sh                                 23
spec/loop.test.sh                               397
spec/nameref.test.sh                            642
spec/nix-idioms.test.sh                         151
spec/nocasematch-match.test.sh                   69
spec/oil-array.test.sh                           86
spec/oil-assign.test.sh                         326
spec/oil-bin.test.sh                             17
spec/oil-blocks.test.sh                         163
spec/oil-builtin-pp.test.sh                      46
spec/oil-builtin-process.test.sh                 85
spec/oil-builtins.test.sh                       554
spec/oil-builtin-try.test.sh                    140
spec/oil-command-sub.test.sh                    108
spec/oil-config.test.sh                         100
spec/oil-demo.test.sh                            59
spec/oil-expr-sub.test.sh                        40
spec/oil-expr.test.sh                           866
spec/oil-funcs-builtin.test.sh                  229
spec/oil-interactive.test.sh                     15
spec/oil-json.test.sh                           119
spec/oil-keywords.test.sh                        37
spec/oil-multiline.test.sh                      179
spec/oil-options.test.sh                        795
spec/oil-proc.test.sh                           252
spec/oil-regex.test.sh                          689
spec/oil-scope.test.sh                          748
spec/oil-slice-range.test.sh                    195
spec/oil-special-vars.test.sh                    66
spec/oil-string.test.sh                         344
spec/oil-tuple.test.sh                           44
spec/oil-user-feedback.test.sh                  209
spec/oil-var-sub.test.sh                         45
spec/oil-word-eval.test.sh                       81
spec/oil-xtrace.test.sh                         477
spec/osh-only.test.sh                            78
spec/parse-errors.test.sh                       192
spec/pipeline.test.sh                           166
spec/posix.test.sh                              146
spec/process-sub.test.sh                        205
spec/prompt.test.sh                             254
spec/quote.test.sh                              248
spec/redirect.test.sh                           654
spec/regex.test.sh                              344
spec/serialize.test.sh                          229
spec/shell-grammar.test.sh                      205
spec/sh-func.test.sh                            150
spec/sh-options.test.sh                         694
spec/sh-usage.test.sh                            94
spec/smoke.test.sh                              123
spec/special-vars.test.sh                       425
spec/strict-options.test.sh                     249
spec/subshell.test.sh                            23
spec/tea-func.test.sh                           288
spec/tilde.test.sh                              172
spec/toysh-posix.test.sh                        347
spec/toysh.test.sh                              144
spec/type-compat.test.sh                         80
spec/var-num.test.sh                             39
spec/var-op-bash.test.sh                        264
spec/var-op-len.test.sh                         219
spec/var-op-patsub.test.sh                      291
spec/var-op-slice.test.sh                       315
spec/var-op-strip.test.sh                       357
spec/var-op-test.test.sh                        339
spec/var-ref.test.sh                            485
spec/var-sub-quote.test.sh                      312
spec/var-sub.test.sh                             74
spec/word-eval.test.sh                           62
spec/word-split.test.sh                         387
spec/xtrace.test.sh                             344

34,188 lines in 127 files  

Gold Tests

Another suite that tests shells "from the outside". Instead of making explicit assertions, we verify that OSH behaves like bash.

test/gold/abuild.sh                               9
test/gold/and-or.sh                              27
test/gold/comments.sh                            22
test/gold/complex-here-docs.sh                  192
test/gold/configure-bug.sh                       13
test/gold/de1.sh                                 24
test/gold/declare.sh                             28
test/gold/dollar-sq.sh                           42
test/gold/echo-e.sh                              36
test/gold/errexit-confusion.sh                  139
test/gold/export.sh                              65
test/gold/glob.sh                                26
test/gold/lineno.sh                              23
test/gold/nix.sh                                 32
test/gold/readlink.sh                            61
test/gold/readonly.sh                            24
test/gold/scope.sh                              126
test/gold/strip-op-char-class.sh                  6
test/gold/word-eval.sh                           57

952 lines in 20 files