| category | total lines | num files |
| mycpp Translator | 4,486 | 11 |
| mycpp Test Data | 2,406 | 28 |
| Zephyr ASDL | 3,011 | 11 |
| pgen2 Parser Generator | 1,717 | 8 |
| Other Code Generators | 2,169 | 9 |
| Hand-written C++ Code | 2,306 | 25 |
| Garbage-Collected Runtime | 4,337 | 25 |
| Unit tests in C++ | 4,870 | 15 |
| Incremental C++ Build | 2,702 | 17 |
| OSH (and common libraries) | 37,649 | 82 |
| Spec Tests | 37,717 | 146 |
| Gold Tests | 2,725 | 26 |
| Generated C++ Code | 80,593 | 24 |
The README for oilshell/oil has another overview of the repository.
This prototype uses the MyPy frontend to translate statically-typed Python to C++. The generated code calls a small runtime which implements things like List[T], Dict[K, V], and Python's len().
mycpp/compare_pairs.py 58 mycpp/const_pass.py 524 mycpp/cppgen_pass.py 2,717 mycpp/crash.py 35 mycpp/debug_pass.py 459 mycpp/format_strings.py 110 mycpp/mycpp_main.py 334 mycpp/mylib.py 160 mycpp/pass_state.py 75 mycpp/util.py 14 4,486 lines in 11 files
Small Python examples that translate to C++, compile, and run.
mycpp/examples/asdl_generated.py 62 mycpp/examples/cartesian.py 75 mycpp/examples/cgi.py 59 mycpp/examples/classes.py 283 mycpp/examples/containers.py 170 mycpp/examples/control_flow.py 110 mycpp/examples/escape.py 67 mycpp/examples/fib_iter.py 54 mycpp/examples/fib_recursive.py 49 mycpp/examples/files.py 62 mycpp/examples/__init__.py 0 mycpp/examples/invalid_types_2.py 32 mycpp/examples/invalid_types.py 32 mycpp/examples/length.py 106 mycpp/examples/lexer_main.py 47 mycpp/examples/loops.py 177 mycpp/examples/modules.py 80 mycpp/examples/parse.py 232 mycpp/examples/pgen2_demo.py 106 mycpp/examples/scoped_resource.py 176 mycpp/examples/strings.py 61 mycpp/examples/test_cast.py 59 mycpp/examples/test_conditional.py 67 mycpp/examples/test_hoist.py 61 mycpp/examples/test_switch.py 42 mycpp/examples/tuple_return_value.py 53 mycpp/examples/varargs.py 84 2,406 lines in 28 files
A DSL for algebraic data types, borrowed from Python. Oil is the most strongly typed Bourne shell implementation!
asdl/asdl_main.py 287 asdl/ast.py 253 asdl/format.py 492 asdl/front_end.py 497 asdl/gen_cpp.py 665 asdl/gen_python.py 528 asdl/NINJA_subgraph.py 54 asdl/pybase.py 63 asdl/runtime.py 41 asdl/visitor.py 131 3,011 lines in 11 files
An LL(1) parser generator used to parse Oil expressions. Also borrowed from CPython.
pgen2/driver.py 101 pgen2/grammar.py 222 pgen2/parse.py 209 pgen2/pgen.py 428 pgen2/pnode.py 29 pgen2/tokenize.py 579 pgen2/token.py 149 1,717 lines in 8 files
In order to make Oil statically typed, we had to abandon Python reflection and use C++ source code generation instead. The lexer, flag definitions, and constants can be easily compiled to C++.
core/optview_gen.py 89 frontend/consts_gen.py 564 frontend/flag_gen.py 476 frontend/lexer_gen.py 490 frontend/option_gen.py 101 frontend/signal_gen.py 110 oil_lang/grammar_gen.py 239 osh/arith_parse_gen.py 100 2,169 lines in 9 files
Includes OS bindings. Small C++ files like cpp/osh_arith_parse.{cc,h} correspond to larger Python files like osh/arith_parse.py.
cpp/leaky_core.cc 307 cpp/leaky_core_error.h 173 cpp/leaky_core.h 107 cpp/leaky_core_pyerror.h 90 cpp/leaky_frontend_flag_spec.cc 292 cpp/leaky_frontend_flag_spec.h 128 cpp/leaky_frontend_match.cc 100 cpp/leaky_frontend_match.h 66 cpp/leaky_frontend_tdop.cc 16 cpp/leaky_frontend_tdop.h 47 cpp/leaky_libc.cc 164 cpp/leaky_libc.h 34 cpp/leaky_osh.cc 155 cpp/leaky_osh_eval_stubs.h 47 cpp/leaky_osh.h 50 cpp/leaky_pgen2.cc 15 cpp/leaky_pgen2.h 53 cpp/leaky_pylib.cc 50 cpp/leaky_pylib.h 20 cpp/leaky_stdlib.cc 164 cpp/leaky_stdlib.h 117 cpp/preamble.h 30 cpp/qsn.h 54 cpp/segfault_handler.h 27 2,306 lines in 25 files
Uses a simple Cheney / semi-space collector.
mycpp/builtins.h 134 mycpp/bump_leak_heap.cc 62 mycpp/bump_leak_heap.h 71 mycpp/cheney_heap.cc 273 mycpp/cheney_heap.h 239 mycpp/common.h 62 mycpp/comparators.h 34 mycpp/gc_dict.h 437 mycpp/gc_heap.h 79 mycpp/gc_list.h 544 mycpp/gc_mylib.cc 8 mycpp/gc_mylib.h 16 mycpp/gc_obj.h 120 mycpp/gc_slab.h 62 mycpp/gc_str.h 201 mycpp/gc_tuple.h 102 mycpp/leaky_builtins.cc 253 mycpp/leaky_containers.cc 481 mycpp/leaky_mylib.cc 343 mycpp/leaky_mylib.h 238 mycpp/marksweep_heap.cc 257 mycpp/marksweep_heap.h 184 mycpp/runtime.h 19 mycpp/smartptr.h 118 4,337 lines in 25 files
The goal is to make the spec tests pass, but unit tests are helpful too.
cpp/data_race_test.cc 85 cpp/gc_binding_test.cc 43 cpp/leaky_binding_test.cc 371 cpp/leaky_core_test.cc 27 cpp/leaky_flag_spec_test.cc 138 mycpp/bump_leak_heap_test.cc 37 mycpp/gc_builtins_test.cc 842 mycpp/gc_heap_test.cc 841 mycpp/gc_mylib_test.cc 246 mycpp/gc_stress_test.cc 220 mycpp/leaky_containers_test.cc 474 mycpp/leaky_str_test.cc 917 mycpp/marksweep_heap_test.cc 486 mycpp/smartptr_test.cc 143 4,870 lines in 15 files
asdl/NINJA_subgraph.py 54 bin/NINJA_subgraph.py 61 build/ninja_lib.py 458 build/ninja_lib_test.py 253 build/ninja_main.py 337 build/ninja-rules-cpp.sh 289 build/ninja-rules-py.sh 359 core/NINJA_subgraph.py 48 cpp/NINJA_subgraph.py 93 frontend/NINJA_subgraph.py 135 mycpp/NINJA_subgraph.py 430 NINJA-config.sh 78 oil_lang/NINJA_subgraph.py 35 osh/NINJA_subgraph.py 30 pea/NINJA_subgraph.py 20 prebuilt/NINJA_subgraph.py 22 2,702 lines in 17 files
This is the input to the translators, written in statically-typed Python. Note that bash is at least 140K lines of code, and OSH implements a large part of bash and more.
bin/oil.py 366 core/alloc.py 166 core/ansi.py 17 core/completion.py 1,263 core/comp_ui.py 585 core/dev.py 598 core/error.py 198 core/executor.py 567 core/main_loop.py 361 core/NINJA_subgraph.py 48 core/optview.py 57 core/process.py 1,530 core/pyerror.py 71 core/pyos.py 355 core/pyutil.py 247 core/runtime.asdl 194 core/shell_native.py 511 core/shell.py 671 core/state.py 2,387 core/ui.py 396 core/util.py 93 core/vm.py 213 frontend/args.py 673 frontend/builtin_def.py 164 frontend/consts.py 342 frontend/flag_def.py 323 frontend/flag_spec.py 391 frontend/id_kind_def.py 638 frontend/lexer_def.py 1,026 frontend/lexer.py 330 frontend/location.py 99 frontend/match.py 265 frontend/NINJA_subgraph.py 135 frontend/option_def.py 382 frontend/parse_lib.py 477 frontend/py_reader.py 81 frontend/reader.py 143 frontend/signal_def.py 46 frontend/syntax_abbrev.py 144 frontend/syntax.asdl 594 frontend/typed_args.py 70 frontend/types.asdl 41 mycpp/mylib.pyi 74 osh/arith_parse.py 184 osh/bool_parse.py 269 osh/bool_stat.py 136 osh/braces.py 498 osh/builtin_assign.py 514 osh/builtin_bracket.py 280 osh/builtin_comp.py 518 osh/builtin_lib.py 94 osh/builtin_meta.py 435 osh/builtin_misc.py 875 osh/builtin_printf.py 478 osh/builtin_process2.py 222 osh/builtin_process.py 131 osh/builtin_pure.py 960 osh/builtin_trap.py 286 osh/cmd_eval.py 1,858 osh/cmd_parse.py 2,415 osh/glob_.py 488 osh/history.py 184 osh/NINJA_subgraph.py 30 osh/prompt.py 300 osh/sh_expr_eval.py 1,048 osh/split.py 307 osh/string_ops.py 454 osh/tdop.py 333 osh/word_compile.py 263 osh/word_eval.py 2,253 osh/word_parse.py 1,838 osh/word_.py 821 pyext/fanos.c 294 pyext/fastlex.c 246 pyext/fastlex.pyi 15 pyext/libc.c 402 pyext/libc.pyi 11 pyext/posix_.pyi 211 pyext/yajl.pyi 3 qsn_/qsn_native.py 75 qsn_/qsn.py 588 37,649 lines in 82 files
A comprehensive test suite that compares OSH against other shells. If OSH passes these tests in BOTH Python and C++, it means that the translation works.
spec/alias.test.sh 536 spec/append.test.sh 291 spec/arith-context.test.sh 81 spec/arith.test.sh 631 spec/array-compat.test.sh 96 spec/array.test.sh 637 spec/assign-deferred.test.sh 105 spec/assign-dialects.test.sh 59 spec/assign-extended.test.sh 810 spec/assign.test.sh 672 spec/assoc.test.sh 631 spec/assoc-zsh.test.sh 53 spec/background.test.sh 182 spec/ble-features.test.sh 644 spec/ble-idioms.test.sh 168 spec/blog1.test.sh 95 spec/blog2.test.sh 49 spec/blog-other1.test.sh 55 spec/brace-expansion.test.sh 441 spec/bugs.test.sh 169 spec/builtin-bash.test.sh 346 spec/builtin-bracket.test.sh 564 spec/builtin-completion.test.sh 412 spec/builtin-dirs.test.sh 205 spec/builtin-eval-source.test.sh 249 spec/builtin-getopts.test.sh 318 spec/builtin-io.test.sh 751 spec/builtin-printf.test.sh 900 spec/builtins2.test.sh 233 spec/builtin-special.test.sh 111 spec/builtins.test.sh 420 spec/builtin-times.test.sh 15 spec/builtin-trap.test.sh 410 spec/builtin-vars.test.sh 710 spec/case_.test.sh 165 spec/command-parsing.test.sh 63 spec/command-sub.test.sh 272 spec/command_.test.sh 192 spec/comments.test.sh 11 spec/dbracket.test.sh 436 spec/dparen.test.sh 202 spec/empty-bodies.test.sh 24 spec/errexit-oil.test.sh 755 spec/errexit.test.sh 393 spec/exit-status.test.sh 238 spec/explore-parsing.test.sh 42 spec/extglob-files.test.sh 391 spec/extglob-match.test.sh 400 spec/fatal-errors.test.sh 186 spec/for-expr.test.sh 92 spec/func-parsing.test.sh 105 spec/glob.test.sh 396 spec/hay-isolation.test.sh 169 spec/hay-meta.test.sh 200 spec/hay.test.sh 645 spec/here-doc.test.sh 365 spec/if_.test.sh 56 spec/interactive.test.sh 216 spec/introspect.test.sh 237 spec/let.test.sh 23 spec/loop.test.sh 408 spec/nameref.test.sh 642 spec/nix-idioms.test.sh 151 spec/nocasematch-match.test.sh 69 spec/nul-bytes.test.sh 153 spec/oil-array.test.sh 38 spec/oil-assign.test.sh 359 spec/oil-bin.test.sh 17 spec/oil-blocks.test.sh 377 spec/oil-bugs.test.sh 48 spec/oil-builtin-argparse.test.sh 61 spec/oil-builtin-describe.test.sh 38 spec/oil-builtin-error.test.sh 424 spec/oil-builtin-pp.test.sh 47 spec/oil-builtin-process.test.sh 85 spec/oil-builtin-shopt.test.sh 179 spec/oil-builtins.test.sh 568 spec/oil-case.test.sh 27 spec/oil-command-sub.test.sh 109 spec/oil-demo.test.sh 59 spec/oil-expr-arith.test.sh 48 spec/oil-expr-compare.test.sh 286 spec/oil-expr-sub.test.sh 40 spec/oil-expr.test.sh 738 spec/oil-for.test.sh 173 spec/oil-funcs-builtin.test.sh 200 spec/oil-funcs-external.test.sh 87 spec/oil-interactive.test.sh 15 spec/oil-json.test.sh 121 spec/oil-keywords.test.sh 37 spec/oil-multiline.test.sh 179 spec/oil-options-assign.test.sh 103 spec/oil-options.test.sh 781 spec/oil-proc.test.sh 270 spec/oil-regex.test.sh 960 spec/oil-reserved.test.sh 46 spec/oil-scope.test.sh 764 spec/oil-slice-range.test.sh 162 spec/oil-special-vars.test.sh 48 spec/oil-string.test.sh 344 spec/oil-tuple.test.sh 26 spec/oil-usage.test.sh 22 spec/oil-user-feedback.test.sh 213 spec/oil-var-sub.test.sh 45 spec/oil-with-sh.test.sh 77 spec/oil-word-eval.test.sh 128 spec/oil-xtrace.test.sh 478 spec/osh-only.test.sh 78 spec/parse-errors.test.sh 192 spec/pipeline.test.sh 166 spec/posix.test.sh 153 spec/process-sub.test.sh 205 spec/prompt.test.sh 253 spec/quote.test.sh 248 spec/redirect.test.sh 654 spec/regex.test.sh 344 spec/serialize.test.sh 229 spec/shell-grammar.test.sh 205 spec/sh-func.test.sh 150 spec/sh-options.test.sh 694 spec/sh-usage.test.sh 90 spec/smoke.test.sh 123 spec/strict-options.test.sh 249 spec/subshell.test.sh 23 spec/tea-func.test.sh 288 spec/tilde.test.sh 172 spec/TODO-deprecate.test.sh 14 spec/toysh-posix.test.sh 347 spec/toysh.test.sh 144 spec/type-compat.test.sh 107 spec/var-num.test.sh 39 spec/var-op-bash.test.sh 309 spec/var-op-len.test.sh 219 spec/var-op-patsub.test.sh 291 spec/var-op-slice.test.sh 315 spec/var-op-strip.test.sh 357 spec/var-op-test.test.sh 339 spec/var-ref.test.sh 485 spec/vars-bash.test.sh 27 spec/vars-special.test.sh 456 spec/var-sub-quote.test.sh 312 spec/var-sub.test.sh 74 spec/word-eval.test.sh 62 spec/word-split.test.sh 387 spec/xtrace.test.sh 344 37,717 lines in 146 files
Another suite that tests shells "from the outside". Instead of making explicit assertions, we verify that OSH behaves like bash.
test/gold/abuild.sh 9 test/gold/and-or.sh 27 test/gold/big-here-doc.sh 1,531 test/gold/case-in-subshell.sh 37 test/gold/char-class.sh 33 test/gold/command-sub-2.sh 12 test/gold/command-sub.sh 39 test/gold/comments.sh 22 test/gold/complex-here-docs.sh 192 test/gold/configure-bug.sh 13 test/gold/de1.sh 24 test/gold/declare.sh 28 test/gold/dollar-sq.sh 42 test/gold/echo-e.sh 36 test/gold/errexit-confusion.sh 139 test/gold/errexit.sh 121 test/gold/export.sh 65 test/gold/glob.sh 26 test/gold/lineno.sh 23 test/gold/nix.sh 32 test/gold/readlink.sh 61 test/gold/readonly.sh 24 test/gold/scope.sh 126 test/gold/strip-op-char-class.sh 6 test/gold/word-eval.sh 57 2,725 lines in 26 files
mycpp generates the big file _gen/bin/osh_eval.mycpp.cc. Other programs like Zephyr ASDL and re2c generate other files.
_gen/asdl/hnode.asdl.h 229 _gen/bin/osh_eval.mycpp.cc 35,965 _gen/bin/osh_eval.mycpp.h 5 _gen/core/optview.h 112 _gen/core/runtime.asdl.cc 1,165 _gen/core/runtime.asdl.h 1,770 _gen/frontend/arg_types.cc 410 _gen/frontend/arg_types.h 389 _gen/frontend/consts.cc 1,354 _gen/frontend/consts.h 46 _gen/frontend/id_kind.asdl.cc 856 _gen/frontend/id_kind.asdl_c.h 382 _gen/frontend/id_kind.asdl.h 441 _gen/frontend/match.re2c.h 24,373 _gen/frontend/option.asdl.h 193 _gen/frontend/signal.cc 268 _gen/frontend/signal.h 16 _gen/frontend/syntax.asdl.cc 5,078 _gen/frontend/syntax.asdl.h 7,218 _gen/frontend/types.asdl_c.h 36 _gen/frontend/types.asdl.h 65 _gen/oil_lang/grammar_nt.h 93 _gen/osh/arith_parse.cc 129 80,593 lines in 24 files