category | total lines | num files |
mycpp Translator | 4,294 | 10 |
mycpp Test Data | 2,182 | 25 |
Zephyr ASDL | 2,904 | 10 |
pgen2 Parser Generator | 1,717 | 8 |
Other Code Generators | 2,062 | 8 |
Hand-written C++ Code | 2,290 | 38 |
Old mycpp Runtime | 1,813 | 3 |
New Garbage-Collected Runtime | 3,068 | 7 |
Unit tests in C++ | 3,064 | 7 |
OSH (and common libraries) | 36,814 | 77 |
Generated C+ Code | 84,624 | 26 |
Spec Tests | 34,188 | 127 |
Gold Tests | 952 | 20 |
The README for oilshell/oil has another overview of the repository.
This prototype uses the MyPy frontend to translate statically-typed Python to C++. The generated code calls a small runtime which implements things like List[T], Dict[K, V], and Python's len().
mycpp/const_pass.py 526 mycpp/cppgen_pass.py 2,602 mycpp/crash.py 37 mycpp/debug_pass.py 459 mycpp/format_strings.py 110 mycpp/mycpp_main.py 346 mycpp/mylib.py 148 mycpp/pass_state.py 54 mycpp/util.py 12 4,294 lines in 10 files
Small Python examples that translate to C++, compile, and run.
mycpp/examples/asdl_generated.py 62 mycpp/examples/cartesian.py 75 mycpp/examples/cgi.py 59 mycpp/examples/classes.py 141 mycpp/examples/containers.py 162 mycpp/examples/control_flow.py 110 mycpp/examples/escape.py 67 mycpp/examples/fib_iter.py 54 mycpp/examples/fib_recursive.py 49 mycpp/examples/files.py 53 mycpp/examples/length.py 106 mycpp/examples/lexer_main.py 48 mycpp/examples/loops.py 177 mycpp/examples/modules.py 78 mycpp/examples/parse.py 232 mycpp/examples/pgen2_demo.py 106 mycpp/examples/scoped_resource.py 176 mycpp/examples/strings.py 61 mycpp/examples/test_cast.py 59 mycpp/examples/test_conditional.py 67 mycpp/examples/test_hoist.py 61 mycpp/examples/test_switch.py 42 mycpp/examples/tuple_return_value.py 53 mycpp/examples/varargs.py 84 2,182 lines in 25 files
A DSL for algebraic data types, borrowed from Python. Oil is the most strongly typed Bourne shell implementation!
asdl/asdl_.py 253 asdl/format.py 494 asdl/front_end.py 488 asdl/gen_cpp.py 622 asdl/gen_python.py 529 asdl/pybase.py 63 asdl/runtime.py 41 asdl/tool.py 280 asdl/visitor.py 134 2,904 lines in 10 files
An LL(1) parser generator used to parse Oil expressions. Also borrowed from CPython.
pgen2/driver.py 101 pgen2/grammar.py 222 pgen2/parse.py 209 pgen2/pgen.py 428 pgen2/pnode.py 29 pgen2/tokenize.py 579 pgen2/token.py 149 1,717 lines in 8 files
In order to make Oil statically typed, we had to abandon Python reflection and use C++ source code generation instead. The lexer, flag definitions, and constants can be easily compiled to C++.
core/optview_gen.py 89 frontend/consts_gen.py 551 frontend/flag_gen.py 485 frontend/lexer_gen.py 495 frontend/option_gen.py 115 oil_lang/grammar_gen.py 226 osh/arith_parse_gen.py 101 2,062 lines in 8 files
Includes OS bindings. Small C++ files like cpp/osh_arith_parse.{cc,h} correspond to larger Python files like osh/arith_parse.py.
cpp/core_error.h 157 cpp/core_pyerror.h 74 cpp/core_pyos.cc 107 cpp/core_pyos.h 79 cpp/core_pyutil.cc 47 cpp/core_pyutil.h 49 cpp/dumb_alloc.cc 88 cpp/dumb_alloc.h 17 cpp/fcntl_.cc 12 cpp/fcntl_.h 30 cpp/frontend_flag_spec.cc 320 cpp/frontend_flag_spec.h 133 cpp/frontend_match.cc 118 cpp/frontend_match.h 66 cpp/frontend_tdop.cc 16 cpp/frontend_tdop.h 49 cpp/libc.cc 149 cpp/libc.h 48 cpp/osh_arith_parse.cc 9 cpp/osh_arith_parse.h 22 cpp/osh_bool_stat.cc 24 cpp/osh_bool_stat.h 68 cpp/osh_eval_stubs.h 81 cpp/osh_sh_expr_eval.h 24 cpp/pgen2_parse.cc 15 cpp/pgen2_parse.h 53 cpp/posix.cc 16 cpp/posix.h 148 cpp/preamble.h 83 cpp/pylib_os_path.cc 35 cpp/pylib_os_path.h 18 cpp/pylib_path_stat.cc 9 cpp/pylib_path_stat.h 15 cpp/qsn_qsn.h 53 cpp/signal_.cc 14 cpp/signal_.h 14 cpp/time_.h 30 2,290 lines in 38 files
This implementation has no garbage collection; it allocates memory forever.
mycpp/mylib.cc 535 mycpp/mylib.h 1,278 1,813 lines in 3 files
Uses a simple Cheney / semi-space collector.
mycpp/gc_heap.cc 303 mycpp/gc_heap.h 1,366 mycpp/mylib2.cc 216 mycpp/mylib2.h 232 mycpp/my_runtime.cc 526 mycpp/my_runtime.h 425 3,068 lines in 7 files
The goal is to make the spec tests pass, but unit tests are helpful too.
cpp/unit_tests.cc 444 mycpp/gc_heap_test.cc 931 mycpp/gc_stress_test.cc 202 mycpp/mylib2_test.cc 175 mycpp/mylib_test.cc 505 mycpp/my_runtime_test.cc 807 3,064 lines in 7 files
This is the input to the translators, written in statically-typed Python. Note that bash is at least 140K lines of code, and OSH implements a large part of bash and more.
bin/oil.py 360 core/alloc.py 133 core/ansi.py 17 core/completion.py 1,261 core/comp_ui.py 583 core/dev.py 597 core/error.py 183 core/executor.py 548 core/main_loop.py 352 core/optview.py 57 core/oven.py 573 core/process.py 1,501 core/pyerror.py 58 core/pyos.py 296 core/pyutil.py 247 core/runtime.asdl 193 core/shell_native.py 491 core/shell.py 695 core/state.py 2,124 core/ui.py 381 core/util.py 93 core/vm.py 209 frontend/args.py 673 frontend/builtin_def.py 155 frontend/consts.py 321 frontend/flag_def.py 318 frontend/flag_spec.py 380 frontend/id_kind_def.py 638 frontend/lexer_def.py 1,012 frontend/lexer.py 330 frontend/location.py 94 frontend/match.py 257 frontend/option_def.py 389 frontend/parse_lib.py 458 frontend/py_reader.py 81 frontend/reader.py 136 frontend/signal_def.py 43 frontend/syntax_abbrev.py 144 frontend/syntax.asdl 576 frontend/typed_args.py 63 frontend/types.asdl 40 mycpp/mylib.pyi 72 native/fanos.c 294 native/fastlex.c 239 native/fastlex.pyi 13 native/libc.c 402 native/libc.pyi 11 native/posix_.pyi 211 osh/arith_parse.py 184 osh/bool_parse.py 269 osh/bool_stat.py 136 osh/braces.py 498 osh/builtin_assign.py 514 osh/builtin_bracket.py 280 osh/builtin_comp.py 501 osh/builtin_lib.py 94 osh/builtin_meta.py 403 osh/builtin_misc.py 844 osh/builtin_printf.py 477 osh/builtin_process.py 539 osh/builtin_pure.py 670 osh/cmd_eval.py 1,772 osh/cmd_parse.py 2,311 osh/glob_.py 498 osh/history.py 184 osh/prompt.py 300 osh/sh_expr_eval.py 1,047 osh/split.py 305 osh/string_ops.py 454 osh/tdop.py 333 osh/word_compile.py 257 osh/word_eval.py 2,297 osh/word_parse.py 1,838 osh/word_.py 817 qsn_/qsn_native.py 75 qsn_/qsn.py 615 36,814 lines in 77 files
mycpp generates the big file _build/cpp/osh_eval.cc. Other programs like Zephyr ASDL and re2c generate other files.
_build/cpp/arg_types.cc 441 _build/cpp/arg_types.h 388 _build/cpp/arith_parse.cc 129 _build/cpp/consts.cc 898 _build/cpp/consts.h 43 _build/cpp/core_optview.h 107 _build/cpp/hnode_asdl.cc 50 _build/cpp/hnode_asdl.h 137 _build/cpp/id_kind_asdl.cc 856 _build/cpp/id_kind_asdl.h 441 _build/cpp/option_asdl.cc 346 _build/cpp/option_asdl.h 188 _build/cpp/osh_eval.cc 32,805 _build/cpp/osh_eval.h 337 _build/cpp/runtime_asdl.cc 2,321 _build/cpp/runtime_asdl.h 1,176 _build/cpp/syntax_asdl.cc 9,999 _build/cpp/syntax_asdl.h 4,221 _build/cpp/types_asdl.cc 96 _build/cpp/types_asdl.h 65 _devbuild/gen/find_nt.h 13 _devbuild/gen/grammar_nt.h 93 _devbuild/gen/id.h 382 _devbuild/gen/osh-lex.h 29,060 _devbuild/gen/osh-types.h 32 84,624 lines in 26 files
A comprehensive test suite that compares OSH against other shells. If OSH passes these tests in BOTH Python and C++, it means that the translation works.
spec/alias.test.sh 536 spec/append.test.sh 291 spec/arith-context.test.sh 81 spec/arith.test.sh 631 spec/array-compat.test.sh 96 spec/array.test.sh 637 spec/assign-deferred.test.sh 105 spec/assign-dialects.test.sh 59 spec/assign-extended.test.sh 810 spec/assign.test.sh 671 spec/assoc.test.sh 631 spec/assoc-zsh.test.sh 53 spec/background.test.sh 150 spec/ble-features.test.sh 631 spec/ble-idioms.test.sh 159 spec/blog1.test.sh 95 spec/blog2.test.sh 49 spec/blog-other1.test.sh 55 spec/brace-expansion.test.sh 441 spec/bugs.test.sh 103 spec/builtin-bash.test.sh 346 spec/builtin-bracket.test.sh 564 spec/builtin-completion.test.sh 399 spec/builtin-dirs.test.sh 177 spec/builtin-eval-source.test.sh 249 spec/builtin-getopts.test.sh 318 spec/builtin-io.test.sh 761 spec/builtin-printf.test.sh 831 spec/builtins2.test.sh 233 spec/builtin-special.test.sh 111 spec/builtins.test.sh 420 spec/builtin-times.test.sh 15 spec/builtin-trap.test.sh 406 spec/builtin-vars.test.sh 710 spec/case_.test.sh 165 spec/command-parsing.test.sh 63 spec/command-sub.test.sh 272 spec/command_.test.sh 192 spec/comments.test.sh 11 spec/dbracket.test.sh 436 spec/dparen.test.sh 202 spec/empty-bodies.test.sh 24 spec/errexit-oil.test.sh 675 spec/errexit.test.sh 392 spec/exit-status.test.sh 238 spec/explore-parsing.test.sh 42 spec/extglob-files.test.sh 391 spec/extglob-match.test.sh 400 spec/for-expr.test.sh 92 spec/func-parsing.test.sh 105 spec/glob.test.sh 396 spec/here-doc.test.sh 365 spec/if_.test.sh 56 spec/interactive.test.sh 216 spec/introspect.test.sh 237 spec/let.test.sh 23 spec/loop.test.sh 397 spec/nameref.test.sh 642 spec/nix-idioms.test.sh 151 spec/nocasematch-match.test.sh 69 spec/oil-array.test.sh 86 spec/oil-assign.test.sh 326 spec/oil-bin.test.sh 17 spec/oil-blocks.test.sh 163 spec/oil-builtin-pp.test.sh 46 spec/oil-builtin-process.test.sh 85 spec/oil-builtins.test.sh 554 spec/oil-builtin-try.test.sh 140 spec/oil-command-sub.test.sh 108 spec/oil-config.test.sh 100 spec/oil-demo.test.sh 59 spec/oil-expr-sub.test.sh 40 spec/oil-expr.test.sh 866 spec/oil-funcs-builtin.test.sh 229 spec/oil-interactive.test.sh 15 spec/oil-json.test.sh 119 spec/oil-keywords.test.sh 37 spec/oil-multiline.test.sh 179 spec/oil-options.test.sh 795 spec/oil-proc.test.sh 252 spec/oil-regex.test.sh 689 spec/oil-scope.test.sh 748 spec/oil-slice-range.test.sh 195 spec/oil-special-vars.test.sh 66 spec/oil-string.test.sh 344 spec/oil-tuple.test.sh 44 spec/oil-user-feedback.test.sh 209 spec/oil-var-sub.test.sh 45 spec/oil-word-eval.test.sh 81 spec/oil-xtrace.test.sh 477 spec/osh-only.test.sh 78 spec/parse-errors.test.sh 192 spec/pipeline.test.sh 166 spec/posix.test.sh 146 spec/process-sub.test.sh 205 spec/prompt.test.sh 254 spec/quote.test.sh 248 spec/redirect.test.sh 654 spec/regex.test.sh 344 spec/serialize.test.sh 229 spec/shell-grammar.test.sh 205 spec/sh-func.test.sh 150 spec/sh-options.test.sh 694 spec/sh-usage.test.sh 94 spec/smoke.test.sh 123 spec/special-vars.test.sh 425 spec/strict-options.test.sh 249 spec/subshell.test.sh 23 spec/tea-func.test.sh 288 spec/tilde.test.sh 172 spec/toysh-posix.test.sh 347 spec/toysh.test.sh 144 spec/type-compat.test.sh 80 spec/var-num.test.sh 39 spec/var-op-bash.test.sh 264 spec/var-op-len.test.sh 219 spec/var-op-patsub.test.sh 291 spec/var-op-slice.test.sh 315 spec/var-op-strip.test.sh 357 spec/var-op-test.test.sh 339 spec/var-ref.test.sh 485 spec/var-sub-quote.test.sh 312 spec/var-sub.test.sh 74 spec/word-eval.test.sh 62 spec/word-split.test.sh 387 spec/xtrace.test.sh 344 34,188 lines in 127 files
Another suite that tests shells "from the outside". Instead of making explicit assertions, we verify that OSH behaves like bash.
test/gold/abuild.sh 9 test/gold/and-or.sh 27 test/gold/comments.sh 22 test/gold/complex-here-docs.sh 192 test/gold/configure-bug.sh 13 test/gold/de1.sh 24 test/gold/declare.sh 28 test/gold/dollar-sq.sh 42 test/gold/echo-e.sh 36 test/gold/errexit-confusion.sh 139 test/gold/export.sh 65 test/gold/glob.sh 26 test/gold/lineno.sh 23 test/gold/nix.sh 32 test/gold/readlink.sh 61 test/gold/readonly.sh 24 test/gold/scope.sh 126 test/gold/strip-op-char-class.sh 6 test/gold/word-eval.sh 57 952 lines in 20 files