| category | total lines | num files |
| mycpp Translator | 4,662 | 11 |
| mycpp Test Data | 2,830 | 32 |
| Zephyr ASDL | 3,028 | 11 |
| pgen2 Parser Generator | 1,717 | 8 |
| Other Code Generators | 2,211 | 9 |
| Hand-written C++ Code | 2,846 | 30 |
| Garbage-Collected Runtime | 3,682 | 18 |
| Unit tests in C++ | 5,345 | 23 |
| Incremental C++ Build | 2,880 | 17 |
| OSH (and common libraries) | 37,055 | 84 |
| Spec Tests | 37,884 | 146 |
| Gold Tests | 2,775 | 27 |
| Generated C++ Code | 77,791 | 24 |
The README for oilshell/oil has another overview of the repository.
This prototype uses the MyPy frontend to translate statically-typed Python to C++. The generated code calls a small runtime which implements things like List[T], Dict[K, V], and Python's len().
mycpp/compare_pairs.py 58 mycpp/const_pass.py 524 mycpp/cppgen_pass.py 2,827 mycpp/crash.py 35 mycpp/debug_pass.py 459 mycpp/format_strings.py 110 mycpp/mycpp_main.py 331 mycpp/mylib.py 210 mycpp/pass_state.py 94 mycpp/util.py 14 4,662 lines in 11 files
Small Python examples that translate to C++, compile, and run.
mycpp/examples/arith_ops.py 45 mycpp/examples/asdl_generated.py 64 mycpp/examples/cartesian.py 78 mycpp/examples/cgi.py 62 mycpp/examples/classes_gc.py 209 mycpp/examples/classes.py 283 mycpp/examples/containers.py 170 mycpp/examples/control_flow.py 113 mycpp/examples/escape.py 70 mycpp/examples/fib_iter.py 54 mycpp/examples/fib_recursive.py 49 mycpp/examples/files.py 64 mycpp/examples/__init__.py 0 mycpp/examples/invalid_format_strings.py 20 mycpp/examples/invalid_types_2.py 32 mycpp/examples/invalid_types.py 32 mycpp/examples/iterators.py 96 mycpp/examples/length.py 110 mycpp/examples/lexer_main.py 47 mycpp/examples/loops.py 177 mycpp/examples/modules.py 80 mycpp/examples/parse.py 234 mycpp/examples/pgen2_demo.py 106 mycpp/examples/scoped_resource.py 176 mycpp/examples/strings.py 88 mycpp/examples/test_cast.py 59 mycpp/examples/test_conditional.py 67 mycpp/examples/test_hoist.py 61 mycpp/examples/test_switch.py 42 mycpp/examples/tuple_return_value.py 55 mycpp/examples/varargs.py 87 2,830 lines in 32 files
A DSL for algebraic data types, borrowed from Python. Oil is the most strongly typed Bourne shell implementation!
asdl/asdl_main.py 287 asdl/ast.py 253 asdl/format.py 498 asdl/front_end.py 497 asdl/gen_cpp.py 674 asdl/gen_python.py 530 asdl/NINJA_subgraph.py 54 asdl/pybase.py 63 asdl/runtime.py 41 asdl/visitor.py 131 3,028 lines in 11 files
An LL(1) parser generator used to parse Oil expressions. Also borrowed from CPython.
pgen2/driver.py 101 pgen2/grammar.py 222 pgen2/parse.py 209 pgen2/pgen.py 428 pgen2/pnode.py 29 pgen2/tokenize.py 579 pgen2/token.py 149 1,717 lines in 8 files
In order to make Oil statically typed, we had to abandon Python reflection and use C++ source code generation instead. The lexer, flag definitions, and constants can be easily compiled to C++.
core/optview_gen.py 95 frontend/consts_gen.py 564 frontend/flag_gen.py 512 frontend/lexer_gen.py 490 frontend/option_gen.py 101 frontend/signal_gen.py 110 oil_lang/grammar_gen.py 239 osh/arith_parse_gen.py 100 2,211 lines in 9 files
Includes OS bindings. Small C++ files like cpp/osh_arith_parse.{cc,h} correspond to larger Python files like osh/arith_parse.py.
cpp/core.cc 350 cpp/core_error.h 195 cpp/core.h 162 cpp/core_pyerror.h 90 cpp/fanos.cc 57 cpp/fanos.h 18 cpp/fanos_shared.h 49 cpp/frontend_flag_spec.cc 293 cpp/frontend_flag_spec.h 157 cpp/frontend_match.cc 100 cpp/frontend_match.h 75 cpp/frontend_pyreadline.cc 79 cpp/frontend_pyreadline.h 41 cpp/libc.cc 218 cpp/libc.h 36 cpp/osh.cc 153 cpp/osh.h 50 cpp/osh_tdop.cc 16 cpp/osh_tdop.h 50 cpp/pgen2.cc 15 cpp/pgen2.h 53 cpp/preamble.h 32 cpp/pylib.cc 58 cpp/pylib.h 22 cpp/qsn.h 56 cpp/segfault_handler.h 27 cpp/stdlib.cc 210 cpp/stdlib.h 137 cpp/translation_stubs.h 47 2,846 lines in 30 files
Uses a simple Cheney / semi-space collector.
mycpp/common.h 66 mycpp/comparators.h 27 mycpp/gc_alloc.h 177 mycpp/gc_builtins.cc 343 mycpp/gc_builtins.h 151 mycpp/gc_dict.h 340 mycpp/gc_list.h 474 mycpp/gc_mylib.cc 238 mycpp/gc_mylib.h 225 mycpp/gc_obj.h 183 mycpp/gc_slab.h 61 mycpp/gc_str.cc 573 mycpp/gc_str.h 169 mycpp/gc_tuple.h 108 mycpp/mark_sweep_heap.cc 366 mycpp/mark_sweep_heap.h 161 mycpp/runtime.h 20 3,682 lines in 18 files
The goal is to make the spec tests pass, but unit tests are helpful too.
cpp/core_test.cc 329 cpp/data_race_test.cc 85 cpp/frontend_flag_spec_test.cc 154 cpp/frontend_match_test.cc 87 cpp/libc_test.cc 135 cpp/obj_layout_test.cc 121 cpp/osh_test.cc 49 cpp/pylib_test.cc 52 cpp/qsn_test.cc 44 cpp/stdlib_test.cc 148 mycpp/bump_leak_heap_test.cc 62 mycpp/cheney_heap_test.cc 30 mycpp/gc_builtins_test.cc 330 mycpp/gc_dict_test.cc 421 mycpp/gc_heap_test.cc 434 mycpp/gc_list_test.cc 434 mycpp/gc_mylib_test.cc 276 mycpp/gc_stress_test.cc 222 mycpp/gc_str_test.cc 1,443 mycpp/gc_tuple_test.cc 134 mycpp/mark_sweep_heap_test.cc 212 mycpp/smartptr_test.cc 143 5,345 lines in 23 files
asdl/NINJA_subgraph.py 54 bin/NINJA_subgraph.py 71 build/ninja_lib.py 475 build/ninja_lib_test.py 253 build/ninja_main.py 362 build/ninja-rules-cpp.sh 312 build/ninja-rules-py.sh 356 core/NINJA_subgraph.py 48 cpp/NINJA_subgraph.py 182 frontend/NINJA_subgraph.py 138 mycpp/NINJA_subgraph.py 413 NINJA-config.sh 104 oil_lang/NINJA_subgraph.py 35 osh/NINJA_subgraph.py 31 pea/NINJA_subgraph.py 20 prebuilt/NINJA_subgraph.py 26 2,880 lines in 17 files
This is the input to the translators, written in statically-typed Python. Note that bash is at least 140K lines of code, and OSH implements a large part of bash and more.
bin/oil.py 69 core/alloc.py 166 core/ansi.py 17 core/completion.py 1,267 core/comp_ui.py 592 core/dev.py 598 core/error.py 198 core/executor.py 559 core/main_loop.py 367 core/NINJA_subgraph.py 48 core/optview.py 57 core/process.py 1,526 core/pyerror.py 71 core/pyos.py 385 core/pyutil.py 227 core/runtime.asdl 194 core/shell_native.py 156 core/shell.py 699 core/state.py 2,387 core/ui.py 396 core/util.py 88 core/vm.py 230 frontend/args.py 673 frontend/builtin_def.py 164 frontend/consts.py 342 frontend/flag_def.py 323 frontend/flag_spec.py 391 frontend/id_kind_def.py 638 frontend/lexer_def.py 1,031 frontend/lexer.py 330 frontend/location.py 99 frontend/match.py 265 frontend/NINJA_subgraph.py 138 frontend/option_def.py 382 frontend/parse_lib.py 478 frontend/py_reader.py 81 frontend/py_readline.py 93 frontend/reader.py 143 frontend/signal_def.py 46 frontend/syntax_abbrev.py 144 frontend/syntax.asdl 594 frontend/typed_args.py 70 frontend/types.asdl 41 mycpp/mylib.pyi 81 osh/arith_parse.py 184 osh/bool_parse.py 269 osh/bool_stat.py 136 osh/braces.py 498 osh/builtin_assign.py 513 osh/builtin_bracket.py 280 osh/builtin_comp.py 518 osh/builtin_lib.py 95 osh/builtin_meta.py 435 osh/builtin_misc.py 873 osh/builtin_printf.py 478 osh/builtin_process2.py 222 osh/builtin_process.py 131 osh/builtin_pure.py 960 osh/builtin_trap.py 286 osh/cmd_eval.py 1,860 osh/cmd_parse.py 2,418 osh/glob_.py 488 osh/history.py 191 osh/NINJA_subgraph.py 31 osh/prompt.py 300 osh/sh_expr_eval.py 1,048 osh/split.py 307 osh/string_ops.py 454 osh/tdop.py 333 osh/word_compile.py 263 osh/word_eval.py 2,253 osh/word_parse.py 1,852 osh/word_.py 821 pyext/fanos.c 119 pyext/fastlex.c 246 pyext/fastlex.pyi 15 pyext/libc.c 414 pyext/libc.pyi 11 pyext/line_input.pyi 32 pyext/posix_.pyi 211 pyext/yajl.pyi 3 qsn_/qsn_native.py 75 qsn_/qsn.py 588 37,055 lines in 84 files
A comprehensive test suite that compares OSH against other shells. If OSH passes these tests in BOTH Python and C++, it means that the translation works.
spec/alias.test.sh 536 spec/append.test.sh 291 spec/arith-context.test.sh 81 spec/arith.test.sh 631 spec/array-compat.test.sh 96 spec/array.test.sh 637 spec/assign-deferred.test.sh 105 spec/assign-dialects.test.sh 59 spec/assign-extended.test.sh 810 spec/assign.test.sh 672 spec/assoc.test.sh 631 spec/assoc-zsh.test.sh 53 spec/background.test.sh 182 spec/ble-features.test.sh 644 spec/ble-idioms.test.sh 168 spec/blog1.test.sh 95 spec/blog2.test.sh 49 spec/blog-other1.test.sh 55 spec/brace-expansion.test.sh 441 spec/bugs.test.sh 258 spec/builtin-bash.test.sh 346 spec/builtin-bracket.test.sh 564 spec/builtin-completion.test.sh 412 spec/builtin-dirs.test.sh 205 spec/builtin-eval-source.test.sh 249 spec/builtin-getopts.test.sh 318 spec/builtin-io.test.sh 751 spec/builtin-printf.test.sh 900 spec/builtins2.test.sh 233 spec/builtin-special.test.sh 111 spec/builtins.test.sh 420 spec/builtin-times.test.sh 15 spec/builtin-trap.test.sh 410 spec/builtin-vars.test.sh 710 spec/case_.test.sh 165 spec/command-parsing.test.sh 63 spec/command-sub.test.sh 274 spec/command_.test.sh 192 spec/comments.test.sh 11 spec/dbracket.test.sh 436 spec/dparen.test.sh 202 spec/empty-bodies.test.sh 24 spec/errexit-oil.test.sh 755 spec/errexit.test.sh 393 spec/exit-status.test.sh 238 spec/explore-parsing.test.sh 42 spec/extglob-files.test.sh 391 spec/extglob-match.test.sh 400 spec/fatal-errors.test.sh 186 spec/for-expr.test.sh 117 spec/func-parsing.test.sh 105 spec/glob.test.sh 396 spec/hay-isolation.test.sh 169 spec/hay-meta.test.sh 200 spec/hay.test.sh 645 spec/here-doc.test.sh 365 spec/if_.test.sh 56 spec/interactive.test.sh 216 spec/introspect.test.sh 237 spec/let.test.sh 23 spec/loop.test.sh 408 spec/nameref.test.sh 642 spec/nix-idioms.test.sh 151 spec/nocasematch-match.test.sh 69 spec/nul-bytes.test.sh 153 spec/oil-array.test.sh 38 spec/oil-assign.test.sh 359 spec/oil-bin.test.sh 17 spec/oil-blocks.test.sh 377 spec/oil-bugs.test.sh 91 spec/oil-builtin-argparse.test.sh 61 spec/oil-builtin-describe.test.sh 38 spec/oil-builtin-error.test.sh 424 spec/oil-builtin-pp.test.sh 47 spec/oil-builtin-process.test.sh 85 spec/oil-builtin-shopt.test.sh 179 spec/oil-builtins.test.sh 568 spec/oil-case.test.sh 27 spec/oil-command-sub.test.sh 109 spec/oil-demo.test.sh 59 spec/oil-expr-arith.test.sh 48 spec/oil-expr-compare.test.sh 286 spec/oil-expr-sub.test.sh 40 spec/oil-expr.test.sh 738 spec/oil-for.test.sh 173 spec/oil-funcs-builtin.test.sh 200 spec/oil-funcs-external.test.sh 87 spec/oil-interactive.test.sh 15 spec/oil-json.test.sh 121 spec/oil-keywords.test.sh 37 spec/oil-multiline.test.sh 179 spec/oil-options-assign.test.sh 103 spec/oil-options.test.sh 781 spec/oil-proc.test.sh 270 spec/oil-regex.test.sh 960 spec/oil-reserved.test.sh 46 spec/oil-scope.test.sh 764 spec/oil-slice-range.test.sh 162 spec/oil-special-vars.test.sh 48 spec/oil-string.test.sh 344 spec/oil-tuple.test.sh 26 spec/oil-usage.test.sh 22 spec/oil-user-feedback.test.sh 213 spec/oil-var-sub.test.sh 45 spec/oil-with-sh.test.sh 77 spec/oil-word-eval.test.sh 128 spec/oil-xtrace.test.sh 478 spec/osh-only.test.sh 78 spec/parse-errors.test.sh 192 spec/pipeline.test.sh 166 spec/posix.test.sh 153 spec/process-sub.test.sh 213 spec/prompt.test.sh 253 spec/quote.test.sh 248 spec/redirect.test.sh 654 spec/regex.test.sh 344 spec/serialize.test.sh 229 spec/shell-grammar.test.sh 205 spec/sh-func.test.sh 150 spec/sh-options.test.sh 694 spec/sh-usage.test.sh 90 spec/smoke.test.sh 123 spec/strict-options.test.sh 249 spec/subshell.test.sh 23 spec/tea-func.test.sh 288 spec/tilde.test.sh 172 spec/TODO-deprecate.test.sh 14 spec/toysh-posix.test.sh 347 spec/toysh.test.sh 144 spec/type-compat.test.sh 107 spec/var-num.test.sh 39 spec/var-op-bash.test.sh 309 spec/var-op-len.test.sh 219 spec/var-op-patsub.test.sh 291 spec/var-op-slice.test.sh 315 spec/var-op-strip.test.sh 357 spec/var-op-test.test.sh 339 spec/var-ref.test.sh 485 spec/vars-bash.test.sh 27 spec/vars-special.test.sh 456 spec/var-sub-quote.test.sh 312 spec/var-sub.test.sh 74 spec/word-eval.test.sh 62 spec/word-split.test.sh 387 spec/xtrace.test.sh 344 37,884 lines in 146 files
Another suite that tests shells "from the outside". Instead of making explicit assertions, we verify that OSH behaves like bash.
test/gold/abuild.sh 9 test/gold/and-or.sh 27 test/gold/autoconf-backtick.sh 50 test/gold/big-here-doc.sh 1,531 test/gold/case-in-subshell.sh 37 test/gold/char-class.sh 33 test/gold/command-sub-2.sh 12 test/gold/command-sub.sh 39 test/gold/comments.sh 22 test/gold/complex-here-docs.sh 192 test/gold/configure-bug.sh 13 test/gold/de1.sh 24 test/gold/declare.sh 28 test/gold/dollar-sq.sh 42 test/gold/echo-e.sh 36 test/gold/errexit-confusion.sh 139 test/gold/errexit.sh 121 test/gold/export.sh 65 test/gold/glob.sh 26 test/gold/lineno.sh 23 test/gold/nix.sh 32 test/gold/readlink.sh 61 test/gold/readonly.sh 24 test/gold/scope.sh 126 test/gold/strip-op-char-class.sh 6 test/gold/word-eval.sh 57 2,775 lines in 27 files
mycpp generates the big file _gen/bin/oils_cpp.mycpp.cc. Other programs like Zephyr ASDL and re2c generate other files.
_gen/asdl/hnode.asdl.h 198 _gen/bin/oils_cpp.mycpp.cc 34,232 _gen/bin/oils_cpp.mycpp.h 5 _gen/core/optview.h 118 _gen/core/runtime.asdl.cc 1,165 _gen/core/runtime.asdl.h 1,571 _gen/frontend/arg_types.cc 410 _gen/frontend/arg_types.h 514 _gen/frontend/consts.cc 1,355 _gen/frontend/consts.h 46 _gen/frontend/id_kind.asdl.cc 858 _gen/frontend/id_kind.asdl_c.h 383 _gen/frontend/id_kind.asdl.h 442 _gen/frontend/match.re2c.h 24,284 _gen/frontend/option.asdl.h 193 _gen/frontend/signal.cc 268 _gen/frontend/signal.h 16 _gen/frontend/syntax.asdl.cc 5,078 _gen/frontend/syntax.asdl.h 6,332 _gen/frontend/types.asdl_c.h 36 _gen/frontend/types.asdl.h 65 _gen/oil_lang/grammar_nt.h 93 _gen/osh/arith_parse.cc 129 77,791 lines in 24 files