| category | total lines | num files |
| mycpp Translator | 4,680 | 11 |
| mycpp Test Data | 3,122 | 39 |
| Zephyr ASDL | 3,093 | 11 |
| pgen2 Parser Generator | 1,717 | 8 |
| Other Code Generators | 2,264 | 9 |
| Hand-written C++ Code | 2,875 | 27 |
| Garbage-Collected Runtime | 3,914 | 18 |
| Unit tests in C++ | 6,088 | 23 |
| Incremental C++ Build | 2,945 | 17 |
| OSH (and common libraries) | 37,592 | 83 |
| Spec Tests | 38,268 | 146 |
| Gold Tests | 2,775 | 27 |
| Generated C++ Code | 82,479 | 24 |
The README for oilshell/oil has another overview of the repository.
This prototype uses the MyPy frontend to translate statically-typed Python to C++. The generated code calls a small runtime which implements things like List[T], Dict[K, V], and Python's len().
mycpp/compare_pairs.py 60 mycpp/const_pass.py 529 mycpp/cppgen_pass.py 2,778 mycpp/crash.py 38 mycpp/debug_pass.py 464 mycpp/format_strings.py 114 mycpp/mycpp_main.py 372 mycpp/mylib.py 218 mycpp/pass_state.py 93 mycpp/util.py 14 4,680 lines in 11 files
Small Python examples that translate to C++, compile, and run.
mycpp/examples/arith_ops.py 45 mycpp/examples/asdl_generated.py 64 mycpp/examples/cartesian.py 78 mycpp/examples/cgi.py 62 mycpp/examples/classes_gc.py 209 mycpp/examples/classes.py 283 mycpp/examples/containers.py 170 mycpp/examples/control_flow.py 113 mycpp/examples/escape.py 70 mycpp/examples/fib_iter.py 54 mycpp/examples/fib_recursive.py 49 mycpp/examples/files.py 64 mycpp/examples/__init__.py 0 mycpp/examples/invalid_condition.py 26 mycpp/examples/invalid_default_args.py 44 mycpp/examples/invalid_format_args.py 15 mycpp/examples/invalid_format_strings.py 20 mycpp/examples/invalid_print.py 13 mycpp/examples/invalid_try_else.py 37 mycpp/examples/invalid_types_2.py 32 mycpp/examples/invalid_types.py 32 mycpp/examples/iterators.py 101 mycpp/examples/length.py 110 mycpp/examples/lexer_main.py 47 mycpp/examples/loops.py 177 mycpp/examples/modules.py 80 mycpp/examples/parse.py 268 mycpp/examples/pgen2_demo.py 106 mycpp/examples/scoped_resource.py 196 mycpp/examples/strings.py 89 mycpp/examples/test_cast.py 59 mycpp/examples/test_conditional.py 67 mycpp/examples/test_default_args.py 75 mycpp/examples/test_globals.py 61 mycpp/examples/test_hoist.py 61 mycpp/examples/test_switch.py 42 mycpp/examples/tuple_return_value.py 55 mycpp/examples/varargs.py 48 3,122 lines in 39 files
A DSL for algebraic data types, borrowed from Python. Oil is the most strongly typed Bourne shell implementation!
asdl/asdl_main.py 300 asdl/ast.py 253 asdl/format.py 498 asdl/front_end.py 497 asdl/gen_cpp.py 688 asdl/gen_python.py 565 asdl/NINJA_subgraph.py 58 asdl/pybase.py 62 asdl/runtime.py 41 asdl/visitor.py 131 3,093 lines in 11 files
An LL(1) parser generator used to parse Oil expressions. Also borrowed from CPython.
pgen2/driver.py 101 pgen2/grammar.py 222 pgen2/parse.py 209 pgen2/pgen.py 428 pgen2/pnode.py 29 pgen2/tokenize.py 579 pgen2/token.py 149 1,717 lines in 8 files
In order to make Oil statically typed, we had to abandon Python reflection and use C++ source code generation instead. The lexer, flag definitions, and constants can be easily compiled to C++.
core/optview_gen.py 99 frontend/consts_gen.py 605 frontend/flag_gen.py 518 frontend/lexer_gen.py 490 frontend/option_gen.py 103 frontend/signal_gen.py 110 oil_lang/grammar_gen.py 239 osh/arith_parse_gen.py 100 2,264 lines in 9 files
Includes OS bindings. Small C++ files like cpp/osh_arith_parse.{cc,h} correspond to larger Python files like osh/arith_parse.py.
cpp/core.cc 338 cpp/core.h 287 cpp/fanos.cc 57 cpp/fanos.h 18 cpp/fanos_shared.h 49 cpp/frontend_flag_spec.cc 268 cpp/frontend_flag_spec.h 165 cpp/frontend_match.cc 100 cpp/frontend_match.h 76 cpp/frontend_pyreadline.cc 300 cpp/frontend_pyreadline.h 76 cpp/libc.cc 210 cpp/libc.h 36 cpp/osh.cc 155 cpp/osh.h 50 cpp/osh_tdop.cc 16 cpp/osh_tdop.h 54 cpp/pgen2.cc 15 cpp/pgen2.h 53 cpp/preamble.h 30 cpp/pylib.cc 58 cpp/pylib.h 22 cpp/qsn.h 56 cpp/stdlib.cc 213 cpp/stdlib.h 137 cpp/translation_stubs.h 36 2,875 lines in 27 files
Uses a simple Cheney / semi-space collector.
mycpp/common.h 66 mycpp/comparators.h 65 mycpp/gc_alloc.h 181 mycpp/gc_builtins.cc 372 mycpp/gc_builtins.h 164 mycpp/gc_dict.h 377 mycpp/gc_list.h 498 mycpp/gc_mylib.cc 249 mycpp/gc_mylib.h 235 mycpp/gc_obj.h 223 mycpp/gc_slab.h 60 mycpp/gc_str.cc 598 mycpp/gc_str.h 170 mycpp/gc_tuple.h 111 mycpp/mark_sweep_heap.cc 369 mycpp/mark_sweep_heap.h 156 mycpp/runtime.h 20 3,914 lines in 18 files
The goal is to make the spec tests pass, but unit tests are helpful too.
cpp/core_test.cc 362 cpp/data_race_test.cc 206 cpp/frontend_flag_spec_test.cc 154 cpp/frontend_match_test.cc 87 cpp/libc_test.cc 135 cpp/obj_layout_test.cc 178 cpp/osh_test.cc 50 cpp/pylib_test.cc 52 cpp/qsn_test.cc 44 cpp/stdlib_test.cc 148 mycpp/bump_leak_heap_test.cc 62 mycpp/cheney_heap_test.cc 30 mycpp/gc_builtins_test.cc 327 mycpp/gc_dict_test.cc 484 mycpp/gc_heap_test.cc 443 mycpp/gc_list_test.cc 481 mycpp/gc_mylib_test.cc 301 mycpp/gc_stress_test.cc 222 mycpp/gc_str_test.cc 1,470 mycpp/gc_tuple_test.cc 134 mycpp/mark_sweep_heap_test.cc 216 mycpp/small_str_test.cc 502 6,088 lines in 23 files
asdl/NINJA_subgraph.py 58 bin/NINJA_subgraph.py 76 build/ninja_lib.py 477 build/ninja_lib_test.py 253 build/ninja_main.py 374 build/ninja-rules-cpp.sh 320 build/ninja-rules-py.sh 347 core/NINJA_subgraph.py 48 cpp/NINJA_subgraph.py 198 frontend/NINJA_subgraph.py 145 mycpp/NINJA_subgraph.py 414 NINJA-config.sh 104 oil_lang/NINJA_subgraph.py 35 osh/NINJA_subgraph.py 31 pea/NINJA_subgraph.py 20 prebuilt/NINJA_subgraph.py 45 2,945 lines in 17 files
This is the input to the translators, written in statically-typed Python. Note that bash is at least 140K lines of code, and OSH implements a large part of bash and more.
bin/oil.py 62 core/alloc.py 269 core/ansi.py 17 core/completion.py 1,377 core/comp_ui.py 564 core/dev.py 598 core/error.py 163 core/executor.py 566 core/main_loop.py 383 core/NINJA_subgraph.py 48 core/optview.py 57 core/process.py 1,525 core/pyerror.py 87 core/pyos.py 403 core/pyutil.py 216 core/runtime.asdl 195 core/shell_native.py 156 core/shell.py 683 core/state.py 2,405 core/ui.py 405 core/util.py 92 core/vm.py 300 frontend/args.py 672 frontend/builtin_def.py 164 frontend/consts.py 350 frontend/flag_def.py 423 frontend/flag_spec.py 390 frontend/id_kind_def.py 638 frontend/lexer_def.py 1,027 frontend/lexer.py 369 frontend/location.py 149 frontend/match.py 265 frontend/NINJA_subgraph.py 145 frontend/option_def.py 375 frontend/parse_lib.py 474 frontend/py_readline.py 97 frontend/reader.py 231 frontend/signal_def.py 46 frontend/syntax_abbrev.py 140 frontend/syntax.asdl 601 frontend/typed_args.py 103 frontend/types.asdl 41 mycpp/mylib.pyi 84 osh/arith_parse.py 184 osh/bool_parse.py 271 osh/bool_stat.py 137 osh/braces.py 516 osh/builtin_assign.py 516 osh/builtin_bracket.py 279 osh/builtin_comp.py 459 osh/builtin_lib.py 95 osh/builtin_meta.py 435 osh/builtin_misc.py 875 osh/builtin_printf.py 482 osh/builtin_process2.py 221 osh/builtin_process.py 131 osh/builtin_pure.py 955 osh/builtin_trap.py 281 osh/cmd_eval.py 1,873 osh/cmd_parse.py 2,421 osh/glob_.py 488 osh/history.py 191 osh/NINJA_subgraph.py 31 osh/prompt.py 300 osh/sh_expr_eval.py 1,057 osh/split.py 308 osh/string_ops.py 459 osh/tdop.py 344 osh/word_compile.py 263 osh/word_eval.py 2,266 osh/word_parse.py 1,871 osh/word_.py 812 pyext/fanos.c 119 pyext/fastlex.c 246 pyext/fastlex.pyi 15 pyext/libc.c 414 pyext/libc.pyi 13 pyext/line_input.pyi 32 pyext/posix_.pyi 211 pyext/yajl.pyi 3 qsn_/qsn_native.py 75 qsn_/qsn.py 588 37,592 lines in 83 files
A comprehensive test suite that compares OSH against other shells. If OSH passes these tests in BOTH Python and C++, it means that the translation works.
spec/alias.test.sh 550 spec/append.test.sh 291 spec/arith-context.test.sh 81 spec/arith.test.sh 625 spec/array-compat.test.sh 96 spec/array.test.sh 631 spec/assign-deferred.test.sh 105 spec/assign-dialects.test.sh 59 spec/assign-extended.test.sh 810 spec/assign.test.sh 672 spec/assoc.test.sh 629 spec/assoc-zsh.test.sh 53 spec/background.test.sh 182 spec/ble-features.test.sh 644 spec/ble-idioms.test.sh 157 spec/blog1.test.sh 95 spec/blog2.test.sh 49 spec/blog-other1.test.sh 55 spec/brace-expansion.test.sh 441 spec/bugs.test.sh 343 spec/builtin-bash.test.sh 346 spec/builtin-bracket.test.sh 564 spec/builtin-completion.test.sh 412 spec/builtin-dirs.test.sh 205 spec/builtin-eval-source.test.sh 249 spec/builtin-getopts.test.sh 318 spec/builtin-io.test.sh 755 spec/builtin-printf.test.sh 898 spec/builtins2.test.sh 233 spec/builtin-special.test.sh 111 spec/builtins.test.sh 420 spec/builtin-times.test.sh 15 spec/builtin-trap.test.sh 410 spec/builtin-vars.test.sh 699 spec/case_.test.sh 165 spec/command-parsing.test.sh 63 spec/command-sub.test.sh 274 spec/command_.test.sh 192 spec/comments.test.sh 11 spec/dbracket.test.sh 434 spec/dparen.test.sh 202 spec/empty-bodies.test.sh 24 spec/errexit-oil.test.sh 755 spec/errexit.test.sh 393 spec/exit-status.test.sh 238 spec/explore-parsing.test.sh 42 spec/extglob-files.test.sh 391 spec/extglob-match.test.sh 400 spec/fatal-errors.test.sh 186 spec/for-expr.test.sh 117 spec/func-parsing.test.sh 105 spec/glob.test.sh 397 spec/hay-isolation.test.sh 169 spec/hay-meta.test.sh 200 spec/hay.test.sh 661 spec/here-doc.test.sh 370 spec/if_.test.sh 56 spec/interactive.test.sh 216 spec/introspect.test.sh 237 spec/let.test.sh 23 spec/loop.test.sh 473 spec/nameref.test.sh 638 spec/nix-idioms.test.sh 150 spec/nocasematch-match.test.sh 69 spec/nul-bytes.test.sh 153 spec/oil-array.test.sh 38 spec/oil-assign.test.sh 359 spec/oil-bin.test.sh 17 spec/oil-blocks.test.sh 386 spec/oil-bugs.test.sh 91 spec/oil-builtin-argparse.test.sh 61 spec/oil-builtin-describe.test.sh 38 spec/oil-builtin-error.test.sh 424 spec/oil-builtin-pp.test.sh 47 spec/oil-builtin-process.test.sh 85 spec/oil-builtin-shopt.test.sh 179 spec/oil-builtins.test.sh 568 spec/oil-case.test.sh 27 spec/oil-command-sub.test.sh 109 spec/oil-demo.test.sh 59 spec/oil-expr-arith.test.sh 48 spec/oil-expr-compare.test.sh 286 spec/oil-expr-sub.test.sh 40 spec/oil-expr.test.sh 738 spec/oil-for.test.sh 173 spec/oil-funcs-builtin.test.sh 200 spec/oil-funcs-external.test.sh 87 spec/oil-interactive.test.sh 15 spec/oil-json.test.sh 121 spec/oil-keywords.test.sh 37 spec/oil-multiline.test.sh 179 spec/oil-options-assign.test.sh 103 spec/oil-options.test.sh 781 spec/oil-proc.test.sh 270 spec/oil-regex.test.sh 960 spec/oil-reserved.test.sh 46 spec/oil-scope.test.sh 763 spec/oil-slice-range.test.sh 162 spec/oil-special-vars.test.sh 48 spec/oil-string.test.sh 344 spec/oil-tuple.test.sh 26 spec/oil-usage.test.sh 22 spec/oil-user-feedback.test.sh 213 spec/oil-var-sub.test.sh 45 spec/oil-with-sh.test.sh 77 spec/oil-word-eval.test.sh 128 spec/oil-xtrace.test.sh 512 spec/osh-only.test.sh 87 spec/parse-errors.test.sh 192 spec/pipeline.test.sh 166 spec/posix.test.sh 153 spec/process-sub.test.sh 213 spec/prompt.test.sh 253 spec/quote.test.sh 248 spec/redirect.test.sh 679 spec/regex.test.sh 344 spec/serialize.test.sh 229 spec/shell-grammar.test.sh 205 spec/sh-func.test.sh 150 spec/sh-options.test.sh 694 spec/sh-usage.test.sh 90 spec/smoke.test.sh 123 spec/strict-options.test.sh 249 spec/subshell.test.sh 23 spec/tea-func.test.sh 288 spec/tilde.test.sh 172 spec/TODO-deprecate.test.sh 18 spec/toysh-posix.test.sh 347 spec/toysh.test.sh 144 spec/type-compat.test.sh 104 spec/var-num.test.sh 39 spec/var-op-bash.test.sh 309 spec/var-op-len.test.sh 219 spec/var-op-patsub.test.sh 291 spec/var-op-slice.test.sh 315 spec/var-op-strip.test.sh 357 spec/var-op-test.test.sh 339 spec/var-ref.test.sh 506 spec/vars-bash.test.sh 27 spec/vars-special.test.sh 597 spec/var-sub-quote.test.sh 312 spec/var-sub.test.sh 74 spec/word-eval.test.sh 62 spec/word-split.test.sh 387 spec/xtrace.test.sh 344 38,268 lines in 146 files
Another suite that tests shells "from the outside". Instead of making explicit assertions, we verify that OSH behaves like bash.
test/gold/abuild.sh 9 test/gold/and-or.sh 27 test/gold/autoconf-backtick.sh 50 test/gold/big-here-doc.sh 1,531 test/gold/case-in-subshell.sh 37 test/gold/char-class.sh 33 test/gold/command-sub-2.sh 12 test/gold/command-sub.sh 39 test/gold/comments.sh 22 test/gold/complex-here-docs.sh 192 test/gold/configure-bug.sh 13 test/gold/de1.sh 24 test/gold/declare.sh 28 test/gold/dollar-sq.sh 42 test/gold/echo-e.sh 36 test/gold/errexit-confusion.sh 139 test/gold/errexit.sh 121 test/gold/export.sh 65 test/gold/glob.sh 26 test/gold/lineno.sh 23 test/gold/nix.sh 32 test/gold/readlink.sh 61 test/gold/readonly.sh 24 test/gold/scope.sh 126 test/gold/strip-op-char-class.sh 6 test/gold/word-eval.sh 57 2,775 lines in 27 files
mycpp generates the big file _gen/bin/oils-for-unix.mycpp.cc. Other programs like Zephyr ASDL and re2c generate other files.
_gen/asdl/hnode.asdl.h 178 _gen/bin/oils_for_unix.mycpp.cc 39,137 _gen/bin/oils_for_unix.mycpp.h 5 _gen/core/optview.h 123 _gen/core/runtime.asdl.cc 1,141 _gen/core/runtime.asdl.h 1,502 _gen/frontend/arg_types.cc 507 _gen/frontend/arg_types.h 725 _gen/frontend/consts.cc 1,574 _gen/frontend/consts.h 52 _gen/frontend/id_kind.asdl.cc 858 _gen/frontend/id_kind.asdl_c.h 383 _gen/frontend/id_kind.asdl.h 446 _gen/frontend/match.re2c.h 24,284 _gen/frontend/option.asdl.h 200 _gen/frontend/signal.cc 268 _gen/frontend/signal.h 16 _gen/frontend/syntax.asdl.cc 4,986 _gen/frontend/syntax.asdl.h 5,766 _gen/frontend/types.asdl_c.h 36 _gen/frontend/types.asdl.h 70 _gen/oil_lang/grammar_nt.h 93 _gen/osh/arith_parse.cc 129 82,479 lines in 24 files