| category | total lines | num files |
| mycpp Translator | 4,681 | 11 |
| mycpp Test Data | 3,305 | 41 |
| Zephyr ASDL | 3,177 | 12 |
| pgen2 Parser Generator | 1,860 | 8 |
| Other Code Generators | 2,255 | 9 |
| Hand-written C++ Code | 3,007 | 27 |
| Garbage-Collected Runtime | 4,135 | 18 |
| Unit tests in C++ | 6,160 | 23 |
| Incremental C++ Build | 3,009 | 17 |
| OSH (and common libraries) | 38,237 | 80 |
| Spec Tests | 38,995 | 148 |
| Gold Tests | 2,775 | 27 |
| Generated C++ Code | 87,895 | 25 |
The README for oilshell/oil has another overview of the repository.
This prototype uses the MyPy frontend to translate statically-typed Python to C++. The generated code calls a small runtime which implements things like List[T], Dict[K, V], and Python's len().
mycpp/compare_pairs.py 60 mycpp/const_pass.py 529 mycpp/cppgen_pass.py 2,782 mycpp/crash.py 38 mycpp/debug_pass.py 464 mycpp/format_strings.py 114 mycpp/mycpp_main.py 372 mycpp/mylib.py 215 mycpp/pass_state.py 93 mycpp/util.py 14 4,681 lines in 11 files
Small Python examples that translate to C++, compile, and run.
mycpp/examples/arith_ops.py 45 mycpp/examples/asdl_generated.py 64 mycpp/examples/cartesian.py 78 mycpp/examples/cgi.py 62 mycpp/examples/classes_gc.py 209 mycpp/examples/classes.py 283 mycpp/examples/containers.py 170 mycpp/examples/control_flow.py 113 mycpp/examples/escape.py 70 mycpp/examples/fib_iter.py 54 mycpp/examples/fib_recursive.py 49 mycpp/examples/files.py 64 mycpp/examples/__init__.py 0 mycpp/examples/invalid_condition.py 34 mycpp/examples/invalid_default_args.py 44 mycpp/examples/invalid_except.py 32 mycpp/examples/invalid_format_args.py 15 mycpp/examples/invalid_format_strings.py 20 mycpp/examples/invalid_print.py 13 mycpp/examples/invalid_try_else.py 37 mycpp/examples/invalid_types_2.py 32 mycpp/examples/invalid_types.py 32 mycpp/examples/iterators.py 101 mycpp/examples/length.py 110 mycpp/examples/lexer_main.py 47 mycpp/examples/loops.py 177 mycpp/examples/modules.py 80 mycpp/examples/parse.py 264 mycpp/examples/pgen2_demo.py 106 mycpp/examples/scoped_resource.py 196 mycpp/examples/strings.py 89 mycpp/examples/test_cast.py 59 mycpp/examples/test_conditional.py 67 mycpp/examples/test_default_args.py 75 mycpp/examples/test_globals.py 77 mycpp/examples/test_hoist.py 61 mycpp/examples/test_io_os_error.py 131 mycpp/examples/test_switch.py 42 mycpp/examples/tuple_return_value.py 55 mycpp/examples/varargs.py 48 3,305 lines in 41 files
A DSL for algebraic data types, borrowed from Python. Oil is the most strongly typed Bourne shell implementation!
asdl/asdl_main.py 299 asdl/ast.py 270 asdl/format.py 500 asdl/front_end.py 511 asdl/gen_cpp.py 734 asdl/gen_python.py 551 asdl/NINJA_subgraph.py 58 asdl/pybase.py 62 asdl/runtime.py 41 asdl/util.py 13 asdl/visitor.py 138 3,177 lines in 12 files
An LL(1) parser generator used to parse Oil expressions. Also borrowed from CPython.
pgen2/driver.py 101 pgen2/grammar.py 303 pgen2/parse.py 220 pgen2/pgen.py 428 pgen2/pnode.py 80 pgen2/tokenize.py 579 pgen2/token.py 149 1,860 lines in 8 files
In order to make Oil statically typed, we had to abandon Python reflection and use C++ source code generation instead. The lexer, flag definitions, and constants can be easily compiled to C++.
core/optview_gen.py 97 frontend/consts_gen.py 604 frontend/flag_gen.py 512 frontend/lexer_gen.py 490 frontend/option_gen.py 99 frontend/signal_gen.py 110 oil_lang/grammar_gen.py 243 osh/arith_parse_gen.py 100 2,255 lines in 9 files
Includes OS bindings. Small C++ files like cpp/osh_arith_parse.{cc,h} correspond to larger Python files like osh/arith_parse.py.
cpp/core.cc 354 cpp/core.h 283 cpp/fanos.cc 57 cpp/fanos.h 18 cpp/fanos_shared.h 49 cpp/frontend_flag_spec.cc 267 cpp/frontend_flag_spec.h 161 cpp/frontend_match.cc 100 cpp/frontend_match.h 75 cpp/frontend_pyreadline.cc 304 cpp/frontend_pyreadline.h 74 cpp/libc.cc 217 cpp/libc.h 36 cpp/osh.cc 157 cpp/osh.h 50 cpp/osh_tdop.cc 16 cpp/osh_tdop.h 52 cpp/pgen2.cc 48 cpp/pgen2.h 87 cpp/preamble.h 37 cpp/pylib.cc 58 cpp/pylib.h 22 cpp/qsn.h 56 cpp/stdlib.cc 228 cpp/stdlib.h 165 cpp/translation_stubs.h 36 3,007 lines in 27 files
Uses a simple Cheney / semi-space collector.
mycpp/common.h 66 mycpp/comparators.h 65 mycpp/gc_alloc.h 234 mycpp/gc_builtins.cc 390 mycpp/gc_builtins.h 159 mycpp/gc_dict.h 378 mycpp/gc_list.h 495 mycpp/gc_mylib.cc 249 mycpp/gc_mylib.h 253 mycpp/gc_obj.h 185 mycpp/gc_slab.h 58 mycpp/gc_str.cc 598 mycpp/gc_str.h 168 mycpp/gc_tuple.h 104 mycpp/mark_sweep_heap.cc 423 mycpp/mark_sweep_heap.h 290 mycpp/runtime.h 20 4,135 lines in 18 files
The goal is to make the spec tests pass, but unit tests are helpful too.
cpp/core_test.cc 382 cpp/data_race_test.cc 206 cpp/frontend_flag_spec_test.cc 154 cpp/frontend_match_test.cc 87 cpp/libc_test.cc 135 cpp/obj_layout_test.cc 178 cpp/osh_test.cc 50 cpp/pylib_test.cc 52 cpp/qsn_test.cc 44 cpp/stdlib_test.cc 152 mycpp/bump_leak_heap_test.cc 62 mycpp/cheney_heap_test.cc 30 mycpp/gc_builtins_test.cc 327 mycpp/gc_dict_test.cc 490 mycpp/gc_heap_test.cc 396 mycpp/gc_list_test.cc 488 mycpp/gc_mylib_test.cc 301 mycpp/gc_stress_test.cc 222 mycpp/gc_str_test.cc 1,475 mycpp/gc_tuple_test.cc 138 mycpp/mark_sweep_heap_test.cc 289 mycpp/small_str_test.cc 502 6,160 lines in 23 files
asdl/NINJA_subgraph.py 58 bin/NINJA_subgraph.py 76 build/ninja_lib.py 480 build/ninja_lib_test.py 253 build/ninja_main.py 402 build/ninja-rules-cpp.sh 349 build/ninja-rules-py.sh 347 core/NINJA_subgraph.py 48 cpp/NINJA_subgraph.py 199 frontend/NINJA_subgraph.py 145 mycpp/NINJA_subgraph.py 415 NINJA-config.sh 104 oil_lang/NINJA_subgraph.py 37 osh/NINJA_subgraph.py 31 pea/NINJA_subgraph.py 20 prebuilt/NINJA_subgraph.py 45 3,009 lines in 17 files
This is the input to the translators, written in statically-typed Python. Note that bash is at least 140K lines of code, and OSH implements a large part of bash and more.
bin/oils_for_unix.py 224 core/alloc.py 269 core/ansi.py 17 core/completion.py 1,376 core/comp_ui.py 564 core/dev.py 598 core/error.py 227 core/executor.py 640 core/main_loop.py 382 core/NINJA_subgraph.py 48 core/optview.py 57 core/process.py 1,752 core/pyos.py 403 core/pyutil.py 216 core/runtime.asdl 208 core/shell.py 838 core/state.py 2,450 core/ui.py 418 core/util.py 92 core/vm.py 299 data_lang/qsn_native.py 76 data_lang/qsn.py 588 frontend/args.py 666 frontend/builtin_def.py 164 frontend/consts.py 350 frontend/flag_def.py 446 frontend/flag_spec.py 390 frontend/id_kind_def.py 638 frontend/lexer_def.py 1,027 frontend/lexer.py 369 frontend/location.py 146 frontend/match.py 265 frontend/NINJA_subgraph.py 145 frontend/option_def.py 373 frontend/parse_lib.py 500 frontend/py_readline.py 97 frontend/reader.py 231 frontend/signal_def.py 95 frontend/syntax_abbrev.py 140 frontend/syntax.asdl 604 frontend/typed_args.py 103 frontend/types.asdl 42 mycpp/mylib.pyi 84 osh/arith_parse.py 184 osh/bool_parse.py 272 osh/bool_stat.py 137 osh/braces.py 515 osh/builtin_assign.py 518 osh/builtin_bracket.py 282 osh/builtin_comp.py 466 osh/builtin_lib.py 132 osh/builtin_meta.py 439 osh/builtin_misc.py 874 osh/builtin_printf.py 480 osh/builtin_process.py 339 osh/builtin_pure.py 952 osh/builtin_trap.py 262 osh/cmd_eval.py 1,960 osh/cmd_parse.py 2,420 osh/glob_.py 486 osh/history.py 191 osh/NINJA_subgraph.py 31 osh/prompt.py 299 osh/sh_expr_eval.py 1,046 osh/split.py 308 osh/string_ops.py 458 osh/tdop.py 344 osh/word_compile.py 263 osh/word_eval.py 2,238 osh/word_parse.py 1,865 osh/word_.py 806 pyext/fanos.c 119 pyext/fastlex.c 246 pyext/fastlex.pyi 15 pyext/libc.c 414 pyext/libc.pyi 13 pyext/line_input.pyi 32 pyext/posix_.pyi 211 pyext/yajl.pyi 3 38,237 lines in 80 files
A comprehensive test suite that compares OSH against other shells. If OSH passes these tests in BOTH Python and C++, it means that the translation works.
spec/alias.test.sh 550 spec/append.test.sh 293 spec/arith-context.test.sh 81 spec/arith.test.sh 626 spec/array-compat.test.sh 102 spec/array.test.sh 627 spec/assign-deferred.test.sh 105 spec/assign-dialects.test.sh 59 spec/assign-extended.test.sh 810 spec/assign.test.sh 672 spec/assoc.test.sh 642 spec/assoc-zsh.test.sh 53 spec/background.test.sh 182 spec/ble-features.test.sh 644 spec/ble-idioms.test.sh 157 spec/blog1.test.sh 95 spec/blog2.test.sh 49 spec/blog-other1.test.sh 55 spec/brace-expansion.test.sh 441 spec/bugs.test.sh 343 spec/builtin-bash.test.sh 346 spec/builtin-bracket.test.sh 564 spec/builtin-completion.test.sh 418 spec/builtin-dirs.test.sh 205 spec/builtin-eval-source.test.sh 249 spec/builtin-getopts.test.sh 347 spec/builtin-history.test.sh 119 spec/builtin-io.test.sh 755 spec/builtin-printf.test.sh 898 spec/builtins2.test.sh 211 spec/builtin-special.test.sh 111 spec/builtins.test.sh 434 spec/builtin-times.test.sh 15 spec/builtin-trap-bash.test.sh 473 spec/builtin-trap.test.sh 337 spec/builtin-vars.test.sh 699 spec/case_.test.sh 165 spec/command-parsing.test.sh 63 spec/command-sub.test.sh 274 spec/command_.test.sh 192 spec/comments.test.sh 11 spec/dbracket.test.sh 434 spec/dparen.test.sh 204 spec/empty-bodies.test.sh 24 spec/errexit-oil.test.sh 755 spec/errexit.test.sh 393 spec/exit-status.test.sh 238 spec/explore-parsing.test.sh 42 spec/extglob-files.test.sh 391 spec/extglob-match.test.sh 400 spec/fatal-errors.test.sh 186 spec/for-expr.test.sh 117 spec/func-parsing.test.sh 105 spec/glob.test.sh 397 spec/hay-isolation.test.sh 169 spec/hay-meta.test.sh 200 spec/hay.test.sh 661 spec/here-doc.test.sh 370 spec/if_.test.sh 56 spec/interactive.test.sh 311 spec/introspect.test.sh 232 spec/let.test.sh 23 spec/loop.test.sh 473 spec/nameref.test.sh 638 spec/nix-idioms.test.sh 150 spec/nocasematch-match.test.sh 69 spec/nul-bytes.test.sh 153 spec/oil-array.test.sh 38 spec/oil-assign.test.sh 359 spec/oil-bin.test.sh 17 spec/oil-blocks.test.sh 386 spec/oil-bugs.test.sh 91 spec/oil-builtin-argparse.test.sh 61 spec/oil-builtin-describe.test.sh 38 spec/oil-builtin-error.test.sh 424 spec/oil-builtin-pp.test.sh 47 spec/oil-builtin-process.test.sh 85 spec/oil-builtin-shopt.test.sh 179 spec/oil-builtins.test.sh 568 spec/oil-case.test.sh 13 spec/oil-command-sub.test.sh 109 spec/oil-demo.test.sh 59 spec/oil-expr-arith.test.sh 97 spec/oil-expr-compare.test.sh 286 spec/oil-expr-sub.test.sh 40 spec/oil-expr.test.sh 738 spec/oil-for.test.sh 173 spec/oil-funcs-builtin.test.sh 200 spec/oil-funcs-external.test.sh 87 spec/oil-interactive.test.sh 15 spec/oil-json.test.sh 121 spec/oil-keywords.test.sh 37 spec/oil-multiline.test.sh 179 spec/oil-options-assign.test.sh 103 spec/oil-options.test.sh 781 spec/oil-proc.test.sh 270 spec/oil-regex.test.sh 960 spec/oil-reserved.test.sh 46 spec/oil-scope.test.sh 763 spec/oil-slice-range.test.sh 162 spec/oil-special-vars.test.sh 48 spec/oil-string.test.sh 344 spec/oil-tuple.test.sh 26 spec/oil-usage.test.sh 22 spec/oil-user-feedback.test.sh 213 spec/oil-var-sub.test.sh 45 spec/oil-with-sh.test.sh 77 spec/oil-word-eval.test.sh 128 spec/oil-xtrace.test.sh 516 spec/osh-only.test.sh 87 spec/parse-errors.test.sh 192 spec/pipeline.test.sh 197 spec/posix.test.sh 153 spec/process-sub.test.sh 213 spec/prompt.test.sh 254 spec/quote.test.sh 248 spec/redirect.test.sh 679 spec/regex.test.sh 344 spec/serialize.test.sh 229 spec/shell-grammar.test.sh 205 spec/sh-func.test.sh 150 spec/sh-options.test.sh 694 spec/sh-usage.test.sh 90 spec/smoke.test.sh 123 spec/strict-options.test.sh 249 spec/subshell.test.sh 23 spec/tea-func.test.sh 288 spec/tilde.test.sh 172 spec/TODO-deprecate.test.sh 18 spec/toysh-posix.test.sh 347 spec/toysh.test.sh 144 spec/type-compat.test.sh 104 spec/var-num.test.sh 39 spec/var-op-bash.test.sh 307 spec/var-op-len.test.sh 219 spec/var-op-patsub.test.sh 291 spec/var-op-slice.test.sh 315 spec/var-op-strip.test.sh 357 spec/var-op-test.test.sh 339 spec/var-ref.test.sh 508 spec/vars-bash.test.sh 27 spec/vars-special.test.sh 597 spec/var-sub-quote.test.sh 312 spec/var-sub.test.sh 74 spec/word-eval.test.sh 62 spec/word-split.test.sh 387 spec/xtrace.test.sh 344 38,995 lines in 148 files
Another suite that tests shells "from the outside". Instead of making explicit assertions, we verify that OSH behaves like bash.
test/gold/abuild.sh 9 test/gold/and-or.sh 27 test/gold/autoconf-backtick.sh 50 test/gold/big-here-doc.sh 1,531 test/gold/case-in-subshell.sh 37 test/gold/char-class.sh 33 test/gold/command-sub-2.sh 12 test/gold/command-sub.sh 39 test/gold/comments.sh 22 test/gold/complex-here-docs.sh 192 test/gold/configure-bug.sh 13 test/gold/de1.sh 24 test/gold/declare.sh 28 test/gold/dollar-sq.sh 42 test/gold/echo-e.sh 36 test/gold/errexit-confusion.sh 139 test/gold/errexit.sh 121 test/gold/export.sh 65 test/gold/glob.sh 26 test/gold/lineno.sh 23 test/gold/nix.sh 32 test/gold/readlink.sh 61 test/gold/readonly.sh 24 test/gold/scope.sh 126 test/gold/strip-op-char-class.sh 6 test/gold/word-eval.sh 57 2,775 lines in 27 files
mycpp generates the big file _gen/bin/oils-for-unix.mycpp.cc. Other programs like Zephyr ASDL and re2c generate other files.
_gen/asdl/hnode.asdl.h 176 _gen/bin/oils_for_unix.mycpp.cc 39,949 _gen/bin/oils_for_unix.mycpp.h 5 _gen/core/optview.h 119 _gen/core/runtime.asdl.cc 1,192 _gen/core/runtime.asdl.h 1,465 _gen/frontend/arg_types.cc 537 _gen/frontend/arg_types.h 709 _gen/frontend/consts.cc 1,564 _gen/frontend/consts.h 52 _gen/frontend/id_kind.asdl.cc 858 _gen/frontend/id_kind.asdl_c.h 383 _gen/frontend/id_kind.asdl.h 446 _gen/frontend/match.re2c.h 24,284 _gen/frontend/option.asdl.h 198 _gen/frontend/signal.cc 205 _gen/frontend/signal.h 16 _gen/frontend/syntax.asdl.cc 4,991 _gen/frontend/syntax.asdl.h 5,457 _gen/frontend/types.asdl_c.h 36 _gen/frontend/types.asdl.h 70 _gen/oil_lang/grammar_nt.h 93 _gen/oil_lang/grammar_tables.cc 4,961 _gen/osh/arith_parse.cc 129 87,895 lines in 25 files