Overview of Oil Code

category total lines num files
OSH (and common libraries) 37,095 77
Oil Language (and Tea) 4,871 11
Code Borrowed from Python's stdlib 215 4
Spec Tests 35,170 130
Gold Tests 2,725 26
Language Unit Tests 7,229 42
Other Unit Tests 924 11
mycpp Translator 4,294 10
mycpp Test Data 2,182 25
Zephyr ASDL 2,904 10
pgen2 Parser Generator 1,717 8
Other Code Generators 2,062 8
Build Automation 5,091 35
Test Automation 12,897 45
Release Automation 2,349 6
Benchmarks 5,633 33
Metrics 1,400 8
Generated Python Code 46,550 27
Tools 1,523 4
Doc Tools 2,315 10
Web 1,058 5

Related Documents

The README for oilshell/oil has another overview of the repository.

OSH (and common libraries)

This is the input to the translators, written in statically-typed Python. Note that bash is at least 140K lines of code, and OSH implements a large part of bash and more.

bin/oil.py                                      360
core/alloc.py                                   133
core/ansi.py                                     17
core/completion.py                            1,261
core/comp_ui.py                                 585
core/dev.py                                     597
core/error.py                                   195
core/executor.py                                545
core/main_loop.py                               361
core/optview.py                                  57
core/oven.py                                    574
core/process.py                               1,498
core/pyerror.py                                  58
core/pyos.py                                    384
core/pyutil.py                                  247
core/runtime.asdl                               193
core/shell_native.py                            492
core/shell.py                                   695
core/state.py                                 2,179
core/ui.py                                      381
core/util.py                                     93
core/vm.py                                      209
frontend/args.py                                673
frontend/builtin_def.py                         156
frontend/consts.py                              321
frontend/flag_def.py                            322
frontend/flag_spec.py                           391
frontend/id_kind_def.py                         638
frontend/lexer_def.py                         1,012
frontend/lexer.py                               330
frontend/location.py                             99
frontend/match.py                               260
frontend/option_def.py                          393
frontend/parse_lib.py                           458
frontend/py_reader.py                            81
frontend/reader.py                              136
frontend/signal_def.py                           43
frontend/syntax_abbrev.py                       144
frontend/syntax.asdl                            576
frontend/typed_args.py                           70
frontend/types.asdl                              40
mycpp/mylib.pyi                                  72
native/fanos.c                                  294
native/fastlex.c                                239
native/fastlex.pyi                               13
native/libc.c                                   402
native/libc.pyi                                  11
native/posix_.pyi                               211
osh/arith_parse.py                              184
osh/bool_parse.py                               269
osh/bool_stat.py                                136
osh/braces.py                                   498
osh/builtin_assign.py                           514
osh/builtin_bracket.py                          280
osh/builtin_comp.py                             518
osh/builtin_lib.py                               94
osh/builtin_meta.py                             435
osh/builtin_misc.py                             853
osh/builtin_printf.py                           477
osh/builtin_process.py                          555
osh/builtin_pure.py                             673
osh/cmd_eval.py                               1,779
osh/cmd_parse.py                              2,311
osh/glob_.py                                    498
osh/history.py                                  184
osh/prompt.py                                   300
osh/sh_expr_eval.py                           1,047
osh/split.py                                    305
osh/string_ops.py                               454
osh/tdop.py                                     333
osh/word_compile.py                             257
osh/word_eval.py                              2,297
osh/word_parse.py                             1,838
osh/word_.py                                    817
qsn_/qsn_native.py                               75
qsn_/qsn.py                                     615

37,095 lines in 77 files  

Oil Language (and Tea)

oil_lang/builtin_oil.py                         338
oil_lang/expr_eval.py                           834
oil_lang/expr_parse.py                          335
oil_lang/expr_to_ast.py                       1,536
oil_lang/funcs_builtin.py                       440
oil_lang/grammar.pgen2                          615
oil_lang/objects.py                             290
oil_lang/regex_translate.py                     262
tea/tea_eval.py                                 122
tea/tea_main.py                                  99

4,871 lines in 11 files  

Code Borrowed from Python's stdlib

pylib/cgi.py                                     19
pylib/os_path.py                                169
pylib/path_stat.py                               27

215 lines in 4 files  

Spec Tests

A comprehensive test suite that compares OSH against other shells. If OSH passes these tests in BOTH Python and C++, it means that the translation works.

spec/alias.test.sh                              536
spec/append.test.sh                             291
spec/arith-context.test.sh                       81
spec/arith.test.sh                              631
spec/array-compat.test.sh                        96
spec/array.test.sh                              637
spec/assign-deferred.test.sh                    105
spec/assign-dialects.test.sh                     59
spec/assign-extended.test.sh                    810
spec/assign.test.sh                             672
spec/assoc.test.sh                              631
spec/assoc-zsh.test.sh                           53
spec/background.test.sh                         182
spec/ble-features.test.sh                       644
spec/ble-idioms.test.sh                         168
spec/blog1.test.sh                               95
spec/blog2.test.sh                               49
spec/blog-other1.test.sh                         55
spec/brace-expansion.test.sh                    441
spec/bugs.test.sh                               155
spec/builtin-bash.test.sh                       346
spec/builtin-bracket.test.sh                    564
spec/builtin-completion.test.sh                 414
spec/builtin-dirs.test.sh                       205
spec/builtin-eval-source.test.sh                249
spec/builtin-getopts.test.sh                    318
spec/builtin-io.test.sh                         751
spec/builtin-printf.test.sh                     831
spec/builtins2.test.sh                          233
spec/builtin-special.test.sh                    111
spec/builtins.test.sh                           420
spec/builtin-times.test.sh                       15
spec/builtin-trap.test.sh                       410
spec/builtin-vars.test.sh                       710
spec/case_.test.sh                              165
spec/command-parsing.test.sh                     63
spec/command-sub.test.sh                        272
spec/command_.test.sh                           192
spec/comments.test.sh                            11
spec/dbracket.test.sh                           436
spec/dparen.test.sh                             202
spec/empty-bodies.test.sh                        24
spec/errexit-oil.test.sh                        755
spec/errexit.test.sh                            393
spec/exit-status.test.sh                        238
spec/explore-parsing.test.sh                     42
spec/extglob-files.test.sh                      391
spec/extglob-match.test.sh                      400
spec/fatal-errors.test.sh                       186
spec/for-expr.test.sh                            92
spec/func-parsing.test.sh                       105
spec/glob.test.sh                               396
spec/here-doc.test.sh                           365
spec/if_.test.sh                                 56
spec/interactive.test.sh                        216
spec/introspect.test.sh                         237
spec/let.test.sh                                 23
spec/loop.test.sh                               397
spec/nameref.test.sh                            642
spec/nix-idioms.test.sh                         151
spec/nocasematch-match.test.sh                   69
spec/oil-array.test.sh                           86
spec/oil-assign.test.sh                         374
spec/oil-bin.test.sh                             17
spec/oil-blocks.test.sh                         330
spec/oil-bugs.test.sh                            22
spec/oil-builtin-error.test.sh                  423
spec/oil-builtin-pp.test.sh                      46
spec/oil-builtin-process.test.sh                 85
spec/oil-builtins.test.sh                       569
spec/oil-command-sub.test.sh                    108
spec/oil-config.test.sh                         100
spec/oil-demo.test.sh                            59
spec/oil-expr-sub.test.sh                        40
spec/oil-expr.test.sh                           866
spec/oil-funcs-builtin.test.sh                  229
spec/oil-interactive.test.sh                     15
spec/oil-json.test.sh                           119
spec/oil-keywords.test.sh                        37
spec/oil-multiline.test.sh                      179
spec/oil-options.test.sh                        795
spec/oil-proc.test.sh                           252
spec/oil-regex.test.sh                          689
spec/oil-scope.test.sh                          748
spec/oil-slice-range.test.sh                    195
spec/oil-special-vars.test.sh                    48
spec/oil-string.test.sh                         344
spec/oil-tuple.test.sh                           44
spec/oil-user-feedback.test.sh                  209
spec/oil-var-sub.test.sh                         45
spec/oil-word-eval.test.sh                       81
spec/oil-xtrace.test.sh                         477
spec/osh-only.test.sh                            78
spec/parse-errors.test.sh                       192
spec/pipeline.test.sh                           166
spec/posix.test.sh                              146
spec/process-sub.test.sh                        205
spec/prompt.test.sh                             254
spec/quote.test.sh                              248
spec/redirect.test.sh                           654
spec/regex.test.sh                              344
spec/serialize.test.sh                          229
spec/shell-grammar.test.sh                      205
spec/sh-func.test.sh                            150
spec/sh-options.test.sh                         694
spec/sh-usage.test.sh                            90
spec/smoke.test.sh                              123
spec/strict-options.test.sh                     249
spec/subshell.test.sh                            23
spec/tea-func.test.sh                           288
spec/tilde.test.sh                              172
spec/toysh-posix.test.sh                        347
spec/toysh.test.sh                              144
spec/type-compat.test.sh                         80
spec/var-num.test.sh                             39
spec/var-op-bash.test.sh                        264
spec/var-op-len.test.sh                         219
spec/var-op-patsub.test.sh                      291
spec/var-op-slice.test.sh                       315
spec/var-op-strip.test.sh                       357
spec/var-op-test.test.sh                        339
spec/var-ref.test.sh                            485
spec/vars-bash.test.sh                           27
spec/vars-special.test.sh                       456
spec/var-sub-quote.test.sh                      312
spec/var-sub.test.sh                             74
spec/word-eval.test.sh                           62
spec/word-split.test.sh                         387
spec/xtrace.test.sh                             344

35,170 lines in 130 files  

Gold Tests

Another suite that tests shells "from the outside". Instead of making explicit assertions, we verify that OSH behaves like bash.

test/gold/abuild.sh                               9
test/gold/and-or.sh                              27
test/gold/big-here-doc.sh                     1,531
test/gold/case-in-subshell.sh                    37
test/gold/char-class.sh                          33
test/gold/command-sub-2.sh                       12
test/gold/command-sub.sh                         39
test/gold/comments.sh                            22
test/gold/complex-here-docs.sh                  192
test/gold/configure-bug.sh                       13
test/gold/de1.sh                                 24
test/gold/declare.sh                             28
test/gold/dollar-sq.sh                           42
test/gold/echo-e.sh                              36
test/gold/errexit-confusion.sh                  139
test/gold/errexit.sh                            121
test/gold/export.sh                              65
test/gold/glob.sh                                26
test/gold/lineno.sh                              23
test/gold/nix.sh                                 32
test/gold/readlink.sh                            61
test/gold/readonly.sh                            24
test/gold/scope.sh                              126
test/gold/strip-op-char-class.sh                  6
test/gold/word-eval.sh                           57

2,725 lines in 26 files  

Language Unit Tests

core/alloc_test.py                               60
core/completion_test.py                         798
core/comp_ui_test.py                            179
core/process_test.py                            196
core/pyutil_test.py                              19
core/state_test.py                              316
core/ui_test.py                                  32
core/util_test.py                                25
frontend/args_test.py                           232
frontend/id_kind_def_test.py                     97
frontend/lexer_def_test.py                      436
frontend/lexer_gen_test.py                       51
frontend/lexer_test.py                           45
frontend/match_test.py                           56
frontend/reader_test.py                          49
native/fanos_test.py                            246
native/fastlex_test.py                           98
native/libc_test.py                             304
native/line_input_test.py                        27
native/posix_test.py                            184
oil_lang/builtin_oil_test.py                    135
oil_lang/expr_parse_test.py                     144
oil_lang/objects_test.py                         20
osh/arith_parse_test.py                         179
osh/bool_parse_test.py                          122
osh/braces_test.py                              187
osh/builtin_bracket_test.py                      29
osh/builtin_comp_test.py                         19
osh/builtin_lib_test.py                          93
osh/builtin_misc_test.py                         48
osh/cmd_eval_test.py                             74
osh/cmd_parse_test.py                         1,264
osh/glob_test.py                                163
osh/history_test.py                             124
osh/prompt_test.py                               54
osh/split_test.py                               169
osh/string_ops_test.py                          133
osh/word_compile_test.py                         16
osh/word_eval_test.py                           144
osh/word_parse_test.py                          561
osh/word_test.py                                101

7,229 lines in 42 files  

Other Unit Tests

asdl/format_test.py                              47
asdl/front_end_test.py                          141
asdl/gen_python_test.py                         153
asdl/typed_arith_parse_test.py                  227
build/app_deps_test.py                           27
pylib/os_path_test.py                            59
pylib/path_stat_test.py                          24
test/sh_spec_test.py                            138
test/wild_report_test.py                         29
tools/osh2oil_test.py                            79

924 lines in 11 files  

mycpp Translator

This prototype uses the MyPy frontend to translate statically-typed Python to C++. The generated code calls a small runtime which implements things like List[T], Dict[K, V], and Python's len().

mycpp/const_pass.py                             526
mycpp/cppgen_pass.py                          2,602
mycpp/crash.py                                   37
mycpp/debug_pass.py                             459
mycpp/format_strings.py                         110
mycpp/mycpp_main.py                             346
mycpp/mylib.py                                  148
mycpp/pass_state.py                              54
mycpp/util.py                                    12

4,294 lines in 10 files  

mycpp Test Data

Small Python examples that translate to C++, compile, and run.

mycpp/examples/asdl_generated.py                 62
mycpp/examples/cartesian.py                      75
mycpp/examples/cgi.py                            59
mycpp/examples/classes.py                       141
mycpp/examples/containers.py                    162
mycpp/examples/control_flow.py                  110
mycpp/examples/escape.py                         67
mycpp/examples/fib_iter.py                       54
mycpp/examples/fib_recursive.py                  49
mycpp/examples/files.py                          53
mycpp/examples/length.py                        106
mycpp/examples/lexer_main.py                     48
mycpp/examples/loops.py                         177
mycpp/examples/modules.py                        78
mycpp/examples/parse.py                         232
mycpp/examples/pgen2_demo.py                    106
mycpp/examples/scoped_resource.py               176
mycpp/examples/strings.py                        61
mycpp/examples/test_cast.py                      59
mycpp/examples/test_conditional.py               67
mycpp/examples/test_hoist.py                     61
mycpp/examples/test_switch.py                    42
mycpp/examples/tuple_return_value.py             53
mycpp/examples/varargs.py                        84

2,182 lines in 25 files  

Zephyr ASDL

A DSL for algebraic data types, borrowed from Python. Oil is the most strongly typed Bourne shell implementation!

asdl/asdl_.py                                   253
asdl/format.py                                  494
asdl/front_end.py                               488
asdl/gen_cpp.py                                 622
asdl/gen_python.py                              529
asdl/pybase.py                                   63
asdl/runtime.py                                  41
asdl/tool.py                                    280
asdl/visitor.py                                 134

2,904 lines in 10 files  

pgen2 Parser Generator

An LL(1) parser generator used to parse Oil expressions. Also borrowed from CPython.

pgen2/driver.py                                 101
pgen2/grammar.py                                222
pgen2/parse.py                                  209
pgen2/pgen.py                                   428
pgen2/pnode.py                                   29
pgen2/tokenize.py                               579
pgen2/token.py                                  149

1,717 lines in 8 files  

Other Code Generators

In order to make Oil statically typed, we had to abandon Python reflection and use C++ source code generation instead. The lexer, flag definitions, and constants can be easily compiled to C++.

core/optview_gen.py                              89
frontend/consts_gen.py                          551
frontend/flag_gen.py                            485
frontend/lexer_gen.py                           495
frontend/option_gen.py                          115
oil_lang/grammar_gen.py                         226
osh/arith_parse_gen.py                          101

2,062 lines in 8 files  

Build Automation

build/actions.sh                                220
build/app_deps.py                               152
build/c_module_srcs.py                           90
build/c_module_toc.py                            51
build/codegen.sh                                140
build/common.sh                                  69
build/compile.sh                                421
build/cpython_defs.py                           504
build/cpython-defs.sh                           347
build/dev.sh                                    442
build/dev-shell.sh                               42
build/doc.sh                                    402
build/hello.mk                                   51
build/make_zip.py                                53
build/mycpp.sh                                  316
build/native_graph.py                           104
build/native.sh                                 120
build/native-steps.sh                           185
build/oil.mk                                     60
build/opy.mk                                     49
build/prepare.sh                                 27
build/setup_fanos.py                             11
build/setup_fastlex.py                           18
build/setup_line_input.py                        13
build/setup_posix.py                             19
build/setup.py                                   11
build/slice-2020.sh                             121
build/test.sh                                    92
build/translate.sh                              184
build/with-clang.sh                               8
configure                                       395
install                                          87
Makefile                                        199
portable-rules.mk                                88

5,091 lines in 35 files  

Test Automation

test/alpine.sh                                  277
test/arena.sh                                    60
test/ble.sh                                      51
test/common.sh                                  121
test/common-test.sh                              22
test/cpp-unit.sh                                 81
test/gentoo.sh                                   83
test/gold.sh                                    163
test/group-session.sh                           127
test/lint.sh                                    331
test/nix.sh                                      24
test/oil-runtime-errors.sh                      159
test/opyc.sh                                    126
test/osh2oil.sh                               1,370
test/oshc-deps.sh                               104
test/osh-usage.sh                               213
test/other.sh                                    34
test/parse-errors.sh                          1,221
test/py3_parse.py                                64
test/py3_parse.sh                                37
test/py_stdlib.sh                                32
test/report.R                                    54
test/runtime-errors.sh                        1,111
test/shebang.sh                                  26
test/sh_spec.py                               1,267
test/signal-state.sh                             96
test/smoosh_import.py                            31
test/smoosh.sh                                  213
test/spec-alpine.sh                             151
test/spec-bin.sh                                242
test/spec-common.sh                              49
test/spec-cpp.sh                                379
test/spec_lib.py                                188
test/spec-runner.sh                             411
test/spec.sh                                  1,081
test/stateful.sh                                216
test/syscall.py                                 219
test/syscall.sh                                 415
test/tsv-lib.sh                                  37
test/unit.sh                                    247
test/vagrant.sh                                  54
test/wild_report.py                             802
test/wild-runner.sh                             189
test/wild.sh                                    719

12,897 lines in 45 files  

Release Automation

devtools/release-history.sh                     207
devtools/release-native.sh                      101
devtools/release-note.sh                        225
devtools/release.sh                             990
devtools/release-version.sh                     826

2,349 lines in 6 files  

Benchmarks

benchmarks/alloclog.py                           41
benchmarks/alloclog.sh                           95
benchmarks/auto.sh                              175
benchmarks/awk-python.sh                         97
benchmarks/builtin-io.sh                         60
benchmarks/common.R                              89
benchmarks/common.sh                             89
benchmarks/compute.sh                           529
benchmarks/fake_libc.py                          13
benchmarks/gperftools.sh                         60
benchmarks/id.sh                                400
benchmarks/__init__.py                            0
benchmarks/micro.sh                              45
benchmarks/mycpp.sh                              83
benchmarks/osh-parser.sh                        456
benchmarks/osh-runtime.sh                       364
benchmarks/ovm-build.sh                         475
benchmarks/parse-help.sh                         59
benchmarks/perf.sh                              165
benchmarks/pypy.sh                               42
benchmarks/pytrace.py                            92
benchmarks/pytrace.sh                           184
benchmarks/report.R                             707
benchmarks/report.sh                            127
benchmarks/startup.sh                           165
benchmarks/systemtap.sh                          75
benchmarks/time_.py                             193
benchmarks/time-test.sh                         260
benchmarks/uftrace_plugin.py                     95
benchmarks/uftrace.sh                           143
benchmarks/virtual_memory.py                    106
benchmarks/vm-baseline.sh                       149

5,633 lines in 33 files  

Metrics

metrics/bytecode.R                              388
metrics/bytecode.sh                             142
metrics/cpython-defs.R                           72
metrics/native-code.R                            88
metrics/native-code.sh                          119
metrics/source-code.sh                          459
metrics/tarball.sh                              132

1,400 lines in 8 files  

Generated Python Code

For the Python App Bundle.

_devbuild/gen/arg_types.py                      261
_devbuild/gen/demo_lib_asdl.py                  201
_devbuild/gen/find_asdl.py                      935
_devbuild/gen/find_nt.h                          13
_devbuild/gen/find_nt.py                         10
_devbuild/gen/grammar_nt.h                       93
_devbuild/gen/grammar_nt.py                      90
_devbuild/gen/help_.py                          210
_devbuild/gen/hnode_asdl_debug.py                 6
_devbuild/gen/hnode_asdl.py                     102
_devbuild/gen/id.h                              382
_devbuild/gen/id_kind_asdl.py                   864
_devbuild/gen/id_kind.py                        492
_devbuild/gen/__init__.py                         0
_devbuild/gen/option_asdl.py                    356
_devbuild/gen/osh-lex.h                      29,037
_devbuild/gen/osh-types.h                        32
_devbuild/gen/runtime_asdl_debug.py              37
_devbuild/gen/runtime_asdl.py                 2,211
_devbuild/gen/shared_variant_asdl.py            326
_devbuild/gen/syntax_asdl_debug.py              169
_devbuild/gen/syntax_asdl.py                  9,087
_devbuild/gen/typed_arith_asdl.py               576
_devbuild/gen/typed_demo_asdl.py                955
_devbuild/gen/types_asdl_debug.py                 3
_devbuild/gen/types_asdl.py                     102

46,550 lines in 27 files  

Tools

tools/deps.py                                   152
tools/osh2oil.py                              1,343
tools/readlink.py                                28

1,523 lines in 4 files  

Doc Tools

doctools/cmark.py                               376
doctools/doc_html.py                            106
doctools/html_head.py                            85
doctools/html_lib.py                             20
doctools/make_help.py                           465
doctools/oil_doc.py                             554
doctools/spelling.py                            142
doctools/split_doc.py                           151
lazylex/html.py                                 416

2,315 lines in 10 files  

Web

web/ajax.js                                     149
web/osh-to-oil.js                                79
web/table/csv2html.py                           397
web/table/table-sort.js                         433

1,058 lines in 5 files