Overview of Oil Code

category total lines num files
OSH (and common libraries) 36,814 77
Oil Language (and Tea) 4,843 11
Code Borrowed from Python's stdlib 215 4
Spec Tests 34,188 127
Gold Tests 952 20
Language Unit Tests 7,225 42
Other Unit Tests 920 11
mycpp Translator 4,294 10
mycpp Test Data 2,182 25
Zephyr ASDL 2,904 10
pgen2 Parser Generator 1,717 8
Other Code Generators 2,062 8
Build Automation 4,847 31
Test Automation 12,657 42
Release Automation 2,117 5
Benchmarks 5,634 33
Metrics 1,398 8
Generated Python Code 46,570 27
Tools 1,523 4
Doc Tools 2,173 9
Web 1,058 5

Related Documents

The README for oilshell/oil has another overview of the repository.

OSH (and common libraries)

This is the input to the translators, written in statically-typed Python. Note that bash is at least 140K lines of code, and OSH implements a large part of bash and more.

bin/oil.py                                      360
core/alloc.py                                   133
core/ansi.py                                     17
core/completion.py                            1,261
core/comp_ui.py                                 583
core/dev.py                                     597
core/error.py                                   183
core/executor.py                                548
core/main_loop.py                               352
core/optview.py                                  57
core/oven.py                                    573
core/process.py                               1,501
core/pyerror.py                                  58
core/pyos.py                                    296
core/pyutil.py                                  247
core/runtime.asdl                               193
core/shell_native.py                            491
core/shell.py                                   695
core/state.py                                 2,124
core/ui.py                                      381
core/util.py                                     93
core/vm.py                                      209
frontend/args.py                                673
frontend/builtin_def.py                         155
frontend/consts.py                              321
frontend/flag_def.py                            318
frontend/flag_spec.py                           380
frontend/id_kind_def.py                         638
frontend/lexer_def.py                         1,012
frontend/lexer.py                               330
frontend/location.py                             94
frontend/match.py                               257
frontend/option_def.py                          389
frontend/parse_lib.py                           458
frontend/py_reader.py                            81
frontend/reader.py                              136
frontend/signal_def.py                           43
frontend/syntax_abbrev.py                       144
frontend/syntax.asdl                            576
frontend/typed_args.py                           63
frontend/types.asdl                              40
mycpp/mylib.pyi                                  72
native/fanos.c                                  294
native/fastlex.c                                239
native/fastlex.pyi                               13
native/libc.c                                   402
native/libc.pyi                                  11
native/posix_.pyi                               211
osh/arith_parse.py                              184
osh/bool_parse.py                               269
osh/bool_stat.py                                136
osh/braces.py                                   498
osh/builtin_assign.py                           514
osh/builtin_bracket.py                          280
osh/builtin_comp.py                             501
osh/builtin_lib.py                               94
osh/builtin_meta.py                             403
osh/builtin_misc.py                             844
osh/builtin_printf.py                           477
osh/builtin_process.py                          539
osh/builtin_pure.py                             670
osh/cmd_eval.py                               1,772
osh/cmd_parse.py                              2,311
osh/glob_.py                                    498
osh/history.py                                  184
osh/prompt.py                                   300
osh/sh_expr_eval.py                           1,047
osh/split.py                                    305
osh/string_ops.py                               454
osh/tdop.py                                     333
osh/word_compile.py                             257
osh/word_eval.py                              2,297
osh/word_parse.py                             1,838
osh/word_.py                                    817
qsn_/qsn_native.py                               75
qsn_/qsn.py                                     615

36,814 lines in 77 files  

Oil Language (and Tea)

oil_lang/builtin_oil.py                         338
oil_lang/expr_eval.py                           806
oil_lang/expr_parse.py                          335
oil_lang/expr_to_ast.py                       1,536
oil_lang/funcs_builtin.py                       440
oil_lang/grammar.pgen2                          615
oil_lang/objects.py                             290
oil_lang/regex_translate.py                     262
tea/tea_eval.py                                 122
tea/tea_main.py                                  99

4,843 lines in 11 files  

Code Borrowed from Python's stdlib

pylib/cgi.py                                     19
pylib/os_path.py                                169
pylib/path_stat.py                               27

215 lines in 4 files  

Spec Tests

A comprehensive test suite that compares OSH against other shells. If OSH passes these tests in BOTH Python and C++, it means that the translation works.

spec/alias.test.sh                              536
spec/append.test.sh                             291
spec/arith-context.test.sh                       81
spec/arith.test.sh                              631
spec/array-compat.test.sh                        96
spec/array.test.sh                              637
spec/assign-deferred.test.sh                    105
spec/assign-dialects.test.sh                     59
spec/assign-extended.test.sh                    810
spec/assign.test.sh                             671
spec/assoc.test.sh                              631
spec/assoc-zsh.test.sh                           53
spec/background.test.sh                         150
spec/ble-features.test.sh                       631
spec/ble-idioms.test.sh                         159
spec/blog1.test.sh                               95
spec/blog2.test.sh                               49
spec/blog-other1.test.sh                         55
spec/brace-expansion.test.sh                    441
spec/bugs.test.sh                               103
spec/builtin-bash.test.sh                       346
spec/builtin-bracket.test.sh                    564
spec/builtin-completion.test.sh                 399
spec/builtin-dirs.test.sh                       177
spec/builtin-eval-source.test.sh                249
spec/builtin-getopts.test.sh                    318
spec/builtin-io.test.sh                         761
spec/builtin-printf.test.sh                     831
spec/builtins2.test.sh                          233
spec/builtin-special.test.sh                    111
spec/builtins.test.sh                           420
spec/builtin-times.test.sh                       15
spec/builtin-trap.test.sh                       406
spec/builtin-vars.test.sh                       710
spec/case_.test.sh                              165
spec/command-parsing.test.sh                     63
spec/command-sub.test.sh                        272
spec/command_.test.sh                           192
spec/comments.test.sh                            11
spec/dbracket.test.sh                           436
spec/dparen.test.sh                             202
spec/empty-bodies.test.sh                        24
spec/errexit-oil.test.sh                        675
spec/errexit.test.sh                            392
spec/exit-status.test.sh                        238
spec/explore-parsing.test.sh                     42
spec/extglob-files.test.sh                      391
spec/extglob-match.test.sh                      400
spec/for-expr.test.sh                            92
spec/func-parsing.test.sh                       105
spec/glob.test.sh                               396
spec/here-doc.test.sh                           365
spec/if_.test.sh                                 56
spec/interactive.test.sh                        216
spec/introspect.test.sh                         237
spec/let.test.sh                                 23
spec/loop.test.sh                               397
spec/nameref.test.sh                            642
spec/nix-idioms.test.sh                         151
spec/nocasematch-match.test.sh                   69
spec/oil-array.test.sh                           86
spec/oil-assign.test.sh                         326
spec/oil-bin.test.sh                             17
spec/oil-blocks.test.sh                         163
spec/oil-builtin-pp.test.sh                      46
spec/oil-builtin-process.test.sh                 85
spec/oil-builtins.test.sh                       554
spec/oil-builtin-try.test.sh                    140
spec/oil-command-sub.test.sh                    108
spec/oil-config.test.sh                         100
spec/oil-demo.test.sh                            59
spec/oil-expr-sub.test.sh                        40
spec/oil-expr.test.sh                           866
spec/oil-funcs-builtin.test.sh                  229
spec/oil-interactive.test.sh                     15
spec/oil-json.test.sh                           119
spec/oil-keywords.test.sh                        37
spec/oil-multiline.test.sh                      179
spec/oil-options.test.sh                        795
spec/oil-proc.test.sh                           252
spec/oil-regex.test.sh                          689
spec/oil-scope.test.sh                          748
spec/oil-slice-range.test.sh                    195
spec/oil-special-vars.test.sh                    66
spec/oil-string.test.sh                         344
spec/oil-tuple.test.sh                           44
spec/oil-user-feedback.test.sh                  209
spec/oil-var-sub.test.sh                         45
spec/oil-word-eval.test.sh                       81
spec/oil-xtrace.test.sh                         477
spec/osh-only.test.sh                            78
spec/parse-errors.test.sh                       192
spec/pipeline.test.sh                           166
spec/posix.test.sh                              146
spec/process-sub.test.sh                        205
spec/prompt.test.sh                             254
spec/quote.test.sh                              248
spec/redirect.test.sh                           654
spec/regex.test.sh                              344
spec/serialize.test.sh                          229
spec/shell-grammar.test.sh                      205
spec/sh-func.test.sh                            150
spec/sh-options.test.sh                         694
spec/sh-usage.test.sh                            94
spec/smoke.test.sh                              123
spec/special-vars.test.sh                       425
spec/strict-options.test.sh                     249
spec/subshell.test.sh                            23
spec/tea-func.test.sh                           288
spec/tilde.test.sh                              172
spec/toysh-posix.test.sh                        347
spec/toysh.test.sh                              144
spec/type-compat.test.sh                         80
spec/var-num.test.sh                             39
spec/var-op-bash.test.sh                        264
spec/var-op-len.test.sh                         219
spec/var-op-patsub.test.sh                      291
spec/var-op-slice.test.sh                       315
spec/var-op-strip.test.sh                       357
spec/var-op-test.test.sh                        339
spec/var-ref.test.sh                            485
spec/var-sub-quote.test.sh                      312
spec/var-sub.test.sh                             74
spec/word-eval.test.sh                           62
spec/word-split.test.sh                         387
spec/xtrace.test.sh                             344

34,188 lines in 127 files  

Gold Tests

Another suite that tests shells "from the outside". Instead of making explicit assertions, we verify that OSH behaves like bash.

test/gold/abuild.sh                               9
test/gold/and-or.sh                              27
test/gold/comments.sh                            22
test/gold/complex-here-docs.sh                  192
test/gold/configure-bug.sh                       13
test/gold/de1.sh                                 24
test/gold/declare.sh                             28
test/gold/dollar-sq.sh                           42
test/gold/echo-e.sh                              36
test/gold/errexit-confusion.sh                  139
test/gold/export.sh                              65
test/gold/glob.sh                                26
test/gold/lineno.sh                              23
test/gold/nix.sh                                 32
test/gold/readlink.sh                            61
test/gold/readonly.sh                            24
test/gold/scope.sh                              126
test/gold/strip-op-char-class.sh                  6
test/gold/word-eval.sh                           57

952 lines in 20 files  

Language Unit Tests

core/alloc_test.py                               60
core/completion_test.py                         798
core/comp_ui_test.py                            179
core/process_test.py                            196
core/pyutil_test.py                              19
core/state_test.py                              316
core/ui_test.py                                  32
core/util_test.py                                25
frontend/args_test.py                           232
frontend/id_kind_def_test.py                     97
frontend/lexer_def_test.py                      436
frontend/lexer_gen_test.py                       51
frontend/lexer_test.py                           45
frontend/match_test.py                           52
frontend/reader_test.py                          49
native/fanos_test.py                            246
native/fastlex_test.py                           98
native/libc_test.py                             304
native/line_input_test.py                        27
native/posix_test.py                            184
oil_lang/builtin_oil_test.py                    135
oil_lang/expr_parse_test.py                     144
oil_lang/objects_test.py                         20
osh/arith_parse_test.py                         179
osh/bool_parse_test.py                          122
osh/braces_test.py                              187
osh/builtin_bracket_test.py                      29
osh/builtin_comp_test.py                         19
osh/builtin_lib_test.py                          93
osh/builtin_misc_test.py                         48
osh/cmd_eval_test.py                             74
osh/cmd_parse_test.py                         1,264
osh/glob_test.py                                163
osh/history_test.py                             124
osh/prompt_test.py                               54
osh/split_test.py                               169
osh/string_ops_test.py                          133
osh/word_compile_test.py                         16
osh/word_eval_test.py                           144
osh/word_parse_test.py                          561
osh/word_test.py                                101

7,225 lines in 42 files  

Other Unit Tests

asdl/format_test.py                              47
asdl/front_end_test.py                          141
asdl/gen_python_test.py                         153
asdl/typed_arith_parse_test.py                  227
build/app_deps_test.py                           27
pylib/os_path_test.py                            59
pylib/path_stat_test.py                          24
test/sh_spec_test.py                            134
test/wild_report_test.py                         29
tools/osh2oil_test.py                            79

920 lines in 11 files  

mycpp Translator

This prototype uses the MyPy frontend to translate statically-typed Python to C++. The generated code calls a small runtime which implements things like List[T], Dict[K, V], and Python's len().

mycpp/const_pass.py                             526
mycpp/cppgen_pass.py                          2,602
mycpp/crash.py                                   37
mycpp/debug_pass.py                             459
mycpp/format_strings.py                         110
mycpp/mycpp_main.py                             346
mycpp/mylib.py                                  148
mycpp/pass_state.py                              54
mycpp/util.py                                    12

4,294 lines in 10 files  

mycpp Test Data

Small Python examples that translate to C++, compile, and run.

mycpp/examples/asdl_generated.py                 62
mycpp/examples/cartesian.py                      75
mycpp/examples/cgi.py                            59
mycpp/examples/classes.py                       141
mycpp/examples/containers.py                    162
mycpp/examples/control_flow.py                  110
mycpp/examples/escape.py                         67
mycpp/examples/fib_iter.py                       54
mycpp/examples/fib_recursive.py                  49
mycpp/examples/files.py                          53
mycpp/examples/length.py                        106
mycpp/examples/lexer_main.py                     48
mycpp/examples/loops.py                         177
mycpp/examples/modules.py                        78
mycpp/examples/parse.py                         232
mycpp/examples/pgen2_demo.py                    106
mycpp/examples/scoped_resource.py               176
mycpp/examples/strings.py                        61
mycpp/examples/test_cast.py                      59
mycpp/examples/test_conditional.py               67
mycpp/examples/test_hoist.py                     61
mycpp/examples/test_switch.py                    42
mycpp/examples/tuple_return_value.py             53
mycpp/examples/varargs.py                        84

2,182 lines in 25 files  

Zephyr ASDL

A DSL for algebraic data types, borrowed from Python. Oil is the most strongly typed Bourne shell implementation!

asdl/asdl_.py                                   253
asdl/format.py                                  494
asdl/front_end.py                               488
asdl/gen_cpp.py                                 622
asdl/gen_python.py                              529
asdl/pybase.py                                   63
asdl/runtime.py                                  41
asdl/tool.py                                    280
asdl/visitor.py                                 134

2,904 lines in 10 files  

pgen2 Parser Generator

An LL(1) parser generator used to parse Oil expressions. Also borrowed from CPython.

pgen2/driver.py                                 101
pgen2/grammar.py                                222
pgen2/parse.py                                  209
pgen2/pgen.py                                   428
pgen2/pnode.py                                   29
pgen2/tokenize.py                               579
pgen2/token.py                                  149

1,717 lines in 8 files  

Other Code Generators

In order to make Oil statically typed, we had to abandon Python reflection and use C++ source code generation instead. The lexer, flag definitions, and constants can be easily compiled to C++.

core/optview_gen.py                              89
frontend/consts_gen.py                          551
frontend/flag_gen.py                            485
frontend/lexer_gen.py                           495
frontend/option_gen.py                          115
oil_lang/grammar_gen.py                         226
osh/arith_parse_gen.py                          101

2,062 lines in 8 files  

Build Automation

build/actions.sh                                220
build/app_deps.py                               152
build/c_module_srcs.py                           90
build/c_module_toc.py                            51
build/codegen.sh                                140
build/common.sh                                  69
build/compile.sh                                421
build/cpython_defs.py                           504
build/cpython-defs.sh                           347
build/dev.sh                                    438
build/dev-shell.sh                               42
build/doc.sh                                    402
build/hello.mk                                   51
build/make_zip.py                                53
build/mycpp.sh                                  669
build/oil.mk                                     60
build/opy.mk                                     49
build/prepare.sh                                 27
build/setup_fanos.py                             11
build/setup_fastlex.py                           18
build/setup_line_input.py                        13
build/setup_posix.py                             19
build/setup.py                                   11
build/slice-2020.sh                             121
build/test.sh                                    92
build/with-clang.sh                               8
configure                                       395
install                                          87
Makefile                                        199
portable-rules.mk                                88

4,847 lines in 31 files  

Test Automation

test/alpine.sh                                  277
test/arena.sh                                    65
test/ble.sh                                      51
test/common.sh                                  137
test/cpp-unit.sh                                 78
test/gentoo.sh                                   83
test/gold.sh                                    200
test/interactive.py                             172
test/interactive.sh                              51
test/lint.sh                                    331
test/nix.sh                                      24
test/oil-runtime-errors.sh                      124
test/opyc.sh                                    142
test/opy.sh                                     121
test/osh2oil.sh                               1,364
test/oshc-deps.sh                               103
test/osh-usage.sh                               205
test/other.sh                                    34
test/parse-errors.sh                          1,224
test/py3_parse.py                                64
test/py3_parse.sh                                37
test/report.R                                    54
test/runtime-errors.sh                        1,039
test/shebang.sh                                  26
test/sh_spec.py                               1,423
test/smoosh_import.py                            31
test/smoosh.sh                                  213
test/spec-alpine.sh                             151
test/spec-bin.sh                                250
test/spec-common.sh                              49
test/spec-cpp.sh                                371
test/spec-runner.sh                             415
test/spec.sh                                  1,066
test/stdlib.sh                                   32
test/syscall.py                                 219
test/syscall.sh                                 415
test/unit.sh                                    248
test/vagrant.sh                                  54
test/wild_report.py                             802
test/wild-runner.sh                             193
test/wild.sh                                    719

12,657 lines in 42 files  

Release Automation

devtools/release-native.sh                      100
devtools/release-note.sh                        242
devtools/release.sh                             976
devtools/release-version.sh                     799

2,117 lines in 5 files  

Benchmarks

benchmarks/alloclog.py                           41
benchmarks/alloclog.sh                           96
benchmarks/auto.sh                              175
benchmarks/awk-python.sh                         97
benchmarks/builtin-io.sh                         60
benchmarks/common.R                              89
benchmarks/common.sh                             97
benchmarks/compute.sh                           524
benchmarks/fake_libc.py                          13
benchmarks/gperftools.sh                         60
benchmarks/id.sh                                400
benchmarks/__init__.py                            0
benchmarks/micro.sh                              45
benchmarks/mycpp.sh                              79
benchmarks/osh-parser.sh                        452
benchmarks/osh-runtime.sh                       364
benchmarks/ovm-build.sh                         469
benchmarks/parse-help.sh                         59
benchmarks/perf.sh                              165
benchmarks/pypy.sh                               42
benchmarks/pytrace.py                            92
benchmarks/pytrace.sh                           184
benchmarks/report.R                             707
benchmarks/report.sh                            127
benchmarks/startup.sh                           165
benchmarks/systemtap.sh                          75
benchmarks/time_.py                             193
benchmarks/time-test.sh                         275
benchmarks/uftrace_plugin.py                     95
benchmarks/uftrace.sh                           140
benchmarks/virtual_memory.py                    106
benchmarks/vm-baseline.sh                       148

5,634 lines in 33 files  

Metrics

metrics/bytecode.R                              388
metrics/bytecode.sh                             142
metrics/cpython-defs.R                           72
metrics/native-code.R                            88
metrics/native-code.sh                          119
metrics/source-code.sh                          457
metrics/tarball.sh                              132

1,398 lines in 8 files  

Generated Python Code

For the Python App Bundle.

_devbuild/gen/arg_types.py                      261
_devbuild/gen/demo_lib_asdl.py                  201
_devbuild/gen/find_asdl.py                      935
_devbuild/gen/find_nt.h                          13
_devbuild/gen/find_nt.py                         10
_devbuild/gen/grammar_nt.h                       93
_devbuild/gen/grammar_nt.py                      90
_devbuild/gen/help_.py                          209
_devbuild/gen/hnode_asdl_debug.py                 6
_devbuild/gen/hnode_asdl.py                     102
_devbuild/gen/id.h                              382
_devbuild/gen/id_kind_asdl.py                   864
_devbuild/gen/id_kind.py                        492
_devbuild/gen/__init__.py                         0
_devbuild/gen/option_asdl.py                    354
_devbuild/gen/osh-lex.h                      29,060
_devbuild/gen/osh-types.h                        32
_devbuild/gen/runtime_asdl_debug.py              37
_devbuild/gen/runtime_asdl.py                 2,211
_devbuild/gen/shared_variant_asdl.py            326
_devbuild/gen/syntax_asdl_debug.py              169
_devbuild/gen/syntax_asdl.py                  9,087
_devbuild/gen/typed_arith_asdl.py               576
_devbuild/gen/typed_demo_asdl.py                955
_devbuild/gen/types_asdl_debug.py                 3
_devbuild/gen/types_asdl.py                     102

46,570 lines in 27 files  

Tools

tools/deps.py                                   152
tools/osh2oil.py                              1,343
tools/readlink.py                                28

1,523 lines in 4 files  

Doc Tools

doctools/cmark.py                               376
doctools/doc_html.py                            106
doctools/html_head.py                            85
doctools/html_lib.py                             20
doctools/make_help.py                           465
doctools/oil_doc.py                             554
doctools/split_doc.py                           151
lazylex/html.py                                 416

2,173 lines in 9 files  

Web

web/ajax.js                                     149
web/osh-to-oil.js                                79
web/table/csv2html.py                           397
web/table/table-sort.js                         433

1,058 lines in 5 files