Why Sponsor Oils? | blog | oilshell.org

Oil Doesn't Confuse Flags and Files (Code and Data)

In last week's recap, I said I'd write a post tagged #real-problems when Oil addresses a problem with shell.

Here's another one: I just implemented shopt -u dashglob, which excludes files beginning with a dash (hyphen) from globs. This option addresses a decades-old problem with security implications.

Read on to see the problem, as well as the reasoning behind the simple solution.

A Typical Example

First, an attacker creates a file named -rf on the file system. Typical vectors are tarballs, git repos, and the like.

attacker$ touch -- -rf  # -- stops flag parsing, so -rf is a file

Then an admin can be fooled into recursively deleting a tree, rather than the intended current directory:

admin$ rm *  # the glob matches -rf and affects 'rm'

Here are two traditional solutions:

admin$ rm -- *  # tell rm to stop processing flags

admin$ rm ./*   # globbed files won't begin with a hyphen

However these solutions require "opting in" on every line. Even if you have the knowledge, you might not do it 100% of the time. Or your co-worker may not write scripts in this style.

In Oil, the common/short thing should be right thing, so it excludes -rf by default:

oil-admin$ rm *    # safe because the dashglob option is OFF

Usage

dashglob is Turned Off by Oil

This is a good time to review the difference between OSH and Oil.

• When you run bin/osh, the shell behaves compatibly, like POSIX and bash. A glob will include a file like -rf.
• When you run bin/oil, it's like running bin/osh with shopt -s oil:all. The binary is the same, but an option group is enabled.
  • The group oil:all disables the dashglob option, so globs will exclude files like -rf.

Other notes:

• The dashglob name is consistent with bash's dotglob option, which controls whether globs include files like .vimrc, .oilrc, etc.
• dashglob is also off in the less aggressive groups strict:all and oil:basic. I'll write more about them later.

What if I Want Globs to Include -rf?

There are a few ways to return all files. You can simply re-enable dashglob globally:

oil$ shopt -s dashglob    # turns it back on
oil$ rm -- *              # includes -rf, which is safe with --

To change the behavior for a single command, you will be able to pass shopt a block (not implemented):

shopt -s dashglob {
  rm -- *
}
rm *  # -rf excluded again

You can also use the more traditional solution:

oil$ rm ./*  # includes ./-rf because it doesn't start with -

Background

I've read about this issue dozens of times over the years. Here's a recent lobste.rs thread about it, where I linked David A. Wheeler's page Filenames and Pathnames in Shell: How to do it Correctly. That page recommends the explicit ./* to avoid the problem.

I knew I wanted Oil to address the problem without opting in, but I wasn't sure how. I filed issue 552 to keep track of it.

Over the next several weeks I got great feedback from Mateusz Czapliński, David A. Wheeler, and others. We considered a number of solutions, and ended up with this simple one. Appendix B explains why other solutions were rejected.

What Other Problems Can Be Fixed in Shell?

As I wrote in August, you can influence the Oil language! I take feedback from both experienced shell users and programmers who avoid shell.

So let us know if there are any other common shell problems that Oil should mitigate. Limitations of ShellCheck could be a source of inspiration.

Again, I've tagged these posts #real-problems. It's absolutely a goal for Oil to deal safely with untrusted data, including all the problems described in detail by Wheeler.

Related Work

• nullglob is also on in Oil, and addresses the find . -name *.jpg issue.
• I'm working on a new doc Simple Word Evaluation in Unix Shell, which describes shopt -s simple_word_eval. Again, the short thing should be the right thing.
• set -e / errexit is another hairy issue. I'd like users to test the solutions that Oil has so far. They still need work and documentation. See point #3 about the "ignored errexit problem" in this section of the last post, as well as the linked Reddit comments.
• CSTR is a proposal for a serialization format that solves more quoting problems. In Oil, you should be able to safely print paths to stdout and read them back in. They shouldn't be mangled in any way. The output should be text, not binary.

Appendix A: GLOBIGNORE= in bash

While discussing the -rf problem, I discovered that bash has a mechanism to omit such files:

GLOBIGNORE='-*'

Although I don't recall any script that does this. For example, I don't see it in a grep.app search for GLOBIGNORE.

But you may want to set GLOBIGNORE in your bash scripts if you deal with untrusted filenames.

I filed issue 609 to implement it in Oil. It will be useful for people who want to run the same program under Oil and bash.

It's labeled #help wanted, and would be a great way to start contributing to Oil. See this recent call for help for other ideas.

Appendix B: Assumptions That Oil Doesn't Make

I like the dashglob solution because it's consistent with dotglob and GLOBIGNORE, and it doesn't involve heuristics or false assumptions.

In particular, Oil doesn't assume:

• That it understands the flag syntax of each command. For example, does the command accept -- to indicate the end of flags?
• That it understands which commands wrap others. For example, sudo -- rm * is incorrect despite having --. The correct command is sudo -- rm -- *.
• That ./foo means the same thing as foo to every command. This is frequently true, but not always, e.g. in the case of tar.