blog | oilshell.org

Oil Doesn't Confuse Flags and Files (Code and Data)

2020-02-27

In last week's recap, I said I'd write a post tagged #real-problems when Oil addresses a problem with shell.

Here's another one: I just implemented shopt -u dashglob, which excludes files beginning with a dash (hyphen) from globs. This option addresses a decades-old problem with security implications.

Read on to see the problem, as well as the reasoning behind the simple solution.

Table of Contents
A Typical Example
Usage
dashglob is Turned Off by Oil
What if I Want Globs to Include -rf?
Background
What Other Problems Can Be Fixed in Shell?
Related Work
Appendix A: GLOBIGNORE= in bash
Appendix B: Assumptions That Oil Doesn't Make

A Typical Example

First, an attacker creates a file named -rf on the file system. Typical vectors are tarballs, git repos, and the like.

attacker$ touch -- -rf  # -- stops flag parsing, so -rf is a file

Then an admin can be fooled into recursively deleting a tree, rather than the intended current directory:

admin$ rm *  # the glob matches -rf and affects 'rm'

Here are two traditional solutions:

admin$ rm -- *  # tell rm to stop processing flags

admin$ rm ./*   # globbed files won't begin with a hyphen

However these solutions require "opting in" on every line. Even if you have the knowledge, you might not do it 100% of the time. Or your co-worker may not write scripts in this style.

In Oil, the common/short thing should be right thing, so it excludes -rf by default:

oil-admin$ rm *    # safe because the dashglob option is OFF

Usage

dashglob is Turned Off by Oil

This is a good time to review the difference between OSH and Oil.

Other notes:

What if I Want Globs to Include -rf?

There are a few ways to return all files. You can simply re-enable dashglob globally:

oil$ shopt -s dashglob    # turns it back on
oil$ rm -- *              # includes -rf, which is safe with --

To change the behavior for a single command, you will be able to pass shopt a block (not implemented):

shopt -s dashglob {
  rm -- *
}
rm *  # -rf excluded again

You can also use the more traditional solution:

oil$ rm ./*  # includes ./-rf because it doesn't start with -

Background

I've read about this issue dozens of times over the years. Here's a recent lobste.rs thread about it, where I linked David A. Wheeler's page Filenames and Pathnames in Shell: How to do it Correctly. That page recommends the explicit ./* to avoid the problem.

I knew I wanted Oil to address the problem without opting in, but I wasn't sure how. I filed issue 552 to keep track of it.

Over the next several weeks I got great feedback from Mateusz CzapliƄski, David A. Wheeler, and others. We considered a number of solutions, and ended up with this simple one. Appendix B explains why other solutions were rejected.

What Other Problems Can Be Fixed in Shell?

As I wrote in August, you can influence the Oil language! I take feedback from both experienced shell users and programmers who avoid shell.

So let us know if there are any other common shell problems that Oil should mitigate. Limitations of ShellCheck could be a source of inspiration.

Again, I've tagged these posts #real-problems. It's absolutely a goal for Oil to deal safely with untrusted data, including all the problems described in detail by Wheeler.

Related Work

Appendix A: GLOBIGNORE= in bash

While discussing the -rf problem, I discovered that bash has a mechanism to omit such files:

GLOBIGNORE='-*'

Although I don't recall any script that does this. For example, I don't see it in a grep.app search for GLOBIGNORE.

But you may want to set GLOBIGNORE in your bash scripts if you deal with untrusted filenames.

I filed issue 609 to implement it in Oil. It will be useful for people who want to run the same program under Oil and bash.

It's labeled #help wanted, and would be a great way to start contributing to Oil. See this recent call for help for other ideas.

Appendix B: Assumptions That Oil Doesn't Make

I like the dashglob solution because it's consistent with dotglob and GLOBIGNORE, and it doesn't involve heuristics or false assumptions.

In particular, Oil doesn't assume: