Why Sponsor Oils? | source | all docs for version 0.21.0 | all versions | oilshell.org

Warning: Work in progress! Leave feedback on Zulip or Github if you'd like this doc to be updated.

JSON / J8 Notation

This chapter in the Oils Reference describes JSON, and its J8 Notation superset.

See the J8 Notation doc for more background. This doc is a quick reference, not the official spec.

J8 Strings

J8 strings are an upgrade of JSON strings that solve the JSON-Unix Mismatch.

That is, Unix deals with byte strings, but JSON can't represent byte strings.

json-string "hi"

All JSON strings are valid J8 strings!

This is important. Encoders often emit JSON-style "" strings rather than u'' or b'' strings.

Example:

"hi μ \n"

json-escape \" \n \u1234

As a reminder, the backslash escapes valid in JSON strings are:

\" \\
\b \f \n \r \t
\u1234

Additional J8 escapes are valid in u'' and b'' strings, described below.

surrogate-pair \ud83e\udd26

JSON's \u1234 escapes can't represent code points above U+10000 or 2¹⁶, so JSON also has a "surrogate pair hack".

That is, there are special code points in the "surrogate range" that can be paired to represent larger numbers.

See the Surrogate Pair Blog Post for an example:

"\ud83e\udd26"

Because JSON strings are valid J8 strings, surrogate pairs are also part of J8 notation. Decoders must accept them, but encoders should avoid them.

You can emit u'\u{1f926}' or b'\u{1f926}' instead of "\ud83\udd26".

u-prefix u'hi'

A type of J8 string.

u'hi μ \n'

It's never necessary to emit, but it can be used to express that a string is valid Unicode. JSON strings can represent strings that aren't Unicode because they may contain surrogate halves.

In contrast, u'' strings can only have escapes like \u{1f926}, with no surrogate pairs or halves.

• The encoded bytes must be valid UTF-8, like JSON strings.
• The decoded bytes must be valid UTF-8, unlike JSON strings.

Escaping:

• u'' strings may not contain \u1234 escapes. They must be \u{1234}, \u{1f926}
• They may not contain \yff escapes, because those would represent a string that's not UTF-8 or Unicode.
• Surrogate pairs are never necessary in u'' or b'' strings. Use the longer form \u{1f926}.
• You can always emit literal UTF-8, so \u{1f926} escapes aren't strictly necessary. Decoders must accept these escapes.
• A literal single quote is escaped with \'
  • Decoders still accept \", but encoders don't emit it.

b-prefix b'hi'

Another J8 string. These b'' strings are identical to u'' strings, but they can also \yff escapes.

Examples:

b'hi μ \n'
b'this isn\'t a valid unicode string \yff\fe \u{3bc}'

j8-escape\u{1f926} \yff

To summarize, the valid J8 escapes are:

\'
\yff   # only valid in b'' strings
\u{3bc} \u{1f926} etc.

no-prefix 'hi'

Single-quoted strings without a u or b prefix are implicitly u''.

u'hi μ \n'  
 'hi μ \n'  # same as above, no \yff escapes accepted

They should be avoided in contexts where "" strings may also appear, because it's easy to confuse single quotes and double quotes.

JSON8

JSON8 is JSON with 4 more things allowed:

1. J8 strings in addition to JSON strings
2. Comments
3. Unquoted keys (TODO)
4. Trailing commas (TODO)

json8-num

Decoding detail, specific to Oils:

If there's a decimal point or e-10 suffix, then it's decoded into YSH Float. Otherwise it's a YSH Int.

42       # decoded to Int
42.0     # decoded to Float
42e1     # decoded to Float
42.0e1   # decoded to Float

json8-str

JSON8 strings are exactly J8 strings:

"hi 🤦 \u03bc"
u'hi 🤦 \u{3bc}'
b'hi 🤦 \u{3bc} \yff'

json8-list

Like JSON lists, but can have trailing comma. Examples:

[42, 43]
[42, 43,]   # same as above

json8-dict

Like JSON "objects", but:

• Can have trailing comma.
• Can have unquoted keys, as long as they're an identifier.

Examples:

{"json8": "message"}
{json8: "message"}     # same as above
{json8: "message",}    # same as above

json8-comment

End-of-line comments in the same style as shell:

{"json8": "message"}   # comment

TSV8

These are the J8 Primitives (Bool, Int, Float, Str), separated by tabs.

column-attrs

!tsv8    name    age
!type    Str     Int
!other   x       y
         Alice   42
         Bob     25

column-types

The primitives:

• Bool
• Int
• Float
• Str

Note: Can null be in all cells? Maybe except Bool?

It can stand in for NA?