Lexer Hints: Disambiguating the Right Parenthesis

2017-12-16

Lexer Hints: How to Tokenize the Right Paren

Theory

Mosts make python's lexer non-regular?
- I guess it's not a single language anymore.
Hint Stack makes it context-sensitive
- Python's lexer is contest-sensitive
- Lua's lexer is context-free but not regular
- Lua's lexer is context-free but not regular

Review:

regular: regular expressions, not regxes.
- cannot match parens
context-free: can match parens
context-sensitive: stack, can match indentation
- https://en.wikipedia.org/wiki/Linear_bounded_automaton
unrestricted: can do anything.

In other words: instead of having potentially infinite tape on which to compute, computation is restricted to the portion of the tape containing the input plus the two tape squares holding the endmarkers.

what's an example of something that's unrestricted but not context-sensitive?
I think it's the C type-name / identifier problem.  The "lexer hack".
Because that requires arbitrary storage/

Why does theory matter? Because of the non-determinism

Lexer is HIGHLY STRUCTURED: modes and a hint stack. Maybe could generate it? Well Unread() is a problem. Unread at most one token.

Parser is structured too but kinda messy.