aboutsummaryrefslogtreecommitdiff
path: root/compiler/lexer_test.go (unfollow)
Commit message (Collapse)AuthorFilesLines
2021-02-14Add bracket expression matching specified characterRyo Nihei1-2/+30
The bracket expression matches any single character specified in it. In the bracket expression, the special characters like ., *, and so on are also handled as normal characters.
2021-02-14Add dot symbol matching any single characterRyo Nihei7-21/+201
The dot symbol matches any single character. When the dot symbol appears, the parser generates an AST matching all of the well-formed UTF-8 byte sequences. Refelences: * https://www.unicode.org/versions/Unicode13.0.0/ch03.pdf#G7404 * Table 3-6. UTF-8 Bit Distribution * Table 3-7. Well-Formed UTF-8 Byte Sequences
2021-02-14Add driverRyo Nihei2-0/+309
The driver takes a DFA and an input text and generates a lexer. The lexer tokenizes the input text according to the lexical specification that the DFA expresses.
2021-02-14Add compilerRyo Nihei9-0/+1268
The compiler takes a lexical specification expressed by regular expressions and generates a DFA accepting the tokens. Operators that you can use in the regular expressions are concatenation, alternation, repeat, and grouping.