| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
| |
* Make the lexer treat ']' as an ordinary character in default mode
* Define values of the syntax error type that represents error information concretely
|
| |
|
|
|
| |
* Print the result of the lex command in JSON format.
* Print the EOF token.
|
| |
|
|
| |
[^a-z] matches any character that is not in the range a-z.
|
| |
|
|
| |
[a-z] matches any one character from a to z. The order of the characters depends on Unicode code points.
|
| |
|
|
|
| |
* a+ matches 'a' one or more times. This is equivalent to aa*.
* a? matches 'a' zero or one time.
|
| |
|
|
|
| |
lex command writes logs out to the maleeni-lex.log file.
When you generate a lexer using driver.NewLexer(), you can choose whether the lexer writes logs or not.
|
| |
|
|
| |
APIs of compiler and driver packages use these types. Because CompiledLexSpec struct a lexer takes has kind names of lexical specification entries, the lexer sets them to tokens.
|
| |
|
|
| |
The bracket expression matches any single character specified in it. In the bracket expression, the special characters like ., *, and so on are also handled as normal characters.
|
| |
|
|
|
|
|
|
|
| |
The dot symbol matches any single character. When the dot symbol appears, the parser generates an AST matching all of the well-formed UTF-8 byte sequences.
Refelences:
* https://www.unicode.org/versions/Unicode13.0.0/ch03.pdf#G7404
* Table 3-6. UTF-8 Bit Distribution
* Table 3-7. Well-Formed UTF-8 Byte Sequences
|
|
|
The driver takes a DFA and an input text and generates a lexer. The lexer tokenizes the input text according to the lexical specification that the DFA expresses.
|
