aboutsummaryrefslogtreecommitdiff
path: root/compiler/dfa.go (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Use new parser and DFA compilerRyo Nihei2021-12-101-139/+0
|
* Change APIsRyo Nihei2021-08-011-9/+9
| | | | | | | | | | | | | | | | | | | | | | Change fields of tokens, results of lexical analysis, as follows: - Rename: mode -> mode_id - Rename: kind_id -> mode_kind_id - Add: kind_id The kind ID is unique across all modes, but the mode kind ID is unique only within a mode. Change fields of a transition table as follows: - Rename: initial_mode -> initial_mode_id - Rename: modes -> mode_names - Rename: kinds -> kind_names - Rename: specs[].kinds -> specs[].kind_names - Rename: specs[].dfa.initial_state -> specs[].dfa.initial_state_id Change public types defined in the spec package as follows: - Rename: LexModeNum -> LexModeID - Rename: LexKind -> LexKindName - Add: LexKindID - Add: StateID
* Fix the initial state numberRyo Nihei2021-05-191-1/+5
| | | | Since 0 represents an invalid value in a transition table, assign a number greater than or equal to 1 to states.
* Change type of acceping_states to sliceRyo Nihei2021-05-071-2/+6
|
* Add transition table compressorRyo Nihei2021-05-071-7/+9
|
* Improve performance of the symbolPositionSetRyo Nihei2021-05-041-6/+6
| | | | | | | | | | When using a map to represent a set, performance degrades due to the increased number of calls of runtime.mapassign. Especially when the number of symbols is large, as in compiling a pattern that contains character properties like \p{Letter}, adding elements to the set alone may take several tens of seconds of CPU time. Therefore, this commit solves this problem by changing the representation of the set from map to array.
* Add types of lexical specificationsRyo Nihei2021-02-161-8/+4
| | | | APIs of compiler and driver packages use these types. Because CompiledLexSpec struct a lexer takes has kind names of lexical specification entries, the lexer sets them to tokens.
* Add dot symbol matching any single characterRyo Nihei2021-02-141-4/+6
| | | | | | | | | The dot symbol matches any single character. When the dot symbol appears, the parser generates an AST matching all of the well-formed UTF-8 byte sequences. Refelences: * https://www.unicode.org/versions/Unicode13.0.0/ch03.pdf#G7404 * Table 3-6. UTF-8 Bit Distribution * Table 3-7. Well-Formed UTF-8 Byte Sequences
* Add compilerRyo Nihei2021-02-141-0/+131
The compiler takes a lexical specification expressed by regular expressions and generates a DFA accepting the tokens. Operators that you can use in the regular expressions are concatenation, alternation, repeat, and grouping.