aboutsummaryrefslogtreecommitdiff
path: root/driver/lexer.go (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Update godocRyo Nihei2021-10-051-8/+14
|
* Remove the ModeName and KindName fields from the driver.Token structRyo Nihei2021-10-031-40/+1
|
* Define a lexical specification interfaceRyo Nihei2021-09-111-127/+65
|
* Remove --debug option from the lex commandRyo Nihei2021-09-081-35/+0
|
* Add lexeme positions to tokensRyo Nihei2021-08-071-37/+85
| | | | close #1
* Change APIsRyo Nihei2021-08-011-68/+69
| | | | | | | | | | | | | | | | | | | | | | Change fields of tokens, results of lexical analysis, as follows: - Rename: mode -> mode_id - Rename: kind_id -> mode_kind_id - Add: kind_id The kind ID is unique across all modes, but the mode kind ID is unique only within a mode. Change fields of a transition table as follows: - Rename: initial_mode -> initial_mode_id - Rename: modes -> mode_names - Rename: kinds -> kind_names - Rename: specs[].kinds -> specs[].kind_names - Rename: specs[].dfa.initial_state -> specs[].dfa.initial_state_id Change public types defined in the spec package as follows: - Rename: LexModeNum -> LexModeID - Rename: LexKind -> LexKindName - Add: LexKindID - Add: StateID
* Add unique kind IDs to tokensRyo Nihei2021-08-011-5/+12
|
* Support passive mode transitionRyo Nihei2021-06-101-18/+30
|
* Rename fields of driver.TokenRyo Nihei2021-05-131-23/+22
|
* Add --compression-level option to compile commandRyo Nihei2021-05-111-6/+22
| | | | --compression-level specifies a compression level. The default value is 2.
* Fix a text representation of an error tokenRyo Nihei2021-05-111-19/+48
| | | | This commit fixes a bug that caused the second and subsequent characters of the text representation of an error token to be missing.
* Update README and godocRyo Nihei2021-05-101-7/+23
|
* Add --break-on-error option to lex commandRyo Nihei2021-05-081-0/+1
| | | | As you use --break-on-error option, break lexical analysis with exit status 1 immediately when an error token appears.
* Add CLI optionsRyo Nihei2021-05-081-15/+15
|
* Change type of acceping_states to sliceRyo Nihei2021-05-071-2/+2
|
* Add transition table compressorRyo Nihei2021-05-071-6/+12
|
* Remove Peek* functionsRyo Nihei2021-05-051-26/+0
|
* Add lex modeRyo Nihei2021-05-041-36/+99
| | | | | | | | | | lex mode is a feature that separates transition tables per each mode. The lexer starts from an initial state indicated by `initial_state` field and transitions between modes according to `push` and `pop` fields. The initial state will always be `default`. Currently, maleeni doesn't provide the ability to change the initial state. You can specify the modes of each lex entry using `modes` field. When the mode isn't indicated explicitly, the entries have `default` mode.
* Generate an invalid token from incompleted input.Ryo Nihei2021-05-021-0/+5
| | | | When the lexer's buffer has unaccepted data and reads the EOF, the lexer treats the buffered data as an invalid token.
* Add validation of lexical specs and improve error messagesRyo Nihei2021-04-171-1/+1
|
* Print the result of the lex command in JSON formatRyo Nihei2021-04-061-20/+66
| | | | | * Print the result of the lex command in JSON format. * Print the EOF token.
* Add logging to lex commandRyo Nihei2021-02-161-3/+58
| | | | | lex command writes logs out to the maleeni-lex.log file. When you generate a lexer using driver.NewLexer(), you can choose whether the lexer writes logs or not.
* Add types of lexical specificationsRyo Nihei2021-02-161-14/+16
| | | | APIs of compiler and driver packages use these types. Because CompiledLexSpec struct a lexer takes has kind names of lexical specification entries, the lexer sets them to tokens.
* Add driverRyo Nihei2021-02-141-0/+162
The driver takes a DFA and an input text and generates a lexer. The lexer tokenizes the input text according to the lexical specification that the DFA expresses.
generated by cgit v1.2.3 (git 2.50.1) at 2025-08-16 06:50:40 +0000