roland/xo-tokenizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

xo-tokenizer: doc fixes

Browse source
This commit is contained in:
Roland Conybeare 2025-06-25 21:45:24 -05:00
commit f93cb3719c
11 changed files with 105 additions and 19 deletions
Download patch file Download diff file Expand all files Collapse all files

8
docs/implementation.rst
View file

@ -29,8 +29,10 @@ Abstraction tower for *xo-tokenizer* components:
| tokenizer | |
+-----------------------------------------+ |
| scan_result | |
+-----------------+-----------------------+ buffer |
| token | tokenizer_error | |
+-----------------+-----------------------+ |
| tokentype | span | |
| | tokenizer_error | buffer |
| token +-----------------------+ |
| | input_state | |
+-----------------+-----------------------+ |
| tokentype | span | |
+-----------------+-----------------------+----------+