roland/xo-alloc
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

xo-tokenizer: + missed doc files prev commit

Browse source
This commit is contained in:
Roland Conybeare 2025-06-25 21:45:44 -05:00
commit 0bd77b3c52
2 changed files with 58 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

29
xo-tokenizer/docs/input-state-class.rst Normal file
View file

@ -0,0 +1,29 @@
.. _input-state-class:
Input State
===========
Track detailed state of input stream to collect information useful for detailed error reporting
Context
-------
.. ditaa::
:--scale: 0.85
+-----------------------------------------+----------+
| tokenizer | |
+-----------------------------------------+ |
| scan_result | |
+-----------------+-----------------------+ |
| | tokenizer_error | buffer |
| token +-----------------------+ |
| |cBLU input_state | |
+-----------------+-----------------------+ |
| tokentype | span | |
+-----------------+-----------------------+----------+
.. code-block:: cpp
#include <xo/tokenizer/input_state.hpp>

29
xo-tokenizer/docs/scan-result-class.rst Normal file
View file

@ -0,0 +1,29 @@
.. _scan-result-class:
Scan Result
===========
Represent the result of a tokenizer scan call
Context
-------
.. ditaa::
:--scale: 0.85
+-----------------------------------------+----------+
| tokenizer | |
+-----------------------------------------+ |
|cBLU scan_result | |
+-----------------+-----------------------+ |
| | tokenizer_error | buffer |
| token +-----------------------+ |
| | input_state | |
+-----------------+-----------------------+ |
| tokentype | span | |
+-----------------+-----------------------+----------+
.. code-block:: cpp
#include <xo/tokenizer/input_state.hpp>