diff --git a/xo-tokenizer/docs/input-state-class.rst b/xo-tokenizer/docs/input-state-class.rst new file mode 100644 index 00000000..01d14267 --- /dev/null +++ b/xo-tokenizer/docs/input-state-class.rst @@ -0,0 +1,29 @@ + +.. _input-state-class: + +Input State +=========== + +Track detailed state of input stream to collect information useful for detailed error reporting + +Context +------- + +.. ditaa:: + :--scale: 0.85 + + +-----------------------------------------+----------+ + | tokenizer | | + +-----------------------------------------+ | + | scan_result | | + +-----------------+-----------------------+ | + | | tokenizer_error | buffer | + | token +-----------------------+ | + | |cBLU input_state | | + +-----------------+-----------------------+ | + | tokentype | span | | + +-----------------+-----------------------+----------+ + +.. code-block:: cpp + + #include diff --git a/xo-tokenizer/docs/scan-result-class.rst b/xo-tokenizer/docs/scan-result-class.rst new file mode 100644 index 00000000..6581a839 --- /dev/null +++ b/xo-tokenizer/docs/scan-result-class.rst @@ -0,0 +1,29 @@ + +.. _scan-result-class: + +Scan Result +=========== + +Represent the result of a tokenizer scan call + +Context +------- + +.. ditaa:: + :--scale: 0.85 + + +-----------------------------------------+----------+ + | tokenizer | | + +-----------------------------------------+ | + |cBLU scan_result | | + +-----------------+-----------------------+ | + | | tokenizer_error | buffer | + | token +-----------------------+ | + | | input_state | | + +-----------------+-----------------------+ | + | tokentype | span | | + +-----------------+-----------------------+----------+ + +.. code-block:: cpp + + #include