Roland Conybeare
1f981a0636
subrepo: subdir: "xo-tokenizer" merged: "6e665ed7" upstream: origin: "git@github.com:Rconybea/xo-tokenizer.git" branch: "main" commit: "6e665ed7" git-subrepo: version: "0.4.9" origin: "???" commit: "???"
29 lines
860 B
ReStructuredText
29 lines
860 B
ReStructuredText
|
|
.. _scan-result-class:
|
|
|
|
Scan Result
|
|
===========
|
|
|
|
Represent the result of a tokenizer scan call
|
|
|
|
Context
|
|
-------
|
|
|
|
.. ditaa::
|
|
:--scale: 0.85
|
|
|
|
+-----------------------------------------+----------+
|
|
| tokenizer | |
|
|
+-----------------------------------------+ |
|
|
|cBLU scan_result | |
|
|
+-----------------+-----------------------+ |
|
|
| | tokenizer_error | buffer |
|
|
| token +-----------------------+ |
|
|
| | input_state | |
|
|
+-----------------+-----------------------+ |
|
|
| tokentype | span | |
|
|
+-----------------+-----------------------+----------+
|
|
|
|
.. code-block:: cpp
|
|
|
|
#include <xo/tokenizer/input_state.hpp>
|