diff --git a/xo-tokenizer/docs/index.rst b/xo-tokenizer/docs/index.rst index b6353f4f..6efad465 100644 --- a/xo-tokenizer/docs/index.rst +++ b/xo-tokenizer/docs/index.rst @@ -1,6 +1,6 @@ .. xo-tokenizer documentation master file. -Xo-tokenizer documentation +xo-tokenizer documentation ========================== xo-tokenizer provides a tokenizer for the Schematika language. @@ -9,12 +9,16 @@ Syntax is generally C-like, but with some important differences. Notably, characters used for arithmetic operators (``+``, ``-``, ``*``, ``/``) may appear in variable names: ``one-of-those-days`` is an ordinary symbol. +Typically applications would use xo-reader to construct Schematika expressions +instead of interacting directly with ``xo::scm::tokenizer``. + .. toctree:: :maxdepth: 2 :caption: xo-tokenizer contents install examples + schematika-tokens implementation tokenizer-class scan-result-class diff --git a/xo-tokenizer/docs/schematika-tokens.rst b/xo-tokenizer/docs/schematika-tokens.rst new file mode 100644 index 00000000..21b4cdad --- /dev/null +++ b/xo-tokenizer/docs/schematika-tokens.rst @@ -0,0 +1,15 @@ +.. _schematika-tokens: + +Schematika Tokens +================= + +.. list-table:: Schematika Tokens + :widths: 15 30 30 + :header-rows: 1 + + * - tokentype + - examples + - description + * - tk_i64 + - 123, -8, + - 64-bit integer literal