xo-tokenizer: + token table

xo-tokenizer/docs/index.rst

@@ -1,6 +1,6 @@
 .. xo-tokenizer documentation master file.
 
-Xo-tokenizer documentation
+xo-tokenizer documentation
 ==========================
 
 xo-tokenizer provides a tokenizer for the Schematika language.
@@ -9,12 +9,16 @@ Syntax is generally C-like, but with some important differences.
 Notably, characters used for arithmetic operators (``+``, ``-``, ``*``, ``/``)
 may appear in variable names:  ``one-of-those-days`` is an ordinary symbol.
 
+Typically applications would use xo-reader to construct Schematika expressions
+instead of interacting directly with ``xo::scm::tokenizer``.
+
 .. toctree::
    :maxdepth: 2
    :caption: xo-tokenizer contents
 
    install
    examples
+   schematika-tokens
    implementation
    tokenizer-class
    scan-result-class

xo-tokenizer/docs/schematika-tokens.rst

@@ -0,0 +1,15 @@
+.. _schematika-tokens:
+
+Schematika Tokens
+=================
+
+.. list-table:: Schematika Tokens
+   :widths: 15 30 30
+   :header-rows: 1
+
+   * - tokentype
+     - examples
+     - description
+   * - tk_i64
+     - 123, -8,
+     - 64-bit integer literal