xo-tokenizer: build + utest + reasonable implementation

This commit is contained in:
Roland Conybeare 2024-07-22 12:30:46 +10:00
commit 9dc37e84e6
15 changed files with 2154 additions and 0 deletions

9
src/tokenizer/token.cpp Normal file
View file

@ -0,0 +1,9 @@
/** @file token.cpp
*
* author: Roland Conybeare
**/
#include "token.hpp"
#include "xo/indentlog/print/tag.hpp"
/** end token.cpp **/