xo-tokenizer2/xo-tokenizer
2025-07-23 23:19:16 -05:00
..
cmake Add 'xo-tokenizer/' from commit '830c6ebe55' 2025-05-11 01:36:13 -05:00
docs xo-reader xo-expression: nested lambdas working properly + docs 2025-07-06 14:13:44 -05:00
example reader reports tokenizer errors through normal return 2025-07-19 16:47:59 -05:00
include/xo/tokenizer xo-reader xo-expression xo-tokenizer xo-jit: comparison + apply 2025-07-23 23:19:16 -05:00
src/tokenizer xo-reader xo-expression xo-tokenizer xo-jit: comparison + apply 2025-07-23 23:19:16 -05:00
utest detailed parser error reporting [wip - 1 example] 2025-07-19 21:09:57 -05:00
.gitignore Add 'xo-tokenizer/' from commit '830c6ebe55' 2025-05-11 01:36:13 -05:00
CMakeLists.txt xo-tokenizer: bugfix: yields token works + 2phase utest 2025-06-22 16:16:23 -05:00
README.md Add 'xo-tokenizer/' from commit '830c6ebe55' 2025-05-11 01:36:13 -05:00

schematica tokenizer library

Getting Started

build + install 'xo-cmake` dependency

Installs a few cmake ingredients, along with a build assistant xo-build for XO projects such as this one.

build + install other required XO dependencies

$ xo-build --clone --configure --build --install xo-indentlog
$ xo-build --clone --configure --build --install xo-refnct
$ xo-build --clone --configure --build --install xo-subsys
$ xo-build --clone --configure --build --install xo-reflectutil

Note: can use -n to dry-run here

copy xo-tokenizer repository locally

$ xo-build --clone xo-tokenizer

or equivalently

$ git clone git@github.com:Rconybea/xo-tokenizer.git

build + install xo-tokenizer

$ xo-build --configure --build --install xo-tokenizer

or equivalently:

$ PREFIX=/usr/local  # or wherever you prefer
$ cmake -DCMAKE_INSTALL_PREFIX=${PREFIX} -S xo-tokenizer -B xo-tokenizer/.build
$ cmake --build xo-tokenizer/.build
$ cmake --install xo-tokenizer/.build

build for unit test coverage

$ cmake -DCMAKE_BUILD_TYPE=coverage -DCMAKE_INSTALL_PREFIX=$PREFIX xo-tokenizer/.build-ccov
$ cmake --build xo-tokenizer/.build-ccov

LSP support

$ cd xo-tokenizer
$ ln -s .build/compile_commands.json  # lsp will look for compile_commands.json in the root of the source tree