xo-numeric/xo-tokenizer
2025-06-25 07:51:01 -05:00
..
cmake Add 'xo-tokenizer/' from commit '830c6ebe55' 2025-05-11 01:36:13 -05:00
docs xo-tokenizer: docs + error-handling improvement 2025-06-23 23:08:12 -05:00
example xo-tokenizer: streamline input_state/tokenizer interaction + docs 2025-06-25 07:48:44 -05:00
include/xo/tokenizer xo-tokenizer: minor streamlining 2025-06-25 07:51:01 -05:00
src/tokenizer Add 'xo-tokenizer/' from commit '830c6ebe55' 2025-05-11 01:36:13 -05:00
utest tokenizer: + input_state helper 2025-06-24 23:50:21 -05:00
.gitignore Add 'xo-tokenizer/' from commit '830c6ebe55' 2025-05-11 01:36:13 -05:00
CMakeLists.txt xo-tokenizer: bugfix: yields token works + 2phase utest 2025-06-22 16:16:23 -05:00
README.md Add 'xo-tokenizer/' from commit '830c6ebe55' 2025-05-11 01:36:13 -05:00

schematica tokenizer library

Getting Started

build + install 'xo-cmake` dependency

Installs a few cmake ingredients, along with a build assistant xo-build for XO projects such as this one.

build + install other required XO dependencies

$ xo-build --clone --configure --build --install xo-indentlog
$ xo-build --clone --configure --build --install xo-refnct
$ xo-build --clone --configure --build --install xo-subsys
$ xo-build --clone --configure --build --install xo-reflectutil

Note: can use -n to dry-run here

copy xo-tokenizer repository locally

$ xo-build --clone xo-tokenizer

or equivalently

$ git clone git@github.com:Rconybea/xo-tokenizer.git

build + install xo-tokenizer

$ xo-build --configure --build --install xo-tokenizer

or equivalently:

$ PREFIX=/usr/local  # or wherever you prefer
$ cmake -DCMAKE_INSTALL_PREFIX=${PREFIX} -S xo-tokenizer -B xo-tokenizer/.build
$ cmake --build xo-tokenizer/.build
$ cmake --install xo-tokenizer/.build

build for unit test coverage

$ cmake -DCMAKE_BUILD_TYPE=coverage -DCMAKE_INSTALL_PREFIX=$PREFIX xo-tokenizer/.build-ccov
$ cmake --build xo-tokenizer/.build-ccov

LSP support

$ cd xo-tokenizer
$ ln -s .build/compile_commands.json  # lsp will look for compile_commands.json in the root of the source tree