Token

Represent a single lexical token in the Schematika language

Context

#include <xo/tokenizer/token.hpp>

• Represent a single lexical token

• Does not share any storage with original input stream (maintains a local copy).

• Remembers copied input extent. Convert on demand to native untagged representation

Example

void foo() {
    using namespace xo::scm;

    token<char> tk = token<char>::i64_token("123");

    tk.is_valid(); // -> true
    tk.text(); // -> "123"s;

    tk.tk_type(); // -> tokentype::tk_i64
    tk.i64_value(); // -> 123

    cout << tk << endl; // -> <token :type i64 :text 123>
}

Class

template<typename CharT>
class token

Represent a Schematika lexical token.

Instance Variables

group Token-instance-vars

Variables

tokentype tk_type_ = tokentype::tk_invalid

category for this token

std::string text_

characters comprising this token. only provided for certain token types:

tk_i64 tk_f64 tk_string tk_symbol

Constructors

group token constructors

Functions

token() = default

default ctor creates token with type tk_invalid

inline token(tokentype tk_type, const std::string &text = "")

create token with type tk_type and input text text

static inline token invalid()

create invalid token (same as null ctor, but explicit)

static inline token bool_token(const std::string &txt)

Create token representing a boolean literal from text txt txt must be true or false

static inline token i64_token(const std::string &txt)

Create token representing 64-bit signed integer literal parsed from decimal txt. The string txt must be a decimal integer literal, since i64_value re-parses txt.

static inline token f64_token(const std::string &txt)

create token representing 64-bit floating-point literal parsed from decimal txt The string txt must be a decimal floating-point literal, since f64_value re-parses txt.

static inline token string_token(const std::string &txt)

create token representing literal string parsed from txt

static inline token symbol_token(const std::string &txt)

create token representing a symbol parsed from txt. Note that not all strings are valid symbol names.

static inline token leftangle()

token representing left angle bracket "<"

static inline token rightangle()

token representing right angle bracket ">"

static inline token leftparen()

token representing left parenthesis "("

static inline token rightparen()

token representing right parenthesis ")"

static inline token leftbracket()

token representing left bracket "["

static inline token rightbracket()

token representing right bracket "]"

static inline token leftbrace()

token representing left brace "{"

static inline token rightbrace()

token representing right brace "}'

static inline token dot()

token representing period "."

static inline token comma()

token representing comma ","

static inline token colon()

token representing colon ":"

static inline token doublecolon()

token representing double-colo "::"

static inline token semicolon()

token representing semicolon ";"

static inline token singleassign()

token representing single-assignment "="

static inline token assign_token()

token representing unrestricted assignment ":="

static inline token yields()

token representing indirection "->"

static inline token plus_token()

token for "+"

static inline token minus_token()

token for "-"

static inline token star_token()

token for "*"

static inline token slash_token()

token for "/"

static inline token type()

token representing keyword type

static inline token def()

token representing keyword def

static inline token lambda()

token representing keyword lambda

static inline token if_token()

token representing keyword if

static inline token else_token()

token representing keyword else

static inline token let()

token representing keyword let

static inline token in()

token representing keyword in

static inline token end()

token representing keyword end

Access Methods

group Token-access-methods

Functions

inline tokentype tk_type() const

inline const std::string &text() const

General Methods

group Token-general-methods

Functions

inline bool is_valid() const

true if token understood to represent valid input i.e. any token type except tk_invalid

inline bool is_invalid() const

true for sentinel token with type tk_invalid

inline bool has_variable_text() const

true for tokens with variable text. false for those with fixed textual representation

bool bool_value() const

expect input matching true or false

std::int64_t i64_value() const

expect input matching [+|-][0-9][0-9]*

double f64_value() const

expect input matching [+|-][0-9]*[.][0-9]*[e|E][+|-][0-9]*

void print(std::ostream &os) const

print human-readable token representation on stream os