# `Grammar.Native.Runtime.Tokenizer`
[🔗](https://github.com/joetjen/ichor_runtime/blob/v0.6.0/lib/grammar/native/runtime/tokenizer.ex#L1)

Char-level helpers shared by every `Grammar.Native.CharCompiler`-generated
function, plus the maximal-munch driver `Grammar.Native.generate/2`'s own
`lex_candidates/2` feeds into -- the Tokenizer stage of Aether's
Reader/Tokenizer/Lexer/Parser split. No captures, no `ref_stack` --
tokens can never contain a `Capture` or `Indent`, so a token body is
just text in, `{matched text, rest}` out.

# `char_fun`

```elixir
@type char_fun() :: (String.t() -> char_result())
```

# `char_result`

```elixir
@type char_result() :: {:ok, String.t(), String.t()} | :fail
```

# `advance_line_col`

```elixir
@spec advance_line_col(String.t(), pos_integer(), pos_integer()) ::
  {pos_integer(), pos_integer()}
```

# `and_pred_char`

```elixir
@spec and_pred_char(char_fun(), String.t()) :: char_result()
```

# `call_n_times_char`

```elixir
@spec call_n_times_char(char_fun(), non_neg_integer(), String.t()) :: char_result()
```

# `first_char_match`

```elixir
@spec first_char_match([char_fun()], String.t()) :: char_result()
```

Ordered choice: tries each of `funs` against `input` in order, short-circuiting on the first success (never evaluating later alternatives, same as PEG's own ordered choice).

# `in_ranges?`

```elixir
@spec in_ranges?(non_neg_integer(), [{non_neg_integer(), non_neg_integer()}]) ::
  boolean()
```

# `not_pred_char`

```elixir
@spec not_pred_char(char_fun(), String.t()) :: char_result()
```

# `opt_char`

```elixir
@spec opt_char(char_fun(), String.t()) :: char_result()
```

# `pick_longest`

```elixir
@spec pick_longest([{atom(), (-> {:ok, String.t(), String.t(), term()} | :fail)}]) ::
  {:ok, atom(), String.t(), String.t(), term()} | :fail
```

Picks the longest-matching candidate at the current lexer position,
ties broken by declaration order (candidates tried in that order,
replaced only on a *strictly* longer match). A zero-width match is
never accepted as "the next token" -- accepting one would advance
nothing and loop forever (mirrors `Grammar.VM.Tokenizer`'s own private
`consider` helper's same guard exactly); such a token can still exist
and be referenced from inside other tokens, it just can't win maximal
munch on its own.

`capture` is `nil` for an ordinary candidate, or a `Grammar.IR.Custom
Lexeme`-matched one's explicit override -- see `Grammar.VM.Token`.

# `plus_char`

```elixir
@spec plus_char(char_fun(), String.t()) :: char_result()
```

# `rep_char`

```elixir
@spec rep_char(
  char_fun(),
  non_neg_integer(),
  non_neg_integer() | :infinity,
  String.t()
) ::
  char_result()
```

# `star_char`

```elixir
@spec star_char(char_fun(), String.t(), String.t()) :: char_result()
```

# `tokenize`

```elixir
@spec tokenize(
  (String.t() ->
     [{atom(), (-> {:ok, String.t(), String.t(), term()} | :fail)}]),
  String.t()
) :: {:ok, [Grammar.VM.Token.t()]} | {:error, Ichor.Error.t()}
```

Tokenizes `input` completely via maximal munch, or reports the first position nothing matches -- `get_candidates` rebuilds the per-token match attempts against whatever's left of the input at each position.

# `tokenize_prefix`

```elixir
@spec tokenize_prefix(
  (String.t() ->
     [{atom(), (-> {:ok, String.t(), String.t(), term()} | :fail)}]),
  String.t()
) :: {:ok, [Grammar.VM.Token.t()]}
```

Like `tokenize/2`, but tokenizes only as much of a *prefix* of `input`
as it can -- stopping (successfully) the moment nothing matches,
instead of failing. For `Ichor.CustomLexeme`'s re-lex-and-match
primitive: the trailing bytes after wherever a dependency rule
actually stops (an interpolated string's closing `}`, say) generally
aren't valid tokens in their own right, and were never supposed to be
-- they belong to whatever's scanning the *outer* token, not to this
inner match.

---

*Consult [api-reference.md](api-reference.md) for complete listing*