# `Grammar.Lexer`
[🔗](https://github.com/joetjen/ichor_runtime/blob/v0.2.0/lib/grammar/lexer.ex#L1)

The Lexer stage of Aether's Reader/Tokenizer/Lexer/Parser split: walks
a Tokenizer's raw token stream left-to-right, applying every
`@keywords`/`@refine` rule an `Aether.Grammar`'s `refiners` map
declares, before the Parser ever runs.

Shared by both backends (`Grammar.VM` calls this directly;
`Grammar.Native`'s generated `tokenize/2` calls it with
`grammar.refiners` spliced in as a literal at compile time) since this
operates purely on already-produced tokens plus a small data table --
unlike the Tokenizer/Parser stages, there's no per-grammar compiled
matching logic to generate here, `@keywords` is just a plain map and
`@refine` just dispatches to an already-compiled module by name.

# `refiner`

```elixir
@type refiner() ::
  {:keywords, %{required(String.t()) => atom()}}
  | {:custom, module :: module(), function :: atom(),
     possible_names :: [atom()]}
```

Mirrors `Aether.Grammar.refiner/0` (dev-only, in `ichor` proper).

# `reclassify`

```elixir
@spec reclassify([Grammar.VM.Token.t()], %{required(atom()) => refiner()}) ::
  {:ok, [Grammar.VM.Token.t()]} | {:error, Ichor.Error.t()}
```

Reclassifies `tokens` in order, or reports the first `@refine`-rejected
one. `refiners` empty is the overwhelmingly common case (most grammars
have no `@keywords`/`@refine` at all), short-circuited without walking
the list.

---

*Consult [api-reference.md](api-reference.md) for complete listing*