# `Grammar.Source`
[🔗](https://github.com/joetjen/ichor_runtime/blob/v0.6.0/lib/grammar/source.ex#L1)

The very first, Reader-adjacent stage in front of the Tokenizer:
confirms `input` is valid UTF-8 before any char-level matcher --
built on Elixir binary pattern matching against `::utf8` codepoints --
ever touches it. Malformed input left unchecked doesn't fail cleanly:
it crashes a compiled matcher with a `MatchError` partway through
tokenizing, deep inside either backend, instead of surfacing as an
ordinary `Ichor.Error`.

Shared by both backends (`Grammar.VM.parse/3`/`run_sequence/4` call
this directly; `Grammar.Native`'s generated `tokenize/2` calls it too)
since, like `Grammar.Lexer`, this is pure data validation with nothing
grammar-specific to compile.

# `placed`

```elixir
@spec placed(Ichor.Error.t() | [Ichor.Error.t()], String.t()) ::
  Ichor.Error.t() | [Ichor.Error.t()]
```

`error` -- or each of a list of them -- placed at the end of `input`
when it says nowhere, and shown: given the line of `input` it is on,
and a caret, when it has a position but no snippet.

For a caller that gets its error from somewhere that never saw the
input: `Grammar.GLR.Runtime` builds "unexpected end of input" from
the token stream alone, and its callers, which have the input, place
it here.

# `position`

```elixir
@spec position(tuple(), non_neg_integer(), String.t()) ::
  {pos_integer(), pos_integer()}
```

Where position `pos` of `stream` is, as `{line, column}`: where its
token starts, or, past the last token, the end of `input`.

**An error always says where.** "Unexpected end of input" and a root
that matched nothing are errors about a place too -- the end, or the
first token -- and a user handed one with no line and column has
nowhere to look. No input at all is `{1, 1}`. Columns count
codepoints, as the tokenizer does.

# `trimmed`

```elixir
@spec trimmed([Grammar.VM.Token.t()], atom() | nil) :: [Grammar.VM.Token.t()]
```

`tokens` without the `skip` tokens before the first other token and
after the last.

**Trivia around the input is not part of it.** Skip-splicing puts a
`skip*` in front of every sequence element but the first, so
nothing inside a grammar ever expects a skip token at the very start
of the input or after its very end. Every engine and both backends
trim the stream here before matching the root, so a document's
leading newline or trailing space means the same to all of them.
The skip tokens between two others are left alone: those are the
grammar's.

`skip` is the grammar's `@skip` token, or `nil` for `@noskip`, which
changes nothing.

# `validate`

```elixir
@spec validate(String.t()) :: {:ok, String.t()} | {:error, Ichor.Error.t()}
```

Returns `input` unchanged if it's valid UTF-8, or an `Ichor.Error` (`stage: :lexer`) otherwise.

---

*Consult [api-reference.md](api-reference.md) for complete listing*