The lossless tree: every byte of the input, as tokens under the rules that matched them.
The ordinary parse result keeps what a grammar captures and steps over
the rest -- trivia, punctuation, the order of differently named
captures. A tool that edits a document and writes it back (a
formatter, a config updater keeping its comments) needs all of it.
A parser generated with lossless: true returns this tree beside its
ordinary value (design/lossless-mode.md in ichor).
The tree holds four promises, whatever the grammar:
- Lossless.
text(root)is the input, byte for byte. - Ordered. The leaves, left to right, are every token of the input: named tokens, literals, punctuation and trivia.
- Trivia sits between, never at an edge. No node but the root begins or ends with a trivia token. A trivia token belongs to the deepest node with something significant on both sides of it; the input's leading and trailing trivia belong to the root.
- Every rule that matched is a node, the ones the ordinary tree passes through included. A rule that matched nothing is an empty node.
Which comment belongs to which value is not decided here: that is a policy of the format, and the tree hands a consumer what it needs to decide it -- each trivia token's siblings, line and column.
Summary
Types
A rule's match as a parser records it: the rule, the token range
[from, to) of the trimmed stream it matched, and the records of the
rules it called, in order (Ichor.Syntax.Record).
Functions
The input a node or token covers: for the root, the whole input.
Every token under a node, in source order.
The lossless tree of a parse: record is the root rule's match, its
token ranges counted in the stream with leading and trailing trivia
trimmed (Grammar.Source.trimmed/2); tokens is the whole stream,
untrimmed, exactly as the tokenizer returned it; skip is the
grammar's @skip token, or nil.
Types
@type record() :: {atom(), non_neg_integer(), non_neg_integer(), [record()]}
A rule's match as a parser records it: the rule, the token range
[from, to) of the trimmed stream it matched, and the records of the
rules it called, in order (Ichor.Syntax.Record).
Functions
@spec text(Ichor.Syntax.Node.t() | Ichor.Syntax.Token.t()) :: String.t()
The input a node or token covers: for the root, the whole input.
iex> Ichor.Syntax.text(%Ichor.Syntax.Token{name: :N, text: "42"})
"42"@spec tokens(Ichor.Syntax.Node.t()) :: [Ichor.Syntax.Token.t()]
Every token under a node, in source order.
iex> Ichor.Syntax.tokens(%Ichor.Syntax.Node{rule: :empty, children: []})
[]@spec tree(record(), [Grammar.VM.Token.t()], atom() | nil) :: Ichor.Syntax.Node.t()
The lossless tree of a parse: record is the root rule's match, its
token ranges counted in the stream with leading and trailing trivia
trimmed (Grammar.Source.trimmed/2); tokens is the whole stream,
untrimmed, exactly as the tokenizer returned it; skip is the
grammar's @skip token, or nil.
Examples
iex> tokens = [
...> %Grammar.VM.Token{name: :N, text: "1", line: 1, column: 1},
...> %Grammar.VM.Token{name: :WS, text: " ", line: 1, column: 2},
...> %Grammar.VM.Token{name: :N, text: "2", line: 1, column: 3}
...> ]
iex> tree = Ichor.Syntax.tree({:pair, 0, 3, [{:n, 2, 3, []}]}, tokens, :WS)
iex> Enum.map(tree.children, & &1.__struct__)
[Ichor.Syntax.Token, Ichor.Syntax.Token, Ichor.Syntax.Node]
iex> Ichor.Syntax.text(tree)
"1 2"