Module lexer 

Lexer for Bynk v0.

Token kinds correspond to the terminals defined in the grammar (spec §3 and §4). Whitespace is skipped; // line comments are emitted as Comment tokens so the formatter can preserve them through round-trips (v1.1 LSP spec §3.5). Doc blocks (---) are emitted as DocBlock tokens, and one-line docs (--- text, #2203) as DocLine tokens, both lexed outside of logos (see tokenize).

#2203: the legacy spellings, a -- comment and a --| doc line, still lex as Comment and DocLine tokens, so every tool keeps working on an unmigrated file and the formatter can rewrite it. The parser reports each as an error with a machine-applicable fix (legacy_comment_error, legacy_doc_line_error); comment_is_legacy and doc_line_is_legacy tell the spellings apart.

Structs

Token

A token plus its source span.

Enums

TokenKind

Token kinds. Discriminants without payload data; the lexeme is recovered from the source string via the token’s Span.

Constants

MAX_SAFE_INT

#1657: the largest Int, Number.MAX_SAFE_INTEGER (2^53 − 1). An Int is a JS safe integer; the range is symmetric.

Functions

comment_body

Extract the body of a Comment trivia token: everything after the leading // marker, preserving its inline whitespace verbatim. Used by the parser when attaching comments to declarations.

comment_is_legacy

#2203: whether a TokenKind::Comment token is spelled the legacy way, --, rather than //.

doc_block_body_range

The doc-block body as a byte range into source — leading/trailing --- marker lines stripped, no further processing (unlike doc_block_content, which additionally strips a common per-line indent — not offset-preserving). Callers that need to map a position in the body back to source (e.g. document-link spans) use this instead of re-deriving it from the string doc_block_content returns.

doc_block_content

Extract the body content of a doc-block token from its source span. Strips the leading and trailing --- marker lines and returns the body verbatim. If every non-empty content line begins with the same horizontal whitespace prefix (e.g., because the doc block sits inside a brace-form commons body), that common prefix is removed so the body reads naturally when emitted as JSDoc.

doc_fence_len

#1885: the dash count of the fence a doc block holding doc is printed with: one more than the longest marker-shaped line in it (a line of three or more dashes alone, surrounding whitespace aside), and never fewer than three. A shorter fence would close at that line.

doc_line_is_legacy

#2203: whether a TokenKind::DocLine token is spelled the legacy way, --| (#1888), rather than --- text.

doc_line_text

The text of a TokenKind::DocLine token: everything after the marker, less one separating space or tab, less a closer (#2203: a final run of exactly the opener’s dashes, preceded by whitespace, with that whitespace), and less the line ending. Further indentation is kept, so a doc line can indent a Markdown code block. A legacy --| line (#1888) has no closer.

doc_line_text_range

#1888: the byte range of a doc line’s text within source, as doc_line_text returns it, for callers that map positions in a doc back to the source (document links).

has_blank_line_between

Returns true if there is a blank line (a line containing only whitespace) in source strictly between byte offsets from (inclusive) and to (exclusive). Used by the parser to detect orphan doc blocks.

legacy_comment_error

#2203 (DECISION E): the bynk.lex.legacy_comment error for the legacy -- comment at span (a TokenKind::Comment token), with the fix to //.

legacy_doc_line_error

#2203 (DECISION E): the bynk.lex.legacy_doc_line error for the legacy --| doc line at span (a TokenKind::DocLine token), with the fix to --- text. A --| with no text has no one-line spelling (a bare --- opens a block), so it gets no fix.

one_line_doc

#2203: the source spelling of a one-line doc whose text is text: --- and the text, or a ---- marker when the text ends in a whitespace-preceded run of three dashes (which a --- marker would read as its closer). Empty text is spelled with a closer, --- ---, since a bare --- opens a block.

tokenize

Tokenise a source string with no real file identity — every span’s FileId defaults to FileId::UNKNOWN. See tokenize_in for the real-identity entry point production callers use.

tokenize_expanding_holes

Like tokenize, but with every interpolated-string token replaced by the tokens of its holes — each hole’s bytes re-lexed and its token spans rebased to absolute source positions (the same rebase crate::parser applies when parsing a hole), recursing through nested interpolation. Chunk (literal) text between holes yields no tokens.

tokenize_expanding_holes_in

Like tokenize_expanding_holes, but stamping every token’s span (including rebased hole tokens) with file.

tokenize_in

Tokenise a source string, stamping every token’s span with file. Returns the full token vector or the first lexical error.

tokenize_legacy

#2203: tokenise source as the lexer read it before // comments, for the formatter’s migration of a file that still uses the legacy spellings. A -- opens a comment wherever it starts the input or follows whitespace (ADR 0188 D1), so --note and a --- Helpers --- divider are comments, as they were; --| first on its line is a doc line. // comments and --- blocks lex as in tokenize, so a half-migrated file reads too. Every legacy token keeps its legacy spelling, so comment_body and doc_line_text read it and the formatter prints the new one.