Function tokenize_in 

pub fn tokenize_in(
    source: &str,
    file: FileId,
) -> Result<Vec<Token>, CompileError>

Tokenise a source string, stamping every token’s span with file. Returns the full token vector or the first lexical error.

Doc blocks (--- … ---), one-line docs (--- text) and line comments (// ...) are recognised outside the logos-generated lexer: we scan the source one segment at a time, dispatching to logos for ordinary tokens between non-token spans.