Skip to content

Comments and strings are never read as code - #4

Merged
tabladrum merged 15 commits into
mainfrom
fix/comment-string-masking
Oct 10, 2026
Merged

Comments and strings are never read as code#4
tabladrum merged 15 commits into
mainfrom
fix/comment-string-masking

Conversation

@tabladrum

@tabladrum tabladrum commented Oct 10, 2026

Copy link
Copy Markdown
Contributor

Part of the fix for the 2026-10-10 parsing audit: 53 places where grove/astkit read comments, docstrings or string literals as code. The grove PR consumes this.

What changes

  • New package textmask. One comment/string masker for every supported language. It replaces seven inconsistent hand-written blankers in grove and astkit.
    • It keeps byte offsets and newlines, and keeps interpolation holes as code.
    • An oracle test compares it with tree-sitter's comment and literal nodes. On ~30k real files across 14 languages it disagrees only on JSX text, a documented gap.
    • It runs at about 1.5 ms/MB.
  • Extractor fixes (audit A1–A9 and the same bug class elsewhere):
    • Signatures and modifiers carry no comments or C#/PHP attribute sections.
    • Python wrapped def signatures include the full header.
    • Java Exported, Lombok accessors, Kotlin fun interface and C# record struct come from the AST, not substring checks.
    • Rust macro and C #define call sites, ObjC macro enums, and COBOL PERFORM/CALL read masked text.
    • .h C++ detection ignores comments, #if 0 and the __cplusplus guard.
    • ObjC containers inside #if/extern "C" are found.
  • Kotlin: comments can no longer move tree-sitter error recovery.
    • Kotlin is parsed with comment lines cut out, then the tree is edited back to source positions.
    • go-tree-sitter's Tree.Edit writes OldEndPoint into new_end_point, so treeedit.go calls ts_tree_edit through cgo. It reaches the native tree by reflection on the unexported BaseTree.c.
    • If that layout changes, it falls back to a plain parse. TestTreeEditReachesNativeTree guards it. Please review this part.
  • New command cmd/decoy. It injects code-shaped comments only where a line ends in code, for the metamorphic decoy invariant in research ci_invariants.py.

Testing

  • go test ./... passes.
  • The grove branch's twin regression suite covers every audit ID.
  • The decoy invariant holds on 23 real repos spanning all 14 languages.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KXa6Cv1ZUXcdMqXcJfQFhX

tabladrum and others added 14 commits October 10, 2026 11:23
@tabladrum @claude
textmask: one comment/string masker for every language
9b582bb 
Mask, MaskComments, MaskFile and MaskInactivePreprocessor replace the
hand-written blankers in grove/astkit that read comments and string
literals as code. One linear forward lexer per language family keeps byte
offsets and line breaks, keeps interpolation holes as code, and handles
the shapes the audit found broken: Python quotes vs Rust lifetimes, `#`
in Rust raw strings / JS private fields / PHP attributes, multi-line
literals (text blocks, raw strings, templates, heredocs), Go raw strings
without escapes, C++ digit separators and `#error` apostrophes, and
Kotlin/Swift/PHP/C#/Python interpolation.

TestMaskMatchesTreeSitter compares Mask byte by byte with tree-sitter's
comment and literal nodes on adversarial testdata; TEXTMASK_CORPUS runs
the same check over real repositories.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tabladrum @claude
extract: declaration signatures and modifiers carry no comments
f2a511e 
Signatures were sliced from source text, so a comment inside a header
stayed in it: `class A(Base):  # was Fake`, `fun g(a: Int, // : Thing`,
`@interface M : Sup /* <Fake> */ <P>`, and grove parsed those words as
types. Comments are tree-sitter extras, so internalast now cuts them
from the tree: SignatureBeforeBody drops comment nodes (replacing the
hand-written stripHeaderComments lexer), and HeaderText / FirstLineSig
give the header or first line without them. Text with nothing to cut is
returned unchanged, so comment-free signatures stay byte-identical.

Also:
- C#/PHP attribute sections are cut from signatures like Java
  annotations (they stay on Annotations): attribute strings no longer
  read as types, and a FirstLine signature that was only the attribute
  line is now the declaration.
- A Python def whose parameters wrap gets its whole header up to the
  colon (`def build( size, ) -> Gadget:`), not `def build(`.
- Java, Kotlin and Swift modifier lists skip comment nodes
  (`@GwtIncompatible // reflection` put the comment in Modifiers).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tabladrum @claude
extract: macros and ObjC macro enums read only code
933033e 
- Rust macro token trees are masked with textmask before the call scan
  (comments and char literals threw off the old string-only regex:
  `assert_eq!(c, '"', "see bar(1)")` called bar).
- C #define bodies are masked before cMacroCallRe: `printf("value(%d)")`
  no longer calls value, nor does a comment on a continued line. Argument
  splitting reads the masked body too. A block comment in the middle of
  a continued macro body made tree-sitter-c emit an ERROR node and the
  macro vanished; cMacroFromError recovers it.
- `#if 0` / `#elif 0` branches are not descended in the C-family walk
  (only their alternative), so disabled code declares nothing.
- objcMacroEnums matches on textmask-masked text with `#if 0` branches
  masked (blankCComments, which skipped strings without blanking them,
  is gone), and directive lines inside an enum body are not members:
  `ModeA, #if X ModeB, #endif ModeC` keeps all three.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tabladrum @claude
extract: Java exports, Lombok, Kotlin fun interface, C# record struct…
6449100 
… from the AST

- Java Exported comes from the modifiers node's `public` keyword only;
  strings.Contains(sig, "public") exported `void publication()` and
  `String mode = "public"`.
- Lombok: isX only for a field whose type node is primitive `boolean`
  (a "boolean x" initializer string made isKind); `@lombok.Getter` /
  `@lombok.Setter` (fully qualified) are recognized.
- Kotlin `fun interface`: the `fun` token is read from the gap between
  the interface keyword and the preceding node, so `// just for fun`
  above a plain interface is not the modifier.
- C# `record struct` is the anonymous `struct` token, not a substring of
  a header that may hold string arguments.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tabladrum @claude
cobol: PERFORM, CALL and data clauses read masked statement text
c8f80d7 
Normalized lines now carry two textmask views aligned with their text:
code (comments and literals blanked) and lits (comments blanked). Verbs,
section/paragraph/data-item structure and PIC/REDEFINES/OCCURS read
code, so `DISPLAY 'PLEASE PERFORM BACKUP'` performs nothing and an
inline `*> PERFORM OLD-PARA` (fixed format kept it) is a comment. The
CALL verb is found in code and its literal target read at the same
offset in lits, so `CALL 'PROG'` still resolves while `DISPLAY 'CALL
SUPPORT'` is no call. COPY member scanning uses code instead of its own
quote regex. The lines are masked as free format because they are
already column-normalized (textmask's own COBOL mode defaults to fixed
format and does not detect free format).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tabladrum @claude
lang: .h C++ sniff ignores comments, #if 0 and __cplusplus-only branches
2e4aa0d 
cppHeaderMarker matched `class handling for the C API` in a block
comment and the standard `#ifdef __cplusplus / extern "C" { / #endif`
guard, making C headers C++. isCPPHeader masks comments and `#if 0`
branches and blanks the C++-only branches of __cplusplus conditionals
before matching, the same rule as grove's header sniff
(internal/parser sniffHeaderCode), so the layers agree. An unguarded
`extern "C"` is not valid C and still makes the header C++.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tabladrum @claude
test: comment/string-as-code cases for astkit extractors (audit A1-A8)
55956bc 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tabladrum @claude
cmd/decoy: inject code-shaped comments for metamorphic index tests
ad72c0e 
Decoys go only where the line ends in code (checked on textmask output), so
the program never changes; an index of the decoyed tree must equal the
original's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tabladrum @claude
Merge extractor fixes: comments and strings never read as code
e6d8928 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tabladrum @claude
kotlin: cut comments from header text by masking, not by comment nodes
4472ddd 
Signatures, constructor parameter lists, type alias signatures and the
fun-interface check read comments out of the text with textmask instead
of relying on comment nodes in the tree, so they stay comment-free on a
tree parsed without comments. internalast gains HeaderTextCut and
FirstLineSigCut, which take extra byte ranges to cut.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tabladrum @claude
kotlin: parse without comments so they cannot move error recovery
913ac67 
The vendored Kotlin grammar fails on valid code (object : X by y,
f(x).p = v, a primary constructor on the next line), and its recovery
depended on comments: comment tokens are extras, and tree-sitter prices
a recovery by the bytes and lines it skips. A comment line after a {
turned classes into ERROR nodes (okio, Exposed), a class body into a
lambda (okio ByteString), method locals into fields (moshi), and a
string holding /* let an end-of-file block comment surface as a class.

Engine.Parse now blanks Kotlin comments (textmask), cuts comment-only
lines out of the parser's input, and puts them back with tree edits so
node positions are src's. go-tree-sitter's Tree.Edit passes the old end
point as the new one (rows never move), so treeEdit calls ts_tree_edit
through cgo.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tabladrum @claude
Merge Kotlin: parse without comments so they cannot move error recovery
f362705 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tabladrum @claude
strategies: Objective-C containers inside #if and extern "C" blocks
0d03808 
extractObjCNodes only looked at the translation unit's direct children.
AFNetworking's UIKit headers wrap the whole @interface in
`#if TARGET_OS_IOS`; once the file parsed cleanly the class sat under
preproc_if and vanished (it had surfaced only as recovery junk from a
broken parse). objcContainers descends into conditional regions (an
`#if 0` branch only through its alternative) and `extern "C"` bodies,
as extractCNodes does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tabladrum @claude
Merge ObjC containers inside conditionals
38078a0 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tabladrum tabladrum mentioned this pull request Oct 10, 2026
Merged
@tabladrum @claude
textmask: keep oracle testdata LF on Windows checkouts
73b9859 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tabladrum
tabladrum merged commit 70352c1 into main Oct 10, 2026
3 checks passed
@tabladrum tabladrum mentioned this pull request Oct 10, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@tabladrum