Repository navigation
fix: grammar, scanner and query fixes - #77
Merged
Merged
Conversation
Several expressions parsed without errors but with the wrong tree. `2**3**2` grouped to the left, though gnuplot evaluates it as `2**(3**2)`. Factorial bound looser than `**`, `~` sat below the additive operators, `^` shared a level with `|`, the comparison operators sat below the bitwise ones, `==` shared a level with `<`, `.` bound looser than `+` and `-`, and nested ternaries grouped to the left. The precedence table now follows the levels of gnuplot's own expression parser, which are the C levels plus `**` and factorial. Every pair of binary operators was checked against gnuplot 6.0.5. Equality and relational operators now need separate tokens. STATE_COUNT 12088 to 12104, TOKEN_COUNT 500 to 501, parser.c +231 KB. Corpus 247.
Option bodies and plot elements have no line boundary in the grammar, so a keyword name at the start of the next line could still join them. `set samples 5` followed by `offset=3` read `offset` as one more samples sub-keyword and failed at `=`, and `lw=3` after a plot command became a line width of the plot element. Command keywords already declined when assignment syntax follows. The plot style, style attribute and option body tables now do the same.
`if (0) print 1; else print 2` is the pre-5.0 form of if/else, still accepted by gnuplot 6. The grammar only allowed a braced else, so the old form failed at `else` and the else branch became an unconditional statement. The old-style body now takes an optional unbraced `else` statement. As in gnuplot, a nested old-style `if` leaves the `else` to the outer one, and an `else if` chain nests. The braced else moved into the braced branch, since gnuplot rejects it after an old-style body. STATE_COUNT 12104 to 12100. Corpus 249.
The opening marker of a datablock stopped at the first character that was not a letter, so `$D << E1` opened a block ending at `E` and left `1` behind, and `END_DATA` failed the same way. Markers longer than 99 characters were cut short and reported as matched. Markers are now read as whole identifiers, with digits and underscores, and the closing marker is compared the same way. The storage grows to the 1024 bytes the serialized state allows, and a marker that does not fit fails the scan instead of being truncated.
gnuplot ends a datablock at the first line that starts with the marker in column zero and ignores the rest of that line. The scanner matched the marker anywhere after skipped whitespace and required a word boundary after it, so `1 EOD` closed the block early and `EODxxx` never closed it. The closing marker now has to start in column zero, is compared as a prefix of the line, and takes the rest of the line with it.
`print |A|` and `set samples |A|+10` are valid, but `|` was not among the characters the same-line gate accepts as the start of a value, so the command ended at its keyword and the expression failed to parse.
gnuplot reads `''` inside a single-quoted string as one quote character, so `print 'a''b'` prints `a'b`. The grammar closed the string at the first quote and failed on the rest. The pair now parses as an escape_sequence. As in gnuplot, it only counts after other text in the string: at the start, or right after another pair, the first quote still closes the string. STATE_COUNT 12100 to 12101, TOKEN_COUNT 501 to 502.
gnuplot expands `@name` before it parses the line, so a macro can stand for an operand: `print @s`, `set xrange [@s:5]`, `plot @f with lines`. The grammar only accepted a macro as a whole statement and failed on every other use. A macro is now also an expression atom and a plot data source. STATE_COUNT unchanged, parser.c +61 KB.
`plot ... with errorbars` and its abbreviation `with e` are valid and plot y error bars, and `errorlines` (from `errorl`) does the same with lines. Only `set style data` knew these names, so every plot command using them failed. They are now rows of the plot style table, with gnuplot's minimum lengths, and `set style data` uses the shared plot_style rule, so its tree now has a plot_style node like the other style names. keywords.json is unchanged. STATE_COUNT 12101 to 12100, TOKEN_COUNT 502 to 500.
gnuplot accepts `every` down to `ev` in plot, splot, stats and fit, but the grammar only took the full word. keywords.json records the new minimum.
`print "abc"[2]` prints `b`, but subscript required a colon, so a literal string with a single index failed to parse. subscript now has a single-index form for string literals. On an identifier the same shape stays an array element. STATE_COUNT unchanged, parser.c +3 KB.
A word that is also a command abbreviation (`p` for plot) followed by brackets is either an array element assignment or a command with a range. The scanner took any colon inside the brackets as range syntax, so an index written as a ternary made `p[1?1:2]=3` a plot command. It now decides on the `=` after the closing bracket alone.
gnuplot accepts a string with no closing quote and ends it at the end of the line, so `print "a` prints `a`. The grammar failed on it. A zero-width external token now closes a string at a newline or at the end of the file. It is only valid inside a string, so it costs no states. Making the closing quote optional in the grammar was measured first and grew the parser to 41459 states. TOKEN_COUNT 500 to 501, parser.c +13 KB.
The README lists .gnuplot, .gp, .plt, .plot and .gnu, but tree-sitter.json only registered .gnuplot, so the tree-sitter CLI found no language for the other four. The parser does not change.
`system "printf hi"` runs `printf hi`, but the whole string literal was
injected as bash, quotes included, so the shell parse saw one quoted
word instead of a command and an argument. The same held for
`system("...")`. Both now offset the injected range past the quotes.
`! "cmd"` keeps the quotes: `!` hands the rest of the line to the shell
as written.
test/highlight-check.lua now also checks injected regions on small
snippets.
In `system cmd`, `cmd` is a variable whose value is the shell command, but the identifier itself was injected as bash. Identifier injection is now limited to `!`, where a bare word is the command (`! pwd`).
The builtin list had `columnhead` but not its full spelling, so a call written `columnheader(1)` was captured only as a function call. The `(columnheader)` pattern covers the bare `title columnheader` form, not calls.
The generated parser is ABI 15, but the declared `tree-sitter` peer was ^0.21, whose runtime stops at ABI 14, so `setLanguage` threw in initializeLanguageNodeClasses. The peer is now ^0.25. The peerDependenciesMeta key was misspelled `tree_sitter`, so the peer was never optional; it now is, and the runtime is also a devDependency so the binding test has it. CI runs the node binding test again.
Old-style if bodies with more than one command, raw datablock and inline plot data, raw `!` shell text, backtick substitution, function blocks, and a postfix factorial that reads across a newline.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
There was a problem hiding this comment.
Note
Copilot was unable to run its full agentic suite in this review.
Copilot review overview
4 open findings
\"cmd\" @_headmatches a literal tokencmdin Tree-sitter query syntax, not a field or node… · Newis_assignment_contextconsumes the first=and can returnfalsewhen encountering==,… · NewMAX_WORD_LENGTHto 1024 risks exhausting the external scanner serialization budget if the… · Newvim.treesitter.get_rangereturns byte offsets at indices 3 and… · New
What changed in this PR
This PR aligns the gnuplot Tree-sitter grammar/scanner and Neovim query behavior more closely with gnuplot 6.0.5, and updates packaging/CI to match ABI 15 requirements.
Changes:
- Adjust expression/operator precedence, string handling, old-style
if ... else, plot styles, and macro/value parsing in the grammar/scanner. - Refine Neovim injection/highlight queries (notably
system/!bash injections) and add targeted query checks in the test harness. - Update registry metadata (file extensions) and Node binding/CI dependencies for tree-sitter ABI 15.
| File | Description |
|---|---|
| tree-sitter.json | Registers additional gnuplot file extensions for editor integration. |
| src/scanner.c | Updates datablock scanning, assignment context detection, and unterminated string handling. |
| grammar.js / src/grammar.json | Brings grammar behavior in line with gnuplot parsing rules (precedence, if/else, strings, macros, subscripts). |
| queries/injections.scm | Changes bash injection extraction rules for system and !. |
| queries/highlights.scm | Updates builtin function highlight set (e.g., columnheader). |
| test/highlight-check.lua | Adds pinned injection/capture checks using Neovim runtime. |
| test/corpus/*.txt | Adds/updates corpus cases to lock in new parsing behavior. |
| package.json | Bumps Node peer dependency expectation to tree-sitter ^0.25 (ABI 15). |
| .github/workflows/ci.yml | Re-enables Node binding smoke tests in CI. |
| docs/deferred.md | Records remaining known grammar gaps. |
🧠 Review effort: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
6
to
11
| (cmd_system | ||
| "cmd" @_head | ||
| (string_literal) @injection.content | ||
| (#eq? @_head "system") | ||
| (#offset! @injection.content 0 1 0 -1) | ||
| (#set! injection.language "bash")) |
Comment on lines
916
to
932
| @@ -905,13 +923,12 @@ static bool is_assignment_context(TSLexer* lexer) { | |||
| depth++; | |||
| else if (c == ']') | |||
| depth--; | |||
| else if (c == ':' && depth == 1) | |||
| has_colon = true; | |||
| } | |||
| if (has_colon) | |||
| return false; | |||
| while (lexer->lookahead == ' ' || lexer->lookahead == '\t') consume(lexer); | |||
| return lexer->lookahead == '='; | |||
| if (lexer->lookahead != '=') | |||
| return false; | |||
| consume(lexer); | |||
| return lexer->lookahead != '='; // '=' but not '==' | |||
| } | |||
Comment on lines
+7
to
+9
| // Datablock marker capacity, NUL included. Bounded by tree-sitter's 1024-byte | ||
| // serialization buffer, since the open marker is the serialized state. | ||
| #define MAX_WORD_LENGTH 1024 |
Comment on lines
+80
to
+84
| local range = vim.treesitter.get_range(node, text, metadata[id]) | ||
| out[#out + 1] = { | ||
| lang = metadata["injection.language"], | ||
| text = text:sub(range[3] + 1, range[6]), | ||
| } |
The factorial `!` was an internal token, so after `print 3` a `! ls` on the next line was read as `3!` followed by an error. The scanner now emits it only when it sits on the same logical line, leaving a `!` after a newline to the shell command. Two conflicts that only the old token needed are dropped.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Grammar, scanner, query and packaging fixes, each checked against gnuplot 6.0.5. Each commit fixes one issue, carries its corpus or highlight test and its regenerated parser.
Operator precedence now follows gnuplot's parser: all pairs of the 18 binary operators were compared against gnuplot, from 38 mismatches to none.
set style datanow uses the shared plot_style rule, so its tree is(style (plot_style)). The node binding requires tree-sitter ^0.25 (ABI 15) and its load test is enabled in CI.Parser: 12088 to 12100 states, parser.c +307 KB (+1.1%).
tree-sitter testpasses (259) on every commit, the oracle check passes, and regeneration leavessrc/unchanged. Remaining gaps are recorded in docs/deferred.md.