Skip to content

Report affected rows on postgres, and rewrite placeholders in linear time - #8

Merged
yogthos merged 2 commits into
mainfrom
pg-affected-rows-and-linear-placeholders
Aug 11, 2026
Merged

Report affected rows on postgres, and rewrite placeholders in linear time#8
yogthos merged 2 commits into
mainfrom
pg-affected-rows-and-linear-placeholders

Conversation

@yogthos

@yogthos yogthos commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

The two loose ends from the v0.1.0 review, plus a correctness bug that fell out of fixing the second one.

Affected rows on postgres

execute! promises rows affected and sqlite delivers it, but postgres returned nil from execute!, update! and delete! alike, so anything branching on the count silently saw nothing. The existing suite missed it because pg update applied checked that an update had happened by counting rows rather than by reading the return value. db.pg/exec now reads PQcmdTuples, which reports 0 for commands that carry no count, matching what sqlite3_changes gives for DDL.

Linear placeholder rewriting

pg-placeholders rebuilt the statement one character at a time with (str out c), copying the accumulator on every character. It now records where each placeholder starts and joins the pieces once. str-join-strs in jolt collects into a list and does a single string-append, so the join itself is linear too.

sql chars before after
500,000 19,343 ms 146 ms

Flat at 29ms per 100k characters from 125KB through 2MB, so it is genuinely linear now rather than just faster.

The bug that fell out of it

Rewriting the scanner also fixed what it recognises. Only single-quoted literals were skipped before, so a ? in a comment, a quoted identifier or a dollar-quoted body got rewritten and, worse, consumed a parameter number, shifting every later placeholder. Both triggers are ordinary things to write in a migration: a prose comment ending in a question mark, or a plpgsql body in dollar quotes.

before:  select /* ? */ ? as c   ->  select /* $1 */ $2 as c    -- $2 with one parameter
after:   select /* ? */ ? as c   ->  select /* ? */ $1 as c

The scanner now skips string literals, quoted identifiers, line and block comments including the nesting postgres allows, dollar-quoted bodies with or without a tag, and E strings where a backslash escapes the next character. That last one was outright broken before, not merely mis-numbered: it closed the string at the escaped quote, so the ? inside became a placeholder and the real one after it did not. Unterminated constructs run to the end of the statement rather than throwing.

Tests

Written first, and they failed as diagnosed: ten of the sixteen lexer cases plus the timing bound, while the basic and single-quote cases passed, which is what pinned the fault to the constructs the old scanner did not know about. The placeholder table runs without a database, so the sqlite-only build covers it, and the timing bound is loose enough not to be flaky while still far under what quadratic needs for half a megabyte.

I also checked it end to end against a migration-shaped statement rather than only unit-level: a plpgsql function in dollar quotes whose body contains a ?, nested block comments, a prose comment with a question mark, and real parameters after all of it. 86 checks pass locally against postgres 16, and the sqlite-only run is clean.

Also

insert!'s docstring claimed it returns nil on postgres, which was never true. It returns lastval, and inserting into a table with no sequence throws lastval is not yet defined in this session. Documented what it actually does. Making that case work would mean generating returning, which is a design change rather than a doc fix, so I left it.

Yogthos added 2 commits August 11, 2026 10:59
…time

Two loose ends from the v0.1.0 review.

execute! promises rows affected and sqlite delivers it, but postgres returned nil
from execute!, update! and delete! alike, so anything branching on the count
silently saw nothing. The existing suite missed it because it checked that an
update had applied by counting rows rather than by reading the return value.
db.pg/exec now reads PQcmdTuples, which reports 0 for commands that carry no
count, matching what sqlite3_changes gives for DDL.

pg-placeholders rebuilt the statement one character at a time with (str out c),
copying the accumulator on every character, so cost grew with the square of the
length: half a megabyte of SQL took 19.3 seconds of pure string copying before
the query was even sent. It now records where each placeholder starts and joins
the pieces once, which takes 145ms for the same input and holds a flat 29ms per
100k characters from 125KB up to 2MB.

Rewriting the scanner also fixed what it recognises, which was a correctness bug
rather than a performance one. Only single-quoted literals were skipped before, so
a ? in a comment, a quoted identifier or a dollar-quoted body was rewritten and,
worse, consumed a parameter number, shifting every later placeholder. A prose
comment ending in a question mark is easy to write in a migration, and a plpgsql
body in dollar quotes is normal in one. The scanner now skips string literals,
quoted identifiers, line and block comments including postgres' nesting,
dollar-quoted bodies with or without a tag, and E'' strings where a backslash
escapes the next character. Unterminated constructs run to the end of the
statement instead of throwing.

Tests came first and failed the way the diagnosis predicted, ten of the sixteen
lexer cases plus the timing bound. The placeholder table needs no database, so the
sqlite-only build covers it. Also verified end to end against a migration-shaped
statement: a plpgsql function in dollar quotes whose body contains a ?, nested
block comments, and real parameters after all of it.

insert!'s docstring claimed it returns nil on postgres, which was never true; it
returns lastval, and inserting into a table with no sequence throws because
lastval has nothing to report. Documented what it actually does.
Review of the new scanner turned up a regression it introduced. Postgres allows $
inside an identifier after the first character, so a$b$c is one name, but the
scanner read $b$ as a dollar quote opening, found no closing tag, and swallowed
the rest of the statement. The placeholder after it was never rewritten and
postgres answered with a syntax error. The old scanner got this right by accident,
since it only ever looked at single quotes.

A dollar quote now only opens where a token can start, meaning the preceding
character is not part of an identifier, which is where postgres' own lexer draws
the line. Note that x$$a$$ is likewise one identifier to postgres rather than a
dollar quote, so rejecting it here matches.

The same rule applies to E'': it introduces backslash escapes only when the E
stands alone, not when it ends a word, so date'2020-01-01' and time'12:00:00' stay
plain literals where a backslash is literal too.

Covered both ways: the identifier cases, a dollar quote after an operator so the
boundary rule cannot simply disable the feature, and live checks that a plpgsql
body in dollar quotes is still skipped and a $-bearing identifier still reaches
postgres intact.
@yogthos

yogthos commented Aug 11, 2026

Copy link
Copy Markdown
Contributor Author

Self-review turned up a regression this PR introduced, now fixed in ec58ee1.

Postgres allows $ inside an identifier after the first character, so a$b$c is a single name. The new scanner read $b$ as a dollar quote opening, found no closing tag, swallowed the rest of the statement, and never rewrote the placeholder after it. Postgres then answered syntax error at end of input. The old scanner got this right by accident, since it only ever looked at single quotes, so this was a real regression rather than a pre-existing gap.

select note from t where a$b$c = ?
  before fix:  select note from t where a$b$c = ?     -- placeholder dropped
  after fix:   select note from t where a$b$c = $1

A dollar quote now only opens where a token can start, meaning the preceding character is not part of an identifier. That is where postgres own lexer draws the line: x$$a$$ is likewise one identifier to postgres, not a dollar quote, so rejecting it here matches rather than merely being conservative. The same rule applies to E, which introduces backslash escapes only when the E stands alone, so date'2020-01-01' and time'12:00:00' stay plain literals where a backslash is literal too.

Tested both directions, since a boundary rule like this can pass its tests by just disabling the feature: the identifier cases, a dollar quote after an operator, and live checks that a plpgsql body in dollar quotes is still skipped and its ? left intact while a $-bearing identifier reaches postgres and returns the right row. 91 checks pass locally against postgres 16 and the sqlite-only run is clean.

@yogthos
yogthos merged commit cb4c4f3 into main Aug 11, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant