Report affected rows on postgres, and rewrite placeholders in linear time - #8
Conversation
…time Two loose ends from the v0.1.0 review. execute! promises rows affected and sqlite delivers it, but postgres returned nil from execute!, update! and delete! alike, so anything branching on the count silently saw nothing. The existing suite missed it because it checked that an update had applied by counting rows rather than by reading the return value. db.pg/exec now reads PQcmdTuples, which reports 0 for commands that carry no count, matching what sqlite3_changes gives for DDL. pg-placeholders rebuilt the statement one character at a time with (str out c), copying the accumulator on every character, so cost grew with the square of the length: half a megabyte of SQL took 19.3 seconds of pure string copying before the query was even sent. It now records where each placeholder starts and joins the pieces once, which takes 145ms for the same input and holds a flat 29ms per 100k characters from 125KB up to 2MB. Rewriting the scanner also fixed what it recognises, which was a correctness bug rather than a performance one. Only single-quoted literals were skipped before, so a ? in a comment, a quoted identifier or a dollar-quoted body was rewritten and, worse, consumed a parameter number, shifting every later placeholder. A prose comment ending in a question mark is easy to write in a migration, and a plpgsql body in dollar quotes is normal in one. The scanner now skips string literals, quoted identifiers, line and block comments including postgres' nesting, dollar-quoted bodies with or without a tag, and E'' strings where a backslash escapes the next character. Unterminated constructs run to the end of the statement instead of throwing. Tests came first and failed the way the diagnosis predicted, ten of the sixteen lexer cases plus the timing bound. The placeholder table needs no database, so the sqlite-only build covers it. Also verified end to end against a migration-shaped statement: a plpgsql function in dollar quotes whose body contains a ?, nested block comments, and real parameters after all of it. insert!'s docstring claimed it returns nil on postgres, which was never true; it returns lastval, and inserting into a table with no sequence throws because lastval has nothing to report. Documented what it actually does.
Review of the new scanner turned up a regression it introduced. Postgres allows $ inside an identifier after the first character, so a$b$c is one name, but the scanner read $b$ as a dollar quote opening, found no closing tag, and swallowed the rest of the statement. The placeholder after it was never rewritten and postgres answered with a syntax error. The old scanner got this right by accident, since it only ever looked at single quotes. A dollar quote now only opens where a token can start, meaning the preceding character is not part of an identifier, which is where postgres' own lexer draws the line. Note that x$$a$$ is likewise one identifier to postgres rather than a dollar quote, so rejecting it here matches. The same rule applies to E'': it introduces backslash escapes only when the E stands alone, not when it ends a word, so date'2020-01-01' and time'12:00:00' stay plain literals where a backslash is literal too. Covered both ways: the identifier cases, a dollar quote after an operator so the boundary rule cannot simply disable the feature, and live checks that a plpgsql body in dollar quotes is still skipped and a $-bearing identifier still reaches postgres intact.
|
Self-review turned up a regression this PR introduced, now fixed in ec58ee1. Postgres allows A dollar quote now only opens where a token can start, meaning the preceding character is not part of an identifier. That is where postgres own lexer draws the line: Tested both directions, since a boundary rule like this can pass its tests by just disabling the feature: the identifier cases, a dollar quote after an operator, and live checks that a plpgsql body in dollar quotes is still skipped and its |
The two loose ends from the v0.1.0 review, plus a correctness bug that fell out of fixing the second one.
Affected rows on postgres
execute!promises rows affected and sqlite delivers it, but postgres returnednilfromexecute!,update!anddelete!alike, so anything branching on the count silently saw nothing. The existing suite missed it becausepg update appliedchecked that an update had happened by counting rows rather than by reading the return value.db.pg/execnow readsPQcmdTuples, which reports 0 for commands that carry no count, matching whatsqlite3_changesgives for DDL.Linear placeholder rewriting
pg-placeholdersrebuilt the statement one character at a time with(str out c), copying the accumulator on every character. It now records where each placeholder starts and joins the pieces once.str-join-strsin jolt collects into a list and does a singlestring-append, so the join itself is linear too.Flat at 29ms per 100k characters from 125KB through 2MB, so it is genuinely linear now rather than just faster.
The bug that fell out of it
Rewriting the scanner also fixed what it recognises. Only single-quoted literals were skipped before, so a
?in a comment, a quoted identifier or a dollar-quoted body got rewritten and, worse, consumed a parameter number, shifting every later placeholder. Both triggers are ordinary things to write in a migration: a prose comment ending in a question mark, or a plpgsql body in dollar quotes.The scanner now skips string literals, quoted identifiers, line and block comments including the nesting postgres allows, dollar-quoted bodies with or without a tag, and
Estrings where a backslash escapes the next character. That last one was outright broken before, not merely mis-numbered: it closed the string at the escaped quote, so the?inside became a placeholder and the real one after it did not. Unterminated constructs run to the end of the statement rather than throwing.Tests
Written first, and they failed as diagnosed: ten of the sixteen lexer cases plus the timing bound, while the basic and single-quote cases passed, which is what pinned the fault to the constructs the old scanner did not know about. The placeholder table runs without a database, so the sqlite-only build covers it, and the timing bound is loose enough not to be flaky while still far under what quadratic needs for half a megabyte.
I also checked it end to end against a migration-shaped statement rather than only unit-level: a plpgsql function in dollar quotes whose body contains a
?, nested block comments, a prose comment with a question mark, and real parameters after all of it. 86 checks pass locally against postgres 16, and the sqlite-only run is clean.Also
insert!'s docstring claimed it returns nil on postgres, which was never true. It returnslastval, and inserting into a table with no sequence throwslastval is not yet defined in this session. Documented what it actually does. Making that case work would mean generatingreturning, which is a design change rather than a doc fix, so I left it.