Skip to content

fix(chat): repair a SQL statement the database rejects, once - #92

Merged
alberthflores95 merged 2 commits into
mainfrom
fix/chat-sql-repair
Oct 27, 2026
Merged

alberthflores95 merged 2 commits into
mainfrom
fix/chat-sql-repair

Conversation

@alberthflores95

Copy link
Copy Markdown
Collaborator

Stacked on #91. The base is fix/persist-chats-artifacts so this diff shows only the chat repair; retarget to main once #91 merges.

Problem

Artifacts repair a rejected SQL statement once (#90). The chat did not: on ClickHouse 22.1 (GONAC PoC server) every dialect slip reached the user as Pipeline error. Errors seen in the server log on 2026-10-27, from the chat and artifacts:

Error What the model wrote
SYNTAX_ERROR, Unrecognized token: '\xc3' (x3) a Spanish alias with an unquoted ñ
ILLEGAL_AGGREGATION (x2) SUM(x) AS x, then another SUM(x)
ILLEGAL_TYPE_OF_ARGUMENT (x2) toDateOrNull(order_date) AS order_date on a column that already is a Date
NO_COMMON_TYPE (x2) a String date compared with a Date
UNKNOWN_IDENTIFIER (x3) a missing column, or arithmetic between window functions
UNKNOWN_TABLE (x1) a table that does not exist in PRD (not fixable by a rewrite)

Change

  • sql_repair.py (new): the repair, its dialect notes and its error hints move out of routers/artifact.py, shared by chat and artifacts. Three hints added for the errors whose text does not name the fix (non-ASCII alias, String vs Date, conversion on a Date). Connection, timeout and authentication errors are not sent to the model.
  • routers/query.py: a rejected chat statement gets one repair (token phase sql_repair). A statement that fails twice keeps its second error; the chat shows the statement that ran.
  • prompts/clickhouse.py: rule 7 asks for ASCII-only aliases; new rules 14 (no alias that shadows the column it reads) and 15 (window arithmetic in an outer query). Prose only: an example in a rules block acts as a template.

Not covered: Flash mode generates and runs its own SQL inside ask-intent-resolution.

Tests

  • test_chat_sql_repair.py: 16 tests (one repair and its result, no repair when the statement runs, no second repair, original error when no usable repair, unreachable database not sent to the model, the three new hints with the real error texts, which errors are repairable). Checked against mutants without the chat repair and without the new hints: they fail.
  • test_artifact_sub_sql_repair.py and test_artifact_env.py pass unchanged apart from patching SettingsCache where the repair now lives.

Measurement on the GONAC server (before/after on the same questions) to follow in a comment.

🤖 Generated with Claude Code

Artifacts got one repair attempt for a rejected statement in PR #90; the
chat had none, so on ClickHouse 22.1 every dialect slip reached the user
as "Pipeline error". On the GONAC server most chat questions failed that
way: a Spanish alias with an unquoted ñ, an alias that shadows the
column it aggregates, arithmetic between window functions, a String
date compared with a Date, a conversion applied to a column that already
is a Date.

- sql_repair.py: the repair, its dialect notes and error hints move out
  of routers/artifact.py and are shared by the chat and the artifacts.
  Three hints are added for the GONAC errors whose text does not name the
  fix. Connection, timeout and authentication errors are not sent to the
  model: no rewrite of the statement fixes them.
- routers/query.py: a rejected chat statement gets one repair, recorded as
  the sql_repair phase; a statement that fails twice keeps its second
  error, and the chat shows the statement that ran.
- prompts/clickhouse.py: ASCII-only aliases, no alias that shadows the
  column it reads, window arithmetic in an outer query. Prose, no worked
  SQL, because an example in a rules block acts as a template.

Flash mode generates and runs its own SQL inside ask-intent-resolution
and is not covered.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alberthflores95

Copy link
Copy Markdown
Collaborator Author

Measurement on the GONAC PoC server (2026-10-27)

Same 12 Spanish questions, workspace resumen-de-ventas-gonac, env dev (ClickHouse 22.1, database dbt_qas_bi), mode precise, Claude Sonnet 5. Each arm in a one-off container with the live configuration; one run per arm, so read it as an indication, not a rate.

SQL ran Failed on the first statement Rescued by the repair
A, sha-c5aed3e (before) 2 / 12 10 n/a
B, sha-b87fbd5 (this PR) 7 / 12 7 2

First-statement errors, by kind:

Kind A B
Unquoted non-ASCII alias (año) 2 0
Alias shadowing the aggregated column 3 0
Arithmetic between window functions 1 0
Column missing from the QAS table 4 6
if() branches of different types (Decimal vs 0) 0 1

The remaining failures are mostly not SQL. dbt_qas_bi.gold_mv_resumen was last modified on 2026-08-11 and has 19 columns; dbt_prd_bi.gold_mv_resumen has 27. Eight exist only in PRD (proyectado, nombre_destinatario, umb_pendiente, facturado_umb, objetivo_umb, proyectado_umb, umb_aa, unidad), and the semantic layer published to dev declares the first three. Any dev question that leads to them fails whatever the SQL; the repair rescued two by answering from the columns QAS has. That is a data refresh, not a code change.

The if() type mismatch is the one dialect failure the repair did not fix; a hint for it is a candidate follow-up, to be measured first.

Deployed to the PoC server as sha-b87fbd5.

@alberthflores95

Copy link
Copy Markdown
Collaborator Author

Same 12 questions on prod (dbt_prd_bi), sha-b87fbd5

12 / 12 ran. One first statement failed (arithmetic between window functions, UNKNOWN_IDENTIFIER) and the repair fixed it; the other eleven ran first time. The if() type mismatch seen on dev did not come up in this run.

dev is not the place to judge this workspace: its only data product has to be published to dev before prod, and dbt_qas_bi.gold_mv_resumen is an obsolete copy that lacks eight of the columns the semantic layer declares.

"Ran" means the database accepted the SQL; the figures in the answers were not checked against the source here.

Brings the artifact delete of PR #91 into this stacked branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alberthflores95
alberthflores95 changed the base branch from fix/persist-chats-artifacts to main October 27, 2026 04:18
@alberthflores95
alberthflores95 merged commit 612d5d9 into main Oct 27, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant