Add ch dictionary primitive#191
Merged
Merged
Conversation
Add DictionaryDefinition/DictionaryAttribute types, the dictionary() builder, validation rules, canonicalization, CREATE DICTIONARY SQL rendering, and a binary planner diff (ClickHouse dictionaries have no ALTER — any structural change becomes a single CREATE OR REPLACE DICTIONARY, with the source password masked so a redacted [HIDDEN] value never causes phantom drift). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… waits Classify drop_dictionary as destructive (SQL-scan rule + planner-marker path, with a dependency-break warning), add dictionary to the drift existence-comparison kind union (following the materialized_view precedent — existence-drift, not deep-shape), and wait on system.tables for create_dictionary/drop_dictionary DDL propagation. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…nto schema files
Add create-dictionary-parser.ts to parse attributes/primaryKey/
source/layout/lifetime/comment out of a dictionary's create_table_query
(verified live: DDL dictionaries populate system.tables.create_table_query,
with inline passwords already redacted to [HIDDEN]). Map engine
'Dictionary' to kind 'dictionary' instead of skipping it, and render
pulled dictionaries as dictionary({...}) calls with a leading comment
when the source password was redacted.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Emit a TypeScript interface (and Zod schema when emitZod is enabled) from a dictionary's attributes, sharing the same field-rendering path as table row types. Dictionaries are always included in codegen output regardless of includeViews, since their shape is structured rather than an arbitrary query. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Add waitForDictionary() (polls system.dictionaries) alongside waitForTable/waitForView, and migrate-dictionary.e2e.test.ts covering create -> insert -> dictGet -> structural CREATE OR REPLACE -> dictGet -> drift-clean -> hand-drop -> drift-dirty (missing_object) -> restore -> drop-blocked-without---allow-destructive. Add CREATE DICTIONARY / CREATE OR REPLACE DICTIONARY / dictionary drop cases to the EXPLAIN AST SQL-validation suite. Verified end-to-end against a local ClickHouse 26.6 Docker container: all cases pass, plus a manual pull round-trip against a CLICKHOUSE()-sourced dictionary confirmed system.tables.create_table_query is populated for DDL dictionaries and the pulled schema file typechecks. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Add dictionary() to the DSL reference (fields, attributes, credential handling via env-var interpolation, no-ALTER semantics), mention it in the schema overview and ClickHouse compatibility guide, and document the pull plugin's [HIDDEN]-password handling and codegen's dictionary interface generation. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…, and BIDIRECTIONAL The dictionary model, SQL renderer, and introspection parser only covered PRIMARY KEY/SOURCE/LAYOUT/LIFETIME and a handful of attribute modifiers, so chkit silently dropped RANGE(...) (required by RANGE_HASHED layouts), SETTINGS(...), and the BIDIRECTIONAL attribute modifier on every pull and re-render. Verified the exact rendered DDL against a live ClickHouse instance, including the full introspect-then-parse round trip. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…STER on replace Dictionary renamedFrom was dead API surface — nothing read it, so a renamed dictionary always diffed as drop_dictionary + create_dictionary. Wired it into the same schema/CLI rename-mapping pipeline tables already use (remap-before-diff + post-hoc operation injection), added a matching --rename-dictionary flag, and confirmed RENAME DICTIONARY IF EXISTS is accepted and idempotent against a live ClickHouse instance. Also fixes a related gap found in the process: ON CLUSTER injection matched on the literal "CREATE DICTIONARY" prefix, so it silently never applied to CREATE OR REPLACE DICTIONARY — meaning every structural change to a dictionary (there is no ALTER DICTIONARY) went unreplicated in cluster mode. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Covers the full dictionary() feature shipped on this branch: DSL/validation/ SQL/planner support, RANGE/SETTINGS/BIDIRECTIONAL, pull introspection, codegen, destructive-op safety, rename support, and the ON CLUSTER fix. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…sword drift chkit generate now flags two SOURCE(...) password gaps: a literal password about to land in the generated migration file, and a password-only change that produces no operation because chkit masks passwords before diffing.
…nconditionally `diffDictionary` masked every SOURCE(...) password before comparing old vs new, so a password-only change was silently dropped and never produced a migration. Now only a source still carrying ClickHouse's [HIDDEN] introspection placeholder is excluded from the diff; a real password change diffs and migrates like any other field. Replaces the now-obsolete "undetected password change" warning in `chkit generate` with a new warning in `chkit pull`, printed when an introspected dictionary's password comes back as [HIDDEN] and chkit can't recover it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
dictionary-password-warnings.md described its diffing fix as "fixed a bug," but that behavior never shipped in a release — it only existed within this same unreleased branch. Fold it into the single dictionary-schema-primitive changeset so the changelog reads as one coherent feature description instead of a feature followed by a fix to itself. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…a prior bug Same issue as the previous changeset cleanup: "now correctly stamps" reads as a fix to previously-shipped behavior, but ON CLUSTER support for dictionaries never shipped before this branch. State the behavior plainly instead. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…lain text ClickHouse's [HIDDEN] redaction on SOURCE(...) can be turned off server-side (display_secrets_in_show_and_select + the displaySecretsInShowAndSelect grant), in which case pull receives the real password and writes it verbatim into the schema file with no other signal. Detect the non-[HIDDEN] case the same way the existing warning detects [HIDDEN], and mention the setting as an escape hatch in the [HIDDEN] warning too. Verified against a local ClickHouse instance with both settings enabled. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… list; wait on system.dictionaries for create_dictionary Two issues from PR #191 review: - create-dictionary-parser.ts searched the whole query text for SOURCE/LAYOUT/LIFETIME/RANGE/SETTINGS, the same bug class just fixed for create-table-parser.ts (#190). RANGE in particular collides with ClickHouse's real range() array function — a dictionary with `EXPRESSION ...range(1, 10)...` on one attribute and a genuine RANGE(MIN ... MAX ...) clause would silently lose that clause on pull. Anchor all of it to the region after the attribute list, like the table parser does. - waitForDDLPropagation's create_dictionary case waited on system.tables (waitForTable) instead of the already-added waitForDictionary (system.dictionaries), so chkit migrate could return before a dictionary had actually finished loading — particularly relevant on replicated/clustered targets. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.