Skip to content
This repository was archived by the owner on Sep 15, 2026. It is now read-only.

null table schema update #277

Merged
jakeloo merged 4 commits into from
Aug 29, 2025
Merged

null table schema update #277

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
6ab912b
null table schema update
nischitpra Aug 29, 2025
c0ebbc1
insert token transfers mv
nischitpra Aug 29, 2025
dc75352
minor change
nischitpra Aug 29, 2025
eea1630
nit schema update
jakeloo Aug 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 21 additions & 12 deletions internal/tools/clickhouse/0004_clickhouse_create_insert_null_table.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
CREATE TABLE IF NOT EXISTS insert_null_block_data (
CREATE TABLE IF NOT EXISTS insert_null_block_data
(
chain_id UInt256,

block Tuple(
block_number UInt256,
block_timestamp DateTime,
Expand All @@ -22,8 +22,7 @@ CREATE TABLE IF NOT EXISTS insert_null_block_data (
gas_limit UInt256,
gas_used UInt256,
withdrawals_root FixedString(66),
base_fee_per_gas Nullable(UInt64)
),
base_fee_per_gas Nullable(UInt64)),
transactions Array(Tuple(
hash FixedString(66),
nonce UInt64,
Expand Down Expand Up @@ -55,8 +54,7 @@ CREATE TABLE IF NOT EXISTS insert_null_block_data (
blob_gas_used Nullable(UInt64),
blob_gas_price Nullable(UInt256),
logs_bloom Nullable(String),
status Nullable(UInt64)
)),
status Nullable(UInt64))),
logs Array(Tuple(
block_number UInt256,
block_hash FixedString(66),
Expand All @@ -69,8 +67,7 @@ CREATE TABLE IF NOT EXISTS insert_null_block_data (
topic_0 String,
topic_1 String,
topic_2 String,
topic_3 String
)),
topic_3 String)),
traces Array(Tuple(
block_number UInt256,
block_hash FixedString(66),
Expand All @@ -91,9 +88,21 @@ CREATE TABLE IF NOT EXISTS insert_null_block_data (
value UInt256,
author Nullable(FixedString(42)),
reward_type LowCardinality(Nullable(String)),
refund_address Nullable(FixedString(42))
)),

refund_address Nullable(FixedString(42)))),
token_transfers Array(Tuple(
token_type LowCardinality(String),
token_address FixedString(42),
token_id UInt256,
from_address FixedString(42),
to_address FixedString(42),
block_number UInt256,
block_timestamp DateTime,
transaction_hash FixedString(66),
transaction_index UInt64,
amount UInt256,
log_index UInt64,
batch_index Nullable(UInt16))),
insert_timestamp DateTime DEFAULT now(),
is_deleted UInt8 DEFAULT 0
) ENGINE = Null;
)
ENGINE = Null
158 changes: 0 additions & 158 deletions internal/tools/clickhouse/0007_clickhouse_create_token_transfers_mv.sql
View file
Open in desktop

This file was deleted.

2 changes: 1 addition & 1 deletion internal/tools/clickhouse/0011_clickhouse_create_address_transactions_mv.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
CREATE MATERIALIZED VIEW address_transactions_mv
CREATE MATERIALIZED VIEW IF NOT EXISTS address_transactions_mv
TO address_transactions
AS
SELECT
Expand Down
2 changes: 1 addition & 1 deletion internal/tools/clickhouse/0013_clickhouse_create_address_transfers_mv.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
CREATE MATERIALIZED VIEW address_transfers_mv
CREATE MATERIALIZED VIEW IF NOT EXISTS address_transfers_mv
TO address_transfers
AS
SELECT
Expand Down
20 changes: 20 additions & 0 deletions internal/tools/clickhouse/0014_clickhouse_create_insert_token_transfers_mv.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
CREATE MATERIALIZED VIEW IF NOT EXISTS insert_token_transfers_mv
TO token_transfers
AS SELECT
chain_id,
tt.1 AS token_type,
tt.2 AS token_address,
tt.3 AS token_id,
tt.4 AS from_address,
tt.5 AS to_address,
tt.6 AS block_number,
tt.7 AS block_timestamp,
tt.8 AS transaction_hash,
tt.9 AS transaction_index,
tt.10 AS amount,
tt.11 AS log_index,
tt.12 AS batch_index,
Comment on lines +5 to +16

@coderabbitai coderabbitai Bot Aug 29, 2025

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Tuple index misalignment: fields shifted by one → data corruption risk.

Per the new schema, the tuple layout starts with chain_id, so token_type should be tt.2, token_address tt.3, …, and batch_index tt.13. Current mapping is off-by-one. Fix as below; also cast batch_index to nullable.

-    tt.1 AS token_type,
-    tt.2 AS token_address,
-    tt.3 AS token_id,
-    tt.4 AS from_address,
-    tt.5 AS to_address,
-    tt.6 AS block_number,
-    tt.7 AS block_timestamp,
-    tt.8 AS transaction_hash,
-    tt.9 AS transaction_index,
-    tt.10 AS amount,
-    tt.11 AS log_index,
-    tt.12 AS batch_index,
+    tt.2 AS token_type,
+    tt.3 AS token_address,
+    tt.4 AS token_id,
+    tt.5 AS from_address,
+    tt.6 AS to_address,
+    tt.7 AS block_number,
+    tt.8 AS block_timestamp,
+    tt.9 AS transaction_hash,
+    tt.10 AS transaction_index,
+    tt.11 AS amount,
+    tt.12 AS log_index,
+    CAST(tt.13 AS Nullable(UInt16)) AS batch_index,
🤖 Prompt for AI Agents 
In
internal/tools/clickhouse/0014_clickhouse_create_insert_token_transfers_mv.sql
around lines 5 to 16, the tuple indexes are off by one because the new schema
prepends chain_id; update every tt.N by incrementing the index by 1 (so
token_type becomes tt.2, token_address tt.3, token_id tt.4, from_address tt.5,
to_address tt.6, block_number tt.7, block_timestamp tt.8, transaction_hash tt.9,
transaction_index tt.10, amount tt.11, log_index tt.12, batch_index tt.13) and
ensure batch_index is cast/defined as nullable per schema (replace non-null type
with its nullable equivalent).

insert_timestamp,
is_deleted
Comment on lines +17 to +18

@coderabbitai coderabbitai Bot Aug 29, 2025

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Verification agent

🧩 Analysis chain

Soft-delete semantics won’t propagate for existing rows.

MVs are append-only. If is_deleted changes in insert_null_block_data after initial insert, the MV won’t update existing token_transfers. Ensure the destination table uses an engine/pattern (e.g., ReplacingMergeTree with a version) that can accept compensating rows, or handle deletes via a separate stream.

🏁 Script executed:

#!/bin/bash
# Inspect token_transfers engine and primary key for upsert/dedup strategy
rg -nC3 -P 'CREATE\s+TABLE\s+(default\.)?token_transfers\b.*\n.*ENGINE\s*=\s*[A-Za-z]+' --type=sql

Length of output: 100

🏁 Script executed:

#!/bin/bash
# Locate the CREATE TABLE definition for token_transfers
rg -nC3 'CREATE\s+TABLE.*token_transfers' -g '*.sql'
# Fallback: list all usages of token_transfers in SQL files for context
rg -nC3 'token_transfers' -g '*.sql'

Length of output: 13207

🏁 Script executed:

#!/bin/bash
# Extract the engine and ordering key for token_transfers
sed -n '1,50p' internal/tools/clickhouse/0006_clickhouse_create_token_transfers.sql

Length of output: 1560

🏁 Script executed:

#!/bin/bash
sed -n '1,200p' internal/tools/clickhouse/0006_clickhouse_create_token_transfers.sql

Length of output: 3784

🏁 Script executed:

#!/bin/bash
# Print the CREATE TABLE definition for insert_null_block_data
sed -n '1,200p' internal/tools/clickhouse/0004_clickhouse_create_insert_null_table.sql

Length of output: 3607

Soft-delete flags won’t propagate through this MV. Materialized views only append on inserts to insert_null_block_data; updates to is_deleted there won’t re-emit rows into token_transfers. To support soft-deletes, switch token_transfers to an engine that collapses on is_deleted (e.g. CollapsingMergeTree(is_deleted)) or use a proper version column with ReplacingMergeTree—and emit compensating delete rows via a dedicated MV/stream.

🤖 Prompt for AI Agents 
In
internal/tools/clickhouse/0014_clickhouse_create_insert_token_transfers_mv.sql
around lines 33-34, the materialized view appends rows from
insert_null_block_data so updates to is_deleted do not propagate and
soft-deletes are lost; either change the token_transfers table engine to a
collapsing engine (e.g. CollapsingMergeTree(is_deleted)) so tombstone markers
collapse rows on read, or implement a proper versioning approach: add a version
column and use ReplacingMergeTree(version) (or maintain explicit tombstone rows)
and create/update a secondary MV or stream that emits compensating
delete/tombstone rows when is_deleted is set so deletes are represented in
token_transfers.

FROM insert_null_block_data
ARRAY JOIN token_transfers AS tt