feat: store metadata once per compilation in compiled_contracts_metadata

[marcocastignoli]

Summary

First of two steps for #2924 (option B: a Sourcify-specific side table). Based on #2929 by @peterlodri-sec — same table shape and first-submitter-wins semantics, restructured for a zero-downtime rollout.

• Migration creates compiled_contracts_metadata (compilation_id uuid PK → compiled_contracts, metadata json NOT NULL) and nothing else — no in-migration backfill.
• Backfill happens out-of-band via services/database/schema-updates/backfill-compiled-contracts-metadata.mjs: batched with a keyset cursor, throttled, idempotent (ON CONFLICT DO NOTHING), resumable, with --dry-run and a --verify completeness check. Same pattern as the runtime-code-prefixes backfill from Fix similarity API database load and timeouts #2918 and the chain_id backfill from /v2/contracts/{chainId} endpoint hangs beyond certain matchId #2111. The variant picked per compilation is deterministic (lowest verified_contracts.id, i.e. first submitter).
• Server dual-writes: every stored verification also inserts its compilation's metadata into the side table (first wins). sourcify_matches.metadata keeps being written and remains the only read source — no read path changes in this PR, so nothing depends on the backfill being complete.

Rollout

1. Deploy this (migration + dual-write).
2. Run the backfill script; check completeness with --verify.
3. Ops: add compiled_contracts_metadata to the BigQuery Datastream mirror — the prod stream's include list is an explicit table allowlist, plus the sourcify_publication publication on the Postgres side, then an object backfill. Must happen before step 5: BigQuery's public_sourcify_matches currently mirrors the metadata column, so dropping it without mirroring the new table would silently remove metadata from BigQuery.
4. Follow-up PR: switch reads (STORED_PROPERTIES_TO_SELECTORS, getCompilationsByIds's LATERAL detour) to the side table and stop writing sourcify_matches.metadata.
5. Separate ops task: drop the column and reclaim the ~139 GB with pg_repack.

Staging reads through the whole window means no COALESCE fallback ever ships.

Notes

• The column deliberately stays json, not jsonb: json preserves the exact stored text, so the blob keeps hashing to the metadata hash embedded in the onchain bytecode (this property is asserted by the existing test in assertContractSaved). The json equality limitation doesn't bite here because grouping happens on the table's primary key.
• The dual-write lives in SourcifyDatabaseService, not AbstractDatabaseService, because the side table is Sourcify-specific and AllianceDatabaseService shares the abstract insert path (the Verifier Alliance DB has no such table).
• assertContractSaved now also checks the side table, so all existing integration verification tests cover the dual-write; dedicated tests cover first-submitter-wins and the no-metadata (Vyper) case.

TODO

• After the migration is deployed, open sourcify-prod-pg-bq-stream-sourcify-matches datastream in the GCP console and tick compiled_contracts_metadata;

Closes #2929

🤖 Generated with Claude Code

[kuzdogan]

While I review this, can you add release todos according to #2945

[kuzdogan]

The core change looks right: the dual-write sends the same object to sourcify_matches.metadata and the side table, the migration only creates a new table, the schema dump is a real dbmate regeneration, and the backfill query uses the existing indexes. Requesting changes for the items below; the rest are nits.

Should be addressed

• replace-metadata (customReplaceMethods.ts) only updates sourcify_matches.metadata; nothing can ever overwrite a compiled_contracts_metadata row, so repairs made before the read switch are lost. Needs a DO UPDATE there, or a listed prerequisite of the read-switch PR.
• --verify compares a count of current matches against count(*) of the side table, which also holds rows for compilations no current match points to any more, so it can hide gaps. A NOT EXISTS gap count is exact (query in the inline comment).
• services/database/README.md "Available upgrade scripts" needs a row for the new script (suggested text inline).
• CLAUDE.md should say PostgreSQL 15 (what CI uses), not 16.

Nits (inline): uncaught errors in the script (program.parseAsync().catch), slow re-runs (NOT EXISTS in batch_metadata), ETA on resume, metadata insert when the compilation already exists, metadata?: Metadata instead of as any, ON CONFLICT style, redundant query in assertContractSaved, new table in the CLAUDE.md join list.

_{Posted with Claude Code}

[kuzdogan]

replace-metadata never updates the side table

The private replace-metadata method in customReplaceMethods.ts (the repair tool for contracts with wrong stored metadata, #2227) runs only:

UPDATE sourcify_matches SET metadata = $2 WHERE id = $1

It does not touch compiled_contracts_metadata, and this insert is ON CONFLICT DO NOTHING, so nothing can ever overwrite a side-table row. After the read switch and the column drop, every repair made in the meantime is silently lost.

Suggest that replaceMetadata also upserts (DO UPDATE) the side-table row for the match's compilation_id — here, or as a listed prerequisite of the read-switch PR.

_{Posted with Claude Code}

[kuzdogan]

Wasted work when the compilation already exists

This runs for every stored verification. When the compilation was already in compiled_contracts (byte-identical contract verified before), the full metadata JSON is sent to Postgres, parsed, and then discarded by DO NOTHING.

The base class already computes isNewCompilation (it uses it to skip re-inserting sources). Returning that flag from insertNewVerifiedContract / updateExistingVerifiedContract and only inserting when it is true avoids that. The backfill covers pre-migration compilations anyway.

_{Posted with Claude Code}

[kuzdogan]

--verify can under-report missing rows

expected_rows counts distinct compilations with a current match that has metadata. actual_rows is count(*) of the side table. But the side table also holds rows for compilations that no current match points to any more:

1. A contract is verified with compilation X after deploy; the dual-write inserts a row for X.
2. A better match with a different compilation Y replaces it. The old verified_contracts row for X stays, but no sourcify_matches row points to it.
3. expected_rows counts Y only; actual_rows counts X and Y.

So surplus rows offset real gaps, and missingRows can even go negative. Counting the gap directly is exact and should be 0:

SELECT count(DISTINCT vc.compilation_id)
FROM verified_contracts vc
JOIN sourcify_matches sm ON sm.verified_contract_id = vc.id
WHERE sm.metadata IS NOT NULL
  AND NOT EXISTS (SELECT 1 FROM compiled_contracts_metadata m WHERE m.compilation_id = vc.compilation_id)

_{Posted with Claude Code}

[kuzdogan]

Missing row in the README's "Available upgrade scripts" table

services/database/README.md tells operators to check that table when they upgrade a live database. This script has no row there. Suggested:

Script	When to run	What it does
backfill-compiled-contracts-metadata.mjs	After applying the migration that adds compiled_contracts_metadata, on a server that already dual-writes to it. Run --verify before switching reads to the table.	Backfills compiled_contracts_metadata from sourcify_matches.metadata, one row per compilation (first current match by verified_contracts.id).

_{Posted with Claude Code}

[kuzdogan]

Errors are not caught

program.parse() does not wait for the async action, so a failed query (a bad --start-id failing the ::uuid cast, a dropped connection, a permission error) is an unhandled rejection: raw stack trace, the logger is bypassed, and closePool() is skipped.

post-v2.12-to-v2.13-upgrade.mjs does it right:

program.parseAsync().catch(async (err) => {
  logger.error(...);
  await closePool();
  process.exit(1);
});

_{Posted with Claude Code}

[kuzdogan]

Should be PostgreSQL 15, not 16

CI's validate-database-schema job runs postgres:15-alpine with postgresql-client-15, and the README says 15.13. The dump's version header is a comment line that CI strips, so 15 passes; there is no reason to require 16 here.

Suggest postgres:15 in the docker run and postgresql-client-15 in the sentence below, wording it as "pg_dump 15, to match CI". Keep the "not 17" warning (SET transaction_timeout), that part is correct.

Optional: this recipe could live in services/database/README.md under "Adding new migrations", which already describes regeneration and which CLAUDE.md links to above.

_{Posted with Claude Code}

[kuzdogan]

Nit: the "How the tables join" list should mention the new table, e.g. compiled_contracts_metadata.compilation_id → compiled_contracts.id (one metadata blob per compilation; sourcify_matches.metadata is kept only until the read switch).

_{Posted with Claude Code}

[kuzdogan]

Nit: metadata is declared required, but insertCompiledContractMetadata guards if (!metadata) return (Vyper has none) and the call site uses as any to get undefined past the type checker. metadata?: Metadata makes the type match reality and removes the cast. (The same as any exists on the sourcify_matches insert, so this is inherited.)

_{Posted with Claude Code}

[kuzdogan]

Nit: every other insert in this file uses ON CONFLICT ON CONSTRAINT <name> DO NOTHING; this is the only ON CONFLICT (column) form. Same behavior, just inconsistent — ON CONFLICT ON CONSTRAINT compiled_contracts_metadata_pkey DO NOTHING.

_{Posted with Claude Code}

[kuzdogan]

Nit: this repeats the three-table join of the query right above it. A LEFT JOIN compiled_contracts_metadata ccm ON ccm.compilation_id = vc.compilation_id on that query, selecting ccm.metadata AS compilation_metadata, does the same job with one query (assert non-null instead of length(1)).

_{Posted with Claude Code}