Repository navigation
Conversation
|
Thanks for putting this together. The FK incompatibility between unlogged shadow tables and logged public tables is a real concern — shipping that without detection or a clear error message would be a frustrating experience for users. And the benchmarks show the improvement is negligible at typical dev scales (slightly slower at 1M rows, 1.19x at 3M). I think #1 captures the meaningful performance win here. If WAL pressure turns out to be a real problem for someone I'm open to revisiting, but I'd want to see that as an issue first so we can align on the use case before code gets written. |
CREATE TABLE LIKE ... INCLUDING ALL creates all indexes up front, so the subsequent INSERT SELECT pays for per-row index maintenance on every index. On large tables this dominates shadow time. Instead, create the shadow with INCLUDING ALL EXCLUDING INDEXES, bulk copy the rows, then recreate indexes in a single sort pass each. The primary key is attached via ALTER TABLE ... USING INDEX so it remains a PRIMARY KEY constraint rather than just a unique index. Benchmarks on a widgets table with 1 PK + 5 indexes (1 unique, 1 composite, 1 partial, 2 plain): 1M rows: 14.05s -> 5.98s (2.35x) 3M rows: 51.18s -> 19.25s (2.66x) Speedup grows with scale because sort-based index builds amortize better than per-row B-tree inserts. Semantics are preserved: same indexes, same PK designation, same uniqueness, same check constraints, same defaults and NOT NULL — covered by new tests pinning each of these. Index redirection is done in SQL via REPLACE with quote_ident(), which matches pg_get_indexdef's own output exactly. Tables whose names need quoting (uppercase, reserved words) work as a result, and the single catalog query returns index name, is-primary flag, and pre-rewritten CREATE INDEX statement together.
Branch schemas are disposable by design — db:branch:reset rebuilds them from public — so the crash-loss semantics of UNLOGGED are a natural fit for dev/test. UNLOGGED skips WAL for writes, which reduces disk pressure and cuts shadow time further on slower storage. Opt-in via unlogged_branches: true in database.yml. Default stays false to preserve current behavior; users can flip it on once they've confirmed nothing in their flow depends on WAL'd shadow tables. Benchmarks on a widgets table with 1 PK + 5 indexes (builds on the bulk-load optimisation in the previous change): 3M rows, logged bulk: 17.35s 3M rows, unlogged bulk: 14.62s (additional 1.19x) Speedup is modest on local SSD because WAL isn't the bottleneck there; larger tables, slower storage, and concurrent workloads widen the gap. Caveat: a foreign key added on a feature branch targeting a public-only table will fail when the shadow is UNLOGGED — Postgres forbids logged/unlogged FK pairs. Shadow the reference first or turn the option off. This does not affect any FKs carried through from public since LIKE ... INCLUDING CONSTRAINTS already excludes FK constraints.
7f60400 to
6cb472b
Compare
I could try and do something about the mismatch between created table in the branch schema and the unlogged tables from the copy. Would that make the suggestion more interesting? You know the size of the TFP database 🫣, I need all the performance I can cram out of it. I am just trying to find a way around where accidentally starting a branch from the folder I have production db running isn't getting in my way. Ideal? No Still, I like the work you've been doing here. Figured I should attempt to make it a better place than I found it at least. |
|
Nah, this won't be worth it. Too much complexity to get it right, I'd be better off contributing that back to postgresql instead. |
Motivation
Branch schemas are disposable by design:
db:branch:resetthrows them away and rebuilds frompublic. Every design decision in this adapter — shadow-on-demand,search_pathfallthrough,schema_migrationsisolation — treats the branch schema as transient. Postgres has an exact match for that:UNLOGGED.UNLOGGED tables skip WAL entirely. On a developer machine restoring a prod dump, that means less disk pressure, a quieter archiver, and faster shadow writes. The trade-off is that unlogged data is lost on a Postgres crash — which, for a branch schema, means running
db:branch:reset && rails db:migrateto get back to the state the previous branch was in. No worse than switching branches.Change
New
unlogged_branchesconfig option (defaultfalse). Whentrue,Shadow#clone_structureemitsCREATE UNLOGGED TABLEinstead ofCREATE TABLE. That's all — indexes on unlogged tables are automatically unlogged, so no additional plumbing.Default is
falseto preserve existing behavior. Happy to flip the default totruein a follow-up once this has some road time, given the gem is explicitly dev-only.Numbers
On top of the bulk-load approach from #1, same widgets table (1 PK + 5 indexes), local SSD:
unlogged_branches: trueLocal SSD + generous
shared_buffersmeans WAL isn't the bottleneck at these sizes. The gap widens on slower storage, larger tables, and machines running concurrent workloads (shared CI boxes, dev VMs with other busy databases).The other win, harder to put a number on: shadowing a 50M-row table writes ~30 GB of WAL under the current behavior. For a developer restoring prod nightly, that's real disk.
Caveats (documented in README)
add_foreign_key :shadowed, :public_only_tableon a feature branch fails when the shadow is unlogged and the reference is still logged inpublic. Postgres disallows mixed logged/unlogged FK pairs. Either shadow the reference first or disable the option. Pre-existing FKs carried frompublicare unaffected becauseLIKE ... INCLUDING CONSTRAINTSdoesn't copy FK constraints anyway.db:branch:reset && rails db:migrate.Tests
5 new assertions pin the expected behavior (81 runs, 180 assertions, all passing):
pg_class.relpersistence = 'p')relpersistence = 'u')publicmust never be UNLOGGEDNot addressed
create_tablestill LOGGED. Making them unlogged is more invasive (needs to interceptcreate_tablein the adapter) and runs into the same FK constraint on every FK intopublic. Worth a separate discussion.synchronous_commit = offfor shadow transactions would stack on top of this. Also a separate PR if desired.