feat(upstash): rewrite store as verified shard lists with batched I/O - #52
Closed
haydenshively wants to merge 3 commits into
Closed
feat(upstash): rewrite store as verified shard lists with batched I/O#52haydenshively wants to merge 3 commits into
haydenshively wants to merge 3 commits into
Conversation
Every value is a Redis LIST of ≤ shardBytes self-indexed shards. Small values are written atomically by a WRITE_DIRECT script, many per HTTP request; large values stage at a per-writer tmp:<key>:<uuid> list (60 s TTL refreshed per push) and go live through a verifying PUBLISH script. Readers check wid uniformity and index contiguity, so they observe either the old value or one complete new one. TTLs are absolute deadlines, so client replays cannot extend them. Fixes replay duplication and mid-write tmp expiry in the previous RENAME-based writer. Adds plural mget/mset/mdelete with exact request-body measurement and pipeline packing; get/set/delete remain as deprecated adapters. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
haydenshively
changed the title
Replace the self-referential measurement test with one that pins direct/staged routing to the exact byte; add packing-pressure coverage; assert script result codes, attempt counts, and identical replayed bodies instead of eventual readback alone. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Move ./plans to a flat, tracked ./docs where the filename carries the kind and the package version the document was written against (NNNNNN-<kind>-<slug>.md, two digits per semver component). The version is a baseline, not a release, so the src/foo.ts:42 references inside a brief stay resolvable and are never refreshed as code moves. Each brief follows one spine: Intent, Context, Design, Scope & files, Verification, Open risks, Notes, Derivation. Notes justifies the design as stated; Derivation holds what exists only because the path was circuitous — alternatives declined, work deferred, findings surfaced along the way. Prose is preserved as written; sections were renamed and reordered, not rewritten. README.md registers the tib and ref prefixes, the frontmatter fields, and the spine. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
cashd
approved these changes
Aug 27, 2026
Collaborator
Author
|
Useful reference implementation, but closing because it's not directly relevant to anything the library needs. If downstream consumers want to build something like this, it's simple enough for them to expose |
haydenshively
added a commit
that referenced
this pull request
Aug 28, 2026
Docs-only. Extracted from #52, whose code half was closed — so the doc backfill never landed. Adds two files: - **`docs/000012-tib-paged-lenses-partial-results.md`** — the design document for the paged-lens work that shipped in `07e49df` and `0df02a9`: RETURN mode deleted, the `(U[] results, uint256[] skipped)` lens contract and its four clauses, the client fill loop, and the decision to respond in the shape the policy's `abi` declares. Written against the `0.0.12` tree, so its `src/foo.ts:42` references resolve there, not against `main`. - **`docs/README.md`** — the conventions for this directory: the `NNNNNN-<kind>-<slug>.md` filename, the version-as-baseline rule, TIB frontmatter (`landed` present only once work ships), the fixed section spine, and the Notes/Derivation split. The index table lists only the document included here; the two `000016` rows from the original branch would have been broken links. The illustrative `000016-*` filenames inside the format examples are left as-is — they're format illustrations, not links. No source or test changes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/morpho-org/viem-dlc/pull/53" target="_blank"> <picture> <source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=3"> <img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=3" alt="Devin Review"> </picture> </a> <!-- devin-review-badge-end --> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Rewrites
UpstashStorearound a single storage format and two Lua scripts, fixing two live bugs in theRENAME-based writer:RPUSHresponse replayed the push, andRENAMEpublished a list with a duplicate shard.RENAMEpublished a truncated list.Design
shardBytes(default 64 KiB) shards, each carryingwid|i|k|/wid|i|headers. Readers verify wid uniformity and index contiguity, so a read is either the old value or one complete new value.WRITE_DIRECTscript): whole value in one atomicEVALSHA; many keys packed into one/pipelinerequest.tmp:<key>:<uuid>list,MULTI RPUSH + EXPIRE 60per batch, then aPUBLISHscript that verifies exact head +LLEN == kbeforeRENAME. One restart (fresh uuid + wid) allowed before publish; never restaged after.PEXPIREAT) computed once, so replays cannot extend them.floor(maxResponseBytes / shardBytes)elements per request.NOSCRIPTrecovery (SCRIPT LOADonce, reissue only failed slots).API
mget/mset/mdelete;get/set/deletekept as deprecated one-liners until the pluralStorecontract lands.UpstashStoreOptionsgainsshardBytes?andmaxResponseBytes?.maxRequestBytesis now an exact bound rather than "value bytes plus headroom".createOptimizedUpstashStoreunchanged.src/stores/upstash.internal.ts(not in theexportsmap).Tests
test/helpers/fake-upstash.tsis an in-memory Upstash REST endpoint driven through the real@upstash/redisclient (stubbedfetch), with virtual time, lazy expiry, fault injection, and delayed replay. 83 tests cover thePUBLISHreturn-code matrix, codec fuzzing, batching/budget assertions, NOSCRIPT recovery, deadline immutability under replay, freeze/thaw, the recombined-list case publication accepts but readers reject, and a randomized keyspace audit.upstash.integration.test.tsruns against a live database whenUPSTASH_REDIS_REST_URL/_TOKENare set — the Lua itself is only exercised there.🤖 Generated with Claude Code