Repository navigation
feat(validate): mirror collection document schemas (civitai#7491) - #865
Merged
Merged
Conversation
…(red first) Tests only, expected to FAIL on this commit. civitai/civitai#7491 adds storage.collections.<name>.schema, a JSON Schema subset every document in the collection must match. Today the vendored schema's collection object has additionalProperties:false, so `civitai app validate` refuses every manifest that uses the key. - accept arm: every keyword of the subset, an empty schema, 16 nested schemas, 16 patterns, a 256-character pattern, each min equal to its max, a shared "own" collection; - the server-only bounds, each with an accept twin at the boundary and the server's whole message: 16384 bytes of JSON.stringify output (rows for <, U+2028, control characters, quotes, multibyte and JavaScript number formatting, where Go's json.Marshal disagrees), depth 16, 16 patterns, min above max for each pair, a count above 2^53-1, a long property name cut to 80 UTF-16 units in the message, and no schema on write "none"; - patterns: one Go's RE2 cannot compile is a WARNING, never an error, and a known-gap test pins that RE2-valid but ECMAScript-invalid patterns ((?i)x, \z, \pL, [[:alpha:]]) draw nothing; - the field-coverage corpus and the field ledger gain the new findings. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…(civitai#7491) Byte-identical to public/schemas/app-block/v1.json at civitai/civitai #7491's head (3b200f1), which is ahead of what civitai.com serves until that PR ships. It adds storage.collections.*.schema, a $ref to the new $defs.storageCollectionSchema (the keyword subset), and refuses a schema on a write "none" collection. This commit alone turns the accept arm green: the collection object's additionalProperties:false no longer refuses the key. The server-only bounds (size, depth, pattern count, min above max, a count above 2^53-1, the write "none" message) and the RE2 pattern advisory stay red until the Go mirror lands. No new regex `pattern` keyword, so the gloss ledger is unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… schema internal/validate/storageschema.go ports the rules of civitai's parseCollectionSchema (civitai/civitai#7491) that the vendored JSON Schema cannot express, as ERRORS with the server's text: - at most 16384 bytes of JSON.stringify output in UTF-8, counted rather than marshalled (json.Marshal escapes <>& and U+2028/9 and writes -0); - at most 16 nested schemas, naming the innermost; - at most 16 patterns per collection schema; - no minLength/minItems/minimum above its max; - no count keyword above 2^53-1; - no schema on a write "none" collection. The vendored schema refuses it too, but its message is only "false schema". Each walk follows the server's: the size first and nothing else when it fails, depth-bounded, only collections whose visibility parsed. Patterns are not checked for parity. The server compiles them as ECMAScript with the "u" flag, and Go has RE2, a different language. A pattern RE2 cannot compile is a WARNING saying the server decides at submit. A pattern RE2 compiles is not claimed valid: (?i)x, \z, \pL and [[:alpha:]] compile in RE2 and are refused by the server (measured on Node 24), and a known-gap test pins that. The submit-time ReDoS check has no local counterpart. The help text and the README say so. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ns (red first) Tests only, expected to FAIL: civitai/civitai#7491's follow-up (64d6412) adds APP_STORAGE_MANIFEST_MAX_PATTERNS = 32, refused at parse with "storage.collections declare N patterns across their schemas — a manifest may declare at most 32 in all". These tests pin that it counts every pattern site (the same source three times is three), and only over collections that parsed (one already over its own 16 adds nothing). The field-coverage corpus and ledger gain the finding. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… see TestEveryCheckEmitsAField keys finding producers by function name. Its AST scan named the method `node`, while the runtime hook reported `(*schemaWalk).node`, so the ledger reported the method as unreached and as unknown at the same time. A plain function, walkSchemaNode, gets the same name from both. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…tai#7491 64d6412 storageManifestPatternCapChecks ports APP_STORAGE_MANIFEST_MAX_PATTERNS: at most 32 patterns across all of a manifest's collection schemas. It uses the server's message, counts every site, and counts only the collections with no error so far (the server counts the collections that parsed). The finding is on `storage.collections`, so the scope rules wait for it as they do on the server. The vendored schema is re-vendored at #7491's new head (64d6412), byte-identical. The only change is the collection `schema` description, which now names the cap and the "u" flag of the ReDoS analysis. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ax, not RE2 (red first)
The pattern advisory compiled with Go's RE2, which is uncorrelated with the
server's new RegExp(p, "u"): it warned on valid lookaround, backreferences,
\u{...}, \cA and \k<n>, and was silent on what "u" mode refuses (\-, \_, lone
braces, (?i), \z, \pL, [[:alpha:]], \Q...\E, (?P<n>...)).
Replaces TestCollectionSchemaPatternRE2AcceptsIsNotParity and the RE2 warning
test with one table pinning both lists with literal expectations. Red against
the RE2 check; the fix follows.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…tax, not RE2
The server compiles a pattern with new RegExp(p, "u"). Go's RE2 disagreed
with that in both directions, so the advisory was noise: it warned on valid
lookaround and backreferences and said nothing on what "u" mode refuses.
jsregexp.go is a conservative syntax checker for that language. It reports
only constructs "u" mode throws on (identity escapes of non-syntax
characters, \- outside a class, lone { } ], inline flag groups, (?P<...>,
\p/\P without {...}, malformed \c \x \u \k \0, unbalanced groups and classes)
and never valid syntax. It stays a WARNING: the server is authoritative,
ReDoS included, and silence is not a claim the pattern compiles.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ZacxDev
marked this pull request as ready for review
November 3, 2026 09:53
This was referenced Nov 3, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
civitai/civitai#7491 adds
storage.collections.<name>.schema, a JSON Schema subset that every document body in the collection must match (the store refuses a violating write with 400schema_violation). This bringscivitai app validatealong with it:Re-vendor
schema/app-block.manifest.schema.jsonfrom #7491 (byte-identical to what merged). Before this, the collection object'sadditionalProperties: falserefused theschemakey outright, so validate rejected every manifest using the feature. The new schema carries the keyword subset, each keyword's shape, the 256-character pattern bound and "a pattern needsmaxLength≤ 1000" ($defs.storageCollectionSchema). No new regexpatternkeyword, so the gloss ledger is unchanged.internal/validate/storageschema.goports the server'sparseCollectionSchemarules that the JSON Schema cannot express, as ERRORS with the server's text:JSON.stringifyoutput in UTF-8. This is counted, not computed withjson.Marshal, which escapes<>&and U+2028/9 and writes-0.write: "none"collection. The JSON Schema refuses this too, but its message is only "false schema", so the server's text is added next to it (the same approach asbuildCommandRe).Patterns: a warning, not parity. The server compiles a pattern with
new RegExp(p, "u"). Go's RE2 disagrees with that language in both directions, so it is not used.internal/validate/jsregexp.gois a conservative syntax checker for ECMAScriptumode: it reports only constructsumode throws on, as a WARNING with the reason, and says nothing on valid syntax. Every row below was checked against Node 24.21 withu:umode\-outside a class,\_,\:,\#,\@,\z,\Q…\E, lone{}],a{,a{,3},[[:alpha:]],(?i),(?s),\pL,(?P<n>x), unbalanced()[, malformed\x\u\c\k\0,[\B]umode\1,\k<n>,\u{1F600},\cA,\p{L},[\-],\d\w\s\b\B,\/,{2,3}It stays a warning because the server is authoritative. A pattern the checker says nothing about is not proven valid: it does not check backreferences to missing groups, group or property names, quantifier bounds, or class-range order, and the server's submit-time ReDoS check (recheck) has no local counterpart. The help text and the README say so.
Tests (red first)
e14457b: tests only, RED.7eb6439: the re-vendor alone. The accept arm goes green; the server-only rules stay red.4a333e3: the Go mirror. The new tests are green, but the field-coverage ledger went red: it could not name a method finding-producer.84e269d: fixes that ledger. With24f7c55it is RED only on the 32-pattern manifest cap.894490c: the cap mirror plus a re-vendor at64d6412.834e301: the pattern table (both lists, literal expectations), tests only, RED against the RE2 check.91794d0: theu-mode syntax checker replaces RE2.Expected values are literals taken from the server's tests and from Node's
Buffer.byteLength(JSON.stringify(...)). They were not derived from the Go. The field-coverage corpus and the field ledger gain the new findings.Not covered offline
🤖 Generated with Claude Code