Skip to content

feat(validate): mirror collection document schemas (civitai#7491) - #865

Merged
ZacxDev merged 8 commits into
mainfrom
feat/storage-collection-schema
Nov 3, 2026
Merged

ZacxDev merged 8 commits into
mainfrom
feat/storage-collection-schema

Conversation

@ZacxDev

@ZacxDev ZacxDev commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

civitai/civitai#7491 (per-collection document schemas) merged as a6053a9a, and civitai.com serves its manifest schema: schema-drift is green against the vendored copy here.

What

civitai/civitai#7491 adds storage.collections.<name>.schema, a JSON Schema subset that every document body in the collection must match (the store refuses a violating write with 400 schema_violation). This brings civitai app validate along with it:

  1. Re-vendor schema/app-block.manifest.schema.json from #7491 (byte-identical to what merged). Before this, the collection object's additionalProperties: false refused the schema key outright, so validate rejected every manifest using the feature. The new schema carries the keyword subset, each keyword's shape, the 256-character pattern bound and "a pattern needs maxLength ≤ 1000" ($defs.storageCollectionSchema). No new regex pattern keyword, so the gloss ledger is unchanged.

  2. internal/validate/storageschema.go ports the server's parseCollectionSchema rules that the JSON Schema cannot express, as ERRORS with the server's text:

    • at most 16384 bytes of JSON.stringify output in UTF-8. This is counted, not computed with json.Marshal, which escapes <>& and U+2028/9 and writes -0.
    • at most 16 nested schemas (the message names the innermost one)
    • at most 16 patterns per collection schema, and at most 32 across all of the manifest's collection schemas. This counts every place a pattern appears, and only in collections that are otherwise valid.
    • minLength/minItems/minimum not above its max
    • a count keyword not above 2^53−1
    • no schema on a write: "none" collection. The JSON Schema refuses this too, but its message is only "false schema", so the server's text is added next to it (the same approach as buildCommandRe).
  3. Patterns: a warning, not parity. The server compiles a pattern with new RegExp(p, "u"). Go's RE2 disagrees with that language in both directions, so it is not used. internal/validate/jsregexp.go is a conservative syntax checker for ECMAScript u mode: it reports only constructs u mode throws on, as a WARNING with the reason, and says nothing on valid syntax. Every row below was checked against Node 24.21 with u:

    patterns CLI
    refused by u mode \- outside a class, \_, \:, \#, \@, \z, \Q…\E, lone { } ], a{, a{,3}, [[:alpha:]], (?i), (?s), \pL, (?P<n>x), unbalanced ( ) [, malformed \x \u \c \k \0, [\B] warning
    valid in u mode lookaround, \1, \k<n>, \u{1F600}, \cA, \p{L}, [\-], \d\w\s\b\B, \/, {2,3} nothing

    It stays a warning because the server is authoritative. A pattern the checker says nothing about is not proven valid: it does not check backreferences to missing groups, group or property names, quantifier bounds, or class-range order, and the server's submit-time ReDoS check (recheck) has no local counterpart. The help text and the README say so.

Tests (red first)

  • e14457b: tests only, RED.
  • 7eb6439: the re-vendor alone. The accept arm goes green; the server-only rules stay red.
  • 4a333e3: the Go mirror. The new tests are green, but the field-coverage ledger went red: it could not name a method finding-producer.
  • 84e269d: fixes that ledger. With 24f7c55 it is RED only on the 32-pattern manifest cap.
  • 894490c: the cap mirror plus a re-vendor at 64d6412.
  • 834e301: the pattern table (both lists, literal expectations), tests only, RED against the RE2 check.
  • 91794d0: the u-mode syntax checker replaces RE2.

Expected values are literals taken from the server's tests and from Node's Buffer.byteLength(JSON.stringify(...)). They were not derived from the Go. The field-coverage corpus and the field ledger gain the new findings.

Not covered offline

  • pattern semantics past syntax (see item 3)
  • ReDoS
  • document validation itself. The CLI checks manifests, not stored documents.

🤖 Generated with Claude Code

ZacxDev and others added 3 commits November 2, 2026 14:48
…(red first)

Tests only, expected to FAIL on this commit. civitai/civitai#7491 adds
storage.collections.<name>.schema, a JSON Schema subset every document
in the collection must match. Today the vendored schema's collection
object has additionalProperties:false, so `civitai app validate`
refuses every manifest that uses the key.

- accept arm: every keyword of the subset, an empty schema, 16 nested
  schemas, 16 patterns, a 256-character pattern, each min equal to its
  max, a shared "own" collection;
- the server-only bounds, each with an accept twin at the boundary and
  the server's whole message: 16384 bytes of JSON.stringify output
  (rows for <, U+2028, control characters, quotes, multibyte and
  JavaScript number formatting, where Go's json.Marshal disagrees),
  depth 16, 16 patterns, min above max for each pair, a count above
  2^53-1, a long property name cut to 80 UTF-16 units in the message,
  and no schema on write "none";
- patterns: one Go's RE2 cannot compile is a WARNING, never an error,
  and a known-gap test pins that RE2-valid but ECMAScript-invalid
  patterns ((?i)x, \z, \pL, [[:alpha:]]) draw nothing;
- the field-coverage corpus and the field ledger gain the new findings.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…(civitai#7491)

Byte-identical to public/schemas/app-block/v1.json at civitai/civitai
#7491's head (3b200f1), which is ahead of what civitai.com serves
until that PR ships. It adds storage.collections.*.schema, a $ref to
the new $defs.storageCollectionSchema (the keyword subset), and refuses
a schema on a write "none" collection.

This commit alone turns the accept arm green: the collection object's
additionalProperties:false no longer refuses the key. The server-only
bounds (size, depth, pattern count, min above max, a count above
2^53-1, the write "none" message) and the RE2 pattern advisory stay
red until the Go mirror lands. No new regex `pattern` keyword, so the
gloss ledger is unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… schema

internal/validate/storageschema.go ports the rules of civitai's
parseCollectionSchema (civitai/civitai#7491) that the vendored JSON
Schema cannot express, as ERRORS with the server's text:

- at most 16384 bytes of JSON.stringify output in UTF-8, counted rather
  than marshalled (json.Marshal escapes <>& and U+2028/9 and writes -0);
- at most 16 nested schemas, naming the innermost;
- at most 16 patterns per collection schema;
- no minLength/minItems/minimum above its max;
- no count keyword above 2^53-1;
- no schema on a write "none" collection. The vendored schema refuses
  it too, but its message is only "false schema".

Each walk follows the server's: the size first and nothing else when it
fails, depth-bounded, only collections whose visibility parsed.

Patterns are not checked for parity. The server compiles them as
ECMAScript with the "u" flag, and Go has RE2, a different language. A
pattern RE2 cannot compile is a WARNING saying the server decides at
submit. A pattern RE2 compiles is not claimed valid: (?i)x, \z, \pL and
[[:alpha:]] compile in RE2 and are refused by the server (measured on
Node 24), and a known-gap test pins that. The submit-time ReDoS check
has no local counterpart. The help text and the README say so.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ZacxDev and others added 5 commits November 2, 2026 15:07
…ns (red first)

Tests only, expected to FAIL: civitai/civitai#7491's follow-up
(64d6412) adds APP_STORAGE_MANIFEST_MAX_PATTERNS = 32, refused at parse
with "storage.collections declare N patterns across their schemas — a
manifest may declare at most 32 in all". These tests pin that it counts
every pattern site (the same source three times is three), and only
over collections that parsed (one already over its own 16 adds
nothing). The field-coverage corpus and ledger gain the finding.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… see

TestEveryCheckEmitsAField keys finding producers by function name. Its
AST scan named the method `node`, while the runtime hook reported
`(*schemaWalk).node`, so the ledger reported the method as unreached
and as unknown at the same time. A plain function, walkSchemaNode, gets
the same name from both.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…tai#7491 64d6412

storageManifestPatternCapChecks ports APP_STORAGE_MANIFEST_MAX_PATTERNS:
at most 32 patterns across all of a manifest's collection schemas. It
uses the server's message, counts every site, and counts only the
collections with no error so far (the server counts the collections
that parsed). The finding is on `storage.collections`, so the scope
rules wait for it as they do on the server.

The vendored schema is re-vendored at #7491's new head (64d6412),
byte-identical. The only change is the collection `schema` description,
which now names the cap and the "u" flag of the ReDoS analysis.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ax, not RE2 (red first)

The pattern advisory compiled with Go's RE2, which is uncorrelated with the
server's new RegExp(p, "u"): it warned on valid lookaround, backreferences,
\u{...}, \cA and \k<n>, and was silent on what "u" mode refuses (\-, \_, lone
braces, (?i), \z, \pL, [[:alpha:]], \Q...\E, (?P<n>...)).

Replaces TestCollectionSchemaPatternRE2AcceptsIsNotParity and the RE2 warning
test with one table pinning both lists with literal expectations. Red against
the RE2 check; the fix follows.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…tax, not RE2

The server compiles a pattern with new RegExp(p, "u"). Go's RE2 disagreed
with that in both directions, so the advisory was noise: it warned on valid
lookaround and backreferences and said nothing on what "u" mode refuses.

jsregexp.go is a conservative syntax checker for that language. It reports
only constructs "u" mode throws on (identity escapes of non-syntax
characters, \- outside a class, lone { } ], inline flag groups, (?P<...>,
\p/\P without {...}, malformed \c \x \u \k \0, unbalanced groups and classes)
and never valid syntax. It stays a WARNING: the server is authoritative,
ReDoS included, and silence is not a claim the pattern compiles.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@ZacxDev ZacxDev changed the title feat(validate): mirror collection document schemas (waits on civitai#7491) feat(validate): mirror collection document schemas (civitai#7491) Nov 3, 2026
@ZacxDev
ZacxDev marked this pull request as ready for review November 3, 2026 09:53
@ZacxDev
ZacxDev merged commit 34a7b08 into main Nov 3, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant