Skip to content

Commit 188fdb1

Browse files
committed
docs(digest): describe the "**/x" quirk by its mechanism
The glob does find a root x, spelled with a double slash the walk's cleaned paths never equal; a root directory of that name keeps its name and loses its contents. The comment and the ADR said it never matched.
1 parent 6a2c3a9 commit 188fdb1

2 files changed

Lines changed: 9 additions & 5 deletions

File tree

‎docs/adr/20260911-s3-fingerprint-from-virtual-tree.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ The fingerprint format itself is fixed. An S3 snapshot must match the fingerprin
3535

3636
3. **Collisions are data errors that name every key involved.** Two keys resolving to one path, and a key under a prefix that is also an object, cannot be represented as a tree. They are detected before the digest package sees them and reported together, bounded by `maxReportedS3KeyProblems` in both directions, at most ten problem lines each naming at most ten keys, with the existing advice to use `--exclude-regex` or narrow the include filter.
3737

38-
4. **A root `.kosli_ignore` is honoured virtually.** Its rules are parsed by `digest.ParseIgnoreRules`, the same reading `DirSha256` gives the file, and resolved by `digest/virtualglob.go`, which reproduces `filepathx.Glob`, `filepath.Glob` and `filepath.Walk` step for step over the virtual tree rather than reimplementing what the globs appear to mean. That is what keeps their quirks identical: a literal `**/x` never matches at the root because the pieces concatenate to a double slash, `**/*.log` does, and excluding `logs/*` leaves an empty directory whose name is still hashed. Exclusion therefore runs inside the tree walk, not by filtering the file list. The ignore file can never exclude itself, as in `DirSha256`. `digest.FilesNeedingContent` shares that walk so excluded objects are not downloaded at all, and `VirtualDirSha256` refuses a tree that needs a digest it was not given, so a skipped download can never leak into a fingerprint. Equivalence with `DirSha256` on a materialised tree is asserted for every rule shape in `TestVirtualIgnoreTestSuite`.
38+
4. **A root `.kosli_ignore` is honoured virtually.** Its rules are parsed by `digest.ParseIgnoreRules`, the same reading `DirSha256` gives the file, and resolved by `digest/virtualglob.go`, which reproduces `filepathx.Glob`, `filepath.Glob` and `filepath.Walk` step for step over the virtual tree rather than reimplementing what the globs appear to mean. That is what keeps their quirks identical: a literal `**/x` finds a root `x` spelled with a double slash, which the walk's cleaned paths never equal, so a root file `x` survives while a root directory `x` keeps its name and loses its contents; `**/*.log` is rebuilt cleaned and matches outright; and excluding `logs/*` leaves an empty directory whose name is still hashed. Exclusion therefore runs inside the tree walk, not by filtering the file list. The ignore file can never exclude itself, as in `DirSha256`. `digest.FilesNeedingContent` shares that walk so excluded objects are not downloaded at all, and `VirtualDirSha256` refuses a tree that needs a digest it was not given, so a skipped download can never leak into a fingerprint. Equivalence with `DirSha256` on a materialised tree is asserted for every rule shape in `TestVirtualIgnoreTestSuite`.
3939

4040
5. **Downloads may run in parallel**, behind a fixed worker pool and a bytes-in-flight budget derived from listing sizes, with results written by index for determinism and a cancellable context so the first transport error stops in-flight multipart downloads. This is a performance property, not a safety one, and is delivered separately from the change this record describes; see #1167. Until it lands, downloads are sequential, exactly as before, and peak temp disk is one object.
4141

‎internal/digest/virtualglob.go‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -18,10 +18,14 @@ const virtualRoot = "tree"
1818
// DirSha256 resolves .kosli_ignore rules with filepathx.Glob, which has its own
1919
// reading of "**" and inherits filepath.Glob's: a pattern without wildcards is
2020
// returned as written, uncleaned, while one with wildcards is rebuilt from the
21-
// directory listing, cleaned. So "**/x" never matches x at the root (the pieces
22-
// concatenate to "tree//x") but "**/*.log" does. Reproducing the algorithm step
23-
// for step, rather than its apparent meaning, is what keeps the virtual digest
24-
// equal to the on-disk one for every rule anyone has already written.
21+
// directory listing, cleaned. So "**/x" does find x at the root, but spelled
22+
// "tree//x" (the pieces concatenate around the empty match), and the tree walk
23+
// compares against the cleaned "tree/x", so a root file x survives; a root
24+
// directory x keeps its name while its descendants, joined cleaned by the walk,
25+
// are excluded. "**/*.log" is rebuilt cleaned and matches outright. Reproducing
26+
// the algorithm step for step, rather than its apparent meaning, is what keeps
27+
// the virtual digest equal to the on-disk one for every rule anyone has
28+
// already written.
2529
type virtualFS struct {
2630
root *virtualNode
2731
}

0 commit comments

Comments
 (0)