docs: add request path, cache tiering, and git restore docs

Amp-Thread-ID: https://ampcode.com/threads/T-019f8629-56a0-727e-826f-c14bdade11f1
Co-authored-by: Amp <amp@ampcode.com>

Author: worstell

README.md

@@ -18,7 +18,8 @@ Redirect Git traffic through cachew:
   insteadOf = https://github.com/
 ```
 
-Restore a repository from a snapshot (with automatic delta bundle to reach HEAD):
+Restore a repository from a snapshot (with automatic delta bundle to reach HEAD); see
+[docs/git-restore.md](docs/git-restore.md) for how the restore flow and parallel snapshot downloads work:
 
 ```sh
 cachew git restore https://github.com/org/repo ./repo
@@ -109,7 +110,9 @@ Multiple backends can be configured simultaneously — they are automatically co
 are ordered from lowest/nearest to highest/authoritative. Reads check each tier in order and backfill lower tiers on a
 hit. Writes go to all tiers in parallel. Replica invalidations evict only non-authoritative tiers; the final cache block
 is authoritative. Tiered caches use the metadata backend to track authoritative ETags and invalidate stale lower-tier
-copies before falling through to the authoritative tier.
+copies before falling through to the authoritative tier. See [docs/tiering.md](docs/tiering.md) for a full explanation
+of the tiering semantics, and [docs/architecture.md](docs/architecture.md) for how requests flow through strategies to
+the cache.
 
 ### Memory
 

docs/architecture.md

@@ -0,0 +1,62 @@
+# Architecture: the request path
+
+How an HTTP request flows through Cachew, from client to upstream.
+
+```
+┌────────┐   ┌──────┐   ┌──────────────┐   ┌──────────────────┐   ┌───────┐
+│ Client │──▶│ OPA  │──▶│ Interceptors │──▶│ Strategy (route) │──▶│ Cache │
+└────────┘   └──────┘   └──────────────┘   └────────┬─────────┘   └───────┘
+                                                    │ miss
+                                                    ▼
+                                              ┌──────────┐
+                                              │ Upstream │
+                                              └──────────┘
+```
+
+- **OPA** authorizes every request (`internal/opa`).
+- **Strategies** are protocol-aware handlers (Git, GitHub releases, Go
+  modules, Hermit, Artifactory, host, proxy). Each is registered against the
+  mux at config load (`internal/config/config.go`); strategies implementing
+  `strategy.Interceptor` wrap the mux instead, so they can inspect the raw
+  request line.
+- Each strategy receives a **namespaced view** of the cache
+  (`cache.Namespace`), so strategies never collide on keys.
+- Most strategies use the shared handler (`internal/strategy/handler`), which
+  implements the cache-or-fetch loop: look up the key, serve from cache on a
+  hit, and on a miss stream the upstream response to the client and the cache
+  simultaneously. Strategies are not limited to the Cache — e.g. the Git
+  strategy also maintains bare clones and serves packs directly.
+- The **api-v1 strategy** (`internal/strategy/apiv1.go`) exposes the composed
+  cache itself over HTTP (`/api/v1/object/{namespace}/{key}`, …). This is the
+  API that `client/` and the `Remote` cache implementation speak, and it is
+  what makes instance-to-instance tiering possible.
+
+Strategies see a single `Cache`; whether it is one backend or several tiers
+composed together is invisible to them. That composition is described in
+[tiering.md](tiering.md).
+
+## Ranged and parallel downloads
+
+`client.ParallelGet` (`client/parallel_get.go`) downloads a large object as
+many concurrent ETag-pinned byte-range requests instead of one stream. Its
+main consumer is `cachew git restore`'s snapshot download — see
+[git-restore.md](git-restore.md) for the full semantics. The same machinery is
+reused inside the S3 backend (`internal/cache/s3_parallel_get.go`): a single
+S3 stream is limited to a fraction of the available bandwidth, so whole-object
+and large ranged reads fan out into parallel sub-range requests against the
+pinned object revision.
+
+Ranged reads interact with cache tiering — a partial body must never be
+backfilled into a lower tier as if it were the whole object; see
+[tiering.md](tiering.md).
+
+## The Cache contract
+
+Strategies and cache backends all program against the same interface
+(`internal/cache/api.go`), whose key guarantees are:
+
+- Expired objects are never returned.
+- Objects are invisible until completely written and closed.
+- `Delete` is atomic; missing objects invalidate successfully.
+- Conditional options (`If-None-Match`, `If-Match`, `If-Range`, `Range`) are
+  evaluated against the stored ETag with RFC 9110 semantics.

docs/git-restore.md

@@ -0,0 +1,50 @@
+# Git restore: snapshots, bundles, and parallel downloads
+
+`cachew git restore <repo-url> <dir>` recreates a working tree from the
+server's cached artifacts instead of running `git clone`. The flow
+(`cmd/cachew/git.go`):
+
+1. **Snapshot download** — fetch `/git/{repo}/snapshot.tar.zst`, a periodic
+   `.tar.zst` archive of the working tree, and pipe it straight into
+   extraction so decompression overlaps the transfer. Extraction goes into a
+   staging directory renamed into place on success, so a failed download never
+   leaves a half-written checkout.
+2. **Delta bundle** — the snapshot response advertises a bundle URL
+   (`X-Cachew-Bundle-Url`) covering commits from the snapshot's commit to the
+   mirror's current HEAD. The client fetches and applies it to catch up
+   without talking to the upstream. A bundle failure is a warning, not an
+   error — the restore continues with the snapshot state.
+3. **Freshen** (only with `--ref`/`--commit`) — ask the server to ensure the
+   required refs/commits exist on its mirror, then `git pull --ff-only` from
+   origin so the working tree catches up. Skipped entirely when the
+   snapshot+bundle already contain everything requested.
+
+## Parallel snapshot download
+
+The snapshot is downloaded as many concurrent byte-range requests
+(`client.ParallelGet`, driven by `--download-concurrency` and
+`--download-chunk-size-mb`):
+
+- **Discovery**: the first chunk is requested with a `Range` header; the
+  response reveals the object's total size and ETag, plus the snapshot
+  metadata headers (commit, bundle URL). The git strategy advertises these on
+  both full (200) and ranged (206) responses, so the client learns everything
+  it needs from this one response (`internal/strategy/git/snapshot.go`).
+- **Pinning**: every subsequent chunk carries `If-Match` with the discovery
+  ETag, so an object rewritten mid-download (e.g. by the periodic snapshotter)
+  is rejected rather than spliced together from two revisions.
+- **Streaming**: chunks complete out of order and are reassembled into a
+  sequential stream with bounded buffering, feeding `tar` extraction as bytes
+  arrive.
+
+## Fallbacks and failure modes
+
+Single-stream fallback happens at discovery time only: a server that ignores
+the range, an object with no ETag to pin to, an object that fits in the first
+chunk, or `--download-concurrency=1` all degrade to one full read.
+
+After discovery, a mid-download rewrite is **fatal**: pinned chunks fail their
+`If-Match` precondition and the download errors rather than delivering a
+corrupt archive. The client does not retry single-stream and does not fall
+back to `git clone` — retrying the restore (or cloning) is the caller's
+decision. Bundle failures, by contrast, are non-fatal as described above.

docs/tiering.md

@@ -0,0 +1,88 @@
+# How cache tiering works
+
+How multiple cache backends are composed into a single tiered cache with
+fallback, backfill, and invalidation semantics. For how a request reaches the
+cache in the first place, see [architecture.md](architecture.md). Source of
+truth: `internal/cache/tiered.go` and `internal/cache/api.go`.
+
+Cache backends are object stores with per-object TTLs and metadata. Three are
+configurable: `memory`, `disk`, and `s3`. Configuring more than one `cache`
+block composes them into a `Tiered` cache automatically
+(`cache.MaybeNewTiered`); with a single block the cache is wrapped so that
+`Invalidate` is a no-op, since the only tier is authoritative.
+
+**Order matters.** Cache blocks are ordered nearest-first. The **final tier is
+authoritative** — typically shared storage like S3 — and everything before it
+is treated as a local copy that can be re-fetched.
+
+```
+cache memory { }          # tier 0: nearest, fastest
+cache disk   { }          # tier 1
+cache s3     { ... }      # tier 2: authoritative
+```
+
+```
+        read: probe in order, first hit wins
+        ────────────────────────────────────▶
+┌────────┐      ┌──────┐      ┌────────────────┐
+│ memory │      │ disk │      │ s3 (authorit.) │
+└────────┘      └──────┘      └────────────────┘
+        ◀────────────────────────────────────
+        backfill tier 0 on a deeper hit
+
+        write: all tiers in parallel
+```
+
+## Reads
+
+`Open`/`Stat` probe tiers in order and return the first definitive answer.
+When a deeper tier hits, the returned reader transparently **backfills tier
+0** as the caller reads (`backfillReadCloser`), so the next read is served
+locally. Backfill is asynchronous and safe: only a stream consumed to EOF
+commits the tier-0 entry; a partial read or mid-stream error discards it.
+
+Conditional requests complicate "definitive". A tier holding a *different
+version* than the request's validators name (failed `If-Match`, `If-Range`
+miss) is not a definitive miss — deeper tiers are consulted for the named
+version. Only when no tier holds it does the first tier's outcome stand. A
+tier that errored while being probed takes precedence, so outages are not
+misreported as missing versions.
+
+Ranged reads return partial bodies, which must never be backfilled as whole
+objects. Instead a bounded background **healer** re-fetches the full object
+from the serving tier and refreshes tier 0 out of band, so a divergent tier 0
+still converges even when clients only ever issue ranged requests — as the
+parallel snapshot downloader does (see [architecture.md](architecture.md)).
+
+## Writes
+
+`Create` writes to **all tiers in parallel** through a single writer; the
+first error aborts every in-flight write. Entries only become readable once
+completely written and closed — a cancelled context discards the object in
+every tier.
+
+## ETags and stale-tier invalidation
+
+The tiered cache records the authoritative ETag for each key in the metadata
+store (the `metadata` block; see [metadatadb-s3.md](metadatadb-s3.md)). On
+reads, a tier whose ETag no longer matches the recorded authoritative ETag is
+**invalidated and skipped**, falling through to the next tier. This is how a
+replica whose local tiers have diverged converges back onto the shared tier.
+
+## Delete vs Invalidate
+
+- `Delete` removes the object from **every** tier.
+- `Invalidate` evicts stale copies from the **non-authoritative** tiers only;
+  the final tier is left intact by construction. This is what replicas use to
+  drop local copies without destroying shared state.
+
+## Instance-to-instance tiering
+
+Cachew is designed to run as a local instance (workstation/CI) backed by a
+shared remote instance. The pieces:
+
+- The remote instance serves its cache via the **api-v1 strategy** (see
+  [architecture.md](architecture.md)).
+- `client/` is a standalone Go client for that API, and
+  `internal/cache/remote.go` adapts it to the `Cache` interface — a remote
+  Cachew instance as a cache tier.