From 0b8294bc1973b269c1fd958b77f418da4e7e0c09 Mon Sep 17 00:00:00 2001 From: "Derrick J. Wippler" Date: Fri, 23 Oct 2026 23:16:39 -0500 Subject: [PATCH 1/4] docs: add Design section to README and fix hot cache size --- README.md | 89 ++++++++++++++++++++++++++++++++++++------------------- 1 file changed, 58 insertions(+), 31 deletions(-) diff --git a/README.md b/README.md index 94301c9..8c09f7f 100644 --- a/README.md +++ b/README.md @@ -13,34 +13,61 @@ Distributed Cache Library Groupcache is a Go-based caching and cache-filling library designed to replace traditional caching solutions like MEMCACHED and REDIS in many scenarios. -## Why Use Groupcache? -- Cost Reduction: Eliminates the need for external system dependencies and additional server infrastructure -- Increased efficiency: Minimizes network calls to external systems and data sources -- Load Protection: Prevents the thundering herd problem through the use of `singleflight` synchronization - -## How It Works -Groupcache functions as an in-memory read-through cache with the following workflow: -- When your application requests data, Groupcache uses key sharding to determine key ownership -- For locally-owned keys: - - If the key exists in the local cache, data is returned immediately - - If the key is missing, Groupcache retrieves it from the data source, caches it, and returns it - - Future requests for the same key are served directly from the hot cache -- For keys owned by other peers: - - Groupcache forwards the request to the appropriate peer instance - - The owning peer handles the cache lookup and retrieval process - -This architecture provides the following benefits: -- Network efficiency by avoiding requests for locally-owned keys -- Load protection by channeling identical key requests to a single Groupcache instance within the cluster -- Avoid unnecessary network calls to external cache systems by using local memory for the cache - -Thanks to `singleflight` synchronization, even if your application makes millions of requests for a -specific key (e.g., "FOO"), Groupcache will only query the underlying data source once. +## Design +Groupcache is a library, not a service. Every process that imports it is both a client and a cache +server, and the processes find each other through the peer list you give them. There is nothing +else to deploy.
Simplified Diagram
+**Every key has one owner.** Peers sit on a [consistent hash ring](transport/peer/picker.go), so +each key maps to exactly one peer. When peers join or leave, only a small share of keys change +owner. Call [`SetPeers()`](instance.go) whenever the topology changes. + +**Misses load through your code.** You give each group a [`Getter`](group.go) that fetches a value from +your database or service. On a miss the owner calls it, stores the result, and returns it. A +`Get()` never asks the caller to fill the cache. + +**One load per key.** Concurrent requests for a missing key wait on a single in-flight load, through +[`singleflight`](internal/singleflight/singleflight.go). A million simultaneous `Get("FOO")` calls on +one peer run your `Getter` once. Requests that arrive at the owner from other peers join the same load, +so the cluster normally runs it once. + +**Peers fall back to loading locally.** If a peer can't get an answer from the owner (a refused +connection, a transport timeout, a bad response), it loads the value itself, still with in-flight +dedup, instead of failing the request. The fallback is skipped when the owner answers "not found" or +"unavailable" (HTTP 404 and 503), or when the caller's own context is cancelled or past its deadline. + +**Popular keys are copied.** Each group has a main cache for keys this peer loaded or was sent by a `Set()`, and a hot cache for +copies of keys fetched from other peers. A key that every peer asks for ends up in every peer's hot +cache, so one owner doesn't take all the traffic. The hot cache is 1/8 of the group's memory limit +and the main cache is 7/8. + +**Writes reach the cluster.** [`Set()`](group.go) writes to the owner and then to every other peer. +`Remove()` and `RemoveKeys()` clear the key on the owner and on every other peer. Delivery is +best effort, so put an expiry on values to bound how long a missed update can last. + +**Expiry is checked on read.** A value can carry an expiry time, and the cache drops it when a `Get()` +finds it expired. No peer coordinates expiry, which means clocks across the cluster should be in sync. + +**Everything with a seam is pluggable.** +- Caches. The default is an LRU. The [Otter cache](contrib/otter.go) uses lock-free S3-FIFO eviction and suits + machines with many cores. See [Pluggable Internal Cache](#pluggable-internal-cache). +- Transports. HTTP is the default, and you can write your own against the + [`transport.Transport`](transport/http_transport.go) interface. +- Hash function and replica count for the ring. + +**Observable.** Group and cache stats are exposed as OpenTelemetry metrics, and the HTTP transport +can emit OpenTelemetry traces for peer calls. + +**Many instances per process.** There is no global state, so tests (and the [`cluster`](cluster) +package) can run a whole cluster inside one process. + +The sequence diagram under [The Cache Loading process](#the-cache-loading-process) shows these +steps for a single `Get()`. + ## Usage ```go @@ -150,16 +177,16 @@ In a nutshell, a groupcache lookup of **Get("foo")** looks like: (On machine #5 of a set of N machines running the same code) -1. Is the value of FOO in local cache because it's super hot? If so, use it. -2. Is the value of FOO in local memory because peer #5 (the current - peer) is the owner of it? If so, use it. +1. Is the value of FOO in local memory because peer #5 (the current + peer) loaded it or was sent it by a `Set()`? If so, use it. +2. Is the value of FOO in the local hot cache because it's super hot? If so, use it. 3. Amongst all the peers in my set of N, am I the owner of the key FOO? (e.g. does it consistent hash to 5?) If so, load it and store in the local cache. If other callers come in, via the same process or via RPC requests from peers, they block waiting for the load to finish and get the same answer. If not, RPC to the peer that's the - owner and get the answer. If the RPC fails, just load it locally (still with - local dup suppression). + owner and get the answer. If the RPC fails to get an answer (see Design for the + exceptions), just load it locally (still with local dup suppression). Simplified Diagram
@@ -345,9 +372,9 @@ accessed by other peers in the cluster. ### groupcache.Cache Is the cache implementation used for both the "hot" and "main" caches. The "main cache" stores items the local -group "owns" and the "hot cache" stores items this instance has retrieved from a remote peer. The "main cache" is -7/8th the size of the max bytes requested when calling `Instance.NewGroup()`. The "hot cache" is 1/8th the size of -the "main cache". +group loaded as owner or was sent by a `Set()`, and the "hot cache" stores items this instance has retrieved from a remote peer. The "main cache" is +7/8th the size of the max bytes requested when calling `Instance.NewGroup()`. The "hot cache" is 1/8th of the max +bytes requested, which makes it 1/7th the size of the "main cache". Third party cache implementations can be used by providing an `Options.CacheFactory` function which returns the third party initialized cache of the requested size. Groupcache includes an optional From bb4565917f20a085a5bd89efd467c7845edace7b Mon Sep 17 00:00:00 2001 From: "Derrick J. Wippler" Date: Fri, 23 Oct 2026 23:32:21 -0500 Subject: [PATCH 2/4] docs: restore Why Use Groupcache and reword Design --- README.md | 92 +++++++++++++++++++++++++++++-------------------------- 1 file changed, 48 insertions(+), 44 deletions(-) diff --git a/README.md b/README.md index 8c09f7f..f60d52c 100644 --- a/README.md +++ b/README.md @@ -13,60 +13,64 @@ Distributed Cache Library Groupcache is a Go-based caching and cache-filling library designed to replace traditional caching solutions like MEMCACHED and REDIS in many scenarios. +## Why Use Groupcache? +- Cost Reduction: Eliminates the need for external system dependencies and additional server infrastructure +- Increased efficiency: Minimizes network calls to external systems and data sources +- Load Protection: Prevents the thundering herd problem through the use of `singleflight` synchronization + ## Design -Groupcache is a library, not a service. Every process that imports it is both a client and a cache -server, and the processes find each other through the peer list you give them. There is nothing -else to deploy. +Most caches are a separate fleet of servers you have to run, size, and keep alive. Groupcache is a +library. Import it, hand each process a list of its peers, and your app servers become the cache. +There is nothing else to deploy.
Simplified Diagram
-**Every key has one owner.** Peers sit on a [consistent hash ring](transport/peer/picker.go), so -each key maps to exactly one peer. When peers join or leave, only a small share of keys change -owner. Call [`SetPeers()`](instance.go) whenever the topology changes. - -**Misses load through your code.** You give each group a [`Getter`](group.go) that fetches a value from -your database or service. On a miss the owner calls it, stores the result, and returns it. A -`Get()` never asks the caller to fill the cache. - -**One load per key.** Concurrent requests for a missing key wait on a single in-flight load, through -[`singleflight`](internal/singleflight/singleflight.go). A million simultaneous `Get("FOO")` calls on -one peer run your `Getter` once. Requests that arrive at the owner from other peers join the same load, -so the cluster normally runs it once. - -**Peers fall back to loading locally.** If a peer can't get an answer from the owner (a refused -connection, a transport timeout, a bad response), it loads the value itself, still with in-flight -dedup, instead of failing the request. The fallback is skipped when the owner answers "not found" or -"unavailable" (HTTP 404 and 503), or when the caller's own context is cancelled or past its deadline. - -**Popular keys are copied.** Each group has a main cache for keys this peer loaded or was sent by a `Set()`, and a hot cache for -copies of keys fetched from other peers. A key that every peer asks for ends up in every peer's hot -cache, so one owner doesn't take all the traffic. The hot cache is 1/8 of the group's memory limit -and the main cache is 7/8. - -**Writes reach the cluster.** [`Set()`](group.go) writes to the owner and then to every other peer. -`Remove()` and `RemoveKeys()` clear the key on the owner and on every other peer. Delivery is -best effort, so put an expiry on values to bound how long a missed update can last. - -**Expiry is checked on read.** A value can carry an expiry time, and the cache drops it when a `Get()` -finds it expired. No peer coordinates expiry, which means clocks across the cluster should be in sync. - -**Everything with a seam is pluggable.** -- Caches. The default is an LRU. The [Otter cache](contrib/otter.go) uses lock-free S3-FIFO eviction and suits - machines with many cores. See [Pluggable Internal Cache](#pluggable-internal-cache). -- Transports. HTTP is the default, and you can write your own against the +**Every key has one home.** Peers sit on a [consistent hash ring](transport/peer/picker.go), so any +peer can tell, with no lookup, which peer owns a key. When a peer joins or leaves, only a small share of +keys move. Call [`SetPeers()`](instance.go) whenever your topology changes. + +**A miss runs your code, once.** Give each group a [`Getter`](group.go) that fetches from your +database or service. On a miss it runs, and the value is cached and returned to everyone who asked. +Concurrent requests for the same key wait on one in-flight load through +[`singleflight`](internal/singleflight/singleflight.go). A million simultaneous `Get("FOO")` calls +on one peer run your `Getter` once, and calls arriving from other peers join the same load, so +the cluster normally runs it once. The stampede that follows a cold cache or a hot key +never reaches your database. + +**A dead owner doesn't take you down.** If a peer can't get an answer from a key's owner (refused +connection, transport timeout, bad response), it loads the value itself, still deduplicated, and +the request succeeds. The fallback is skipped when the owner answers "not found" or "unavailable" +(HTTP 404 and 503), or when the caller's own context is cancelled or past its deadline. + +**Hot keys spread themselves.** Each group keeps a main cache for keys this peer loaded or was sent +by a `Set()`, and a hot cache for copies of keys it fetched from other peers. A key that every peer +asks for ends up in every peer's hot cache, so its owner isn't the bottleneck. The hot cache is 1/8 +of the group's memory limit and the main cache is 7/8. + +**Writes reach the whole cluster.** [`Set()`](group.go) goes to the owner and then to every other +peer. `Remove()` and `RemoveKeys()` clear the key on the owner and on every other peer. Delivery is +best effort, so put an expiry on values to cap how long a missed update can live. + +**Expiry needs no coordination.** A value can carry an expiry time, and a `Get()` that finds it +expired drops it. No peer talks to another about it, so keep clocks in sync across the cluster. + +**Swap the parts you want to.** +- Caches. The default is an LRU. The [Otter cache](contrib/otter.go) uses lock-free S3-FIFO eviction + and suits machines with many cores. See [Pluggable Internal Cache](#pluggable-internal-cache). +- Transports. HTTP is the default. Write your own against the [`transport.Transport`](transport/http_transport.go) interface. -- Hash function and replica count for the ring. +- The hash function and replica count for the ring. -**Observable.** Group and cache stats are exposed as OpenTelemetry metrics, and the HTTP transport -can emit OpenTelemetry traces for peer calls. +**See what it's doing.** Group and cache stats are exposed as OpenTelemetry metrics, and the HTTP +transport can emit OpenTelemetry traces for peer calls. -**Many instances per process.** There is no global state, so tests (and the [`cluster`](cluster) -package) can run a whole cluster inside one process. +**Run a whole cluster in one process.** There is no global state, so you can start several instances +in a single process. The [`cluster`](cluster) package does this for tests. -The sequence diagram under [The Cache Loading process](#the-cache-loading-process) shows these -steps for a single `Get()`. +The sequence diagram under [The Cache Loading process](#the-cache-loading-process) walks one `Get()` +through these steps. ## Usage From ac221b5fdfffac08f85dba9d69347773ff92f4ff Mon Sep 17 00:00:00 2001 From: "Derrick J. Wippler" Date: Fri, 23 Oct 2026 23:34:36 -0500 Subject: [PATCH 3/4] docs: say the owner lookup is a local hash, and fix the stampede claim --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index f60d52c..328dff8 100644 --- a/README.md +++ b/README.md @@ -28,7 +28,7 @@ There is nothing else to deploy. **Every key has one home.** Peers sit on a [consistent hash ring](transport/peer/picker.go), so any -peer can tell, with no lookup, which peer owns a key. When a peer joins or leaves, only a small share of +peer finds a key's owner with a local hash lookup, and no network call. When a peer joins or leaves, only a small share of keys move. Call [`SetPeers()`](instance.go) whenever your topology changes. **A miss runs your code, once.** Give each group a [`Getter`](group.go) that fetches from your @@ -36,8 +36,8 @@ database or service. On a miss it runs, and the value is cached and returned to Concurrent requests for the same key wait on one in-flight load through [`singleflight`](internal/singleflight/singleflight.go). A million simultaneous `Get("FOO")` calls on one peer run your `Getter` once, and calls arriving from other peers join the same load, so -the cluster normally runs it once. The stampede that follows a cold cache or a hot key -never reaches your database. +the cluster normally runs it once. After a cold start or a burst on one hot key, your +database sees one query per key, not one per caller. **A dead owner doesn't take you down.** If a peer can't get an answer from a key's owner (refused connection, transport timeout, bad response), it loads the value itself, still deduplicated, and From ed70adf0b0b5511612805bb474e2b53373852fb0 Mon Sep 17 00:00:00 2001 From: "Derrick J. Wippler" Date: Fri, 23 Oct 2026 23:44:22 -0500 Subject: [PATCH 4/4] docs: tighten Design prose --- README.md | 77 ++++++++++++++++++++++++------------------------------- 1 file changed, 33 insertions(+), 44 deletions(-) diff --git a/README.md b/README.md index 328dff8..7f8a249 100644 --- a/README.md +++ b/README.md @@ -19,55 +19,44 @@ like MEMCACHED and REDIS in many scenarios. - Load Protection: Prevents the thundering herd problem through the use of `singleflight` synchronization ## Design -Most caches are a separate fleet of servers you have to run, size, and keep alive. Groupcache is a -library. Import it, hand each process a list of its peers, and your app servers become the cache. -There is nothing else to deploy. +Groupcache is a Go library for caching across a cluster of processes. Every process that imports it +holds part of the cache and serves that part to the others, so there are no cache servers to run. +Each process is given the list of its peers through [`SetPeers()`](instance.go), and gets a new list +whenever peers join or leave.
Simplified Diagram
-**Every key has one home.** Peers sit on a [consistent hash ring](transport/peer/picker.go), so any -peer finds a key's owner with a local hash lookup, and no network call. When a peer joins or leaves, only a small share of -keys move. Call [`SetPeers()`](instance.go) whenever your topology changes. - -**A miss runs your code, once.** Give each group a [`Getter`](group.go) that fetches from your -database or service. On a miss it runs, and the value is cached and returned to everyone who asked. -Concurrent requests for the same key wait on one in-flight load through -[`singleflight`](internal/singleflight/singleflight.go). A million simultaneous `Get("FOO")` calls -on one peer run your `Getter` once, and calls arriving from other peers join the same load, so -the cluster normally runs it once. After a cold start or a burst on one hot key, your -database sees one query per key, not one per caller. - -**A dead owner doesn't take you down.** If a peer can't get an answer from a key's owner (refused -connection, transport timeout, bad response), it loads the value itself, still deduplicated, and -the request succeeds. The fallback is skipped when the owner answers "not found" or "unavailable" -(HTTP 404 and 503), or when the caller's own context is cancelled or past its deadline. - -**Hot keys spread themselves.** Each group keeps a main cache for keys this peer loaded or was sent -by a `Set()`, and a hot cache for copies of keys it fetched from other peers. A key that every peer -asks for ends up in every peer's hot cache, so its owner isn't the bottleneck. The hot cache is 1/8 -of the group's memory limit and the main cache is 7/8. - -**Writes reach the whole cluster.** [`Set()`](group.go) goes to the owner and then to every other -peer. `Remove()` and `RemoveKeys()` clear the key on the owner and on every other peer. Delivery is -best effort, so put an expiry on values to cap how long a missed update can live. - -**Expiry needs no coordination.** A value can carry an expiry time, and a `Get()` that finds it -expired drops it. No peer talks to another about it, so keep clocks in sync across the cluster. - -**Swap the parts you want to.** -- Caches. The default is an LRU. The [Otter cache](contrib/otter.go) uses lock-free S3-FIFO eviction - and suits machines with many cores. See [Pluggable Internal Cache](#pluggable-internal-cache). -- Transports. HTTP is the default. Write your own against the - [`transport.Transport`](transport/http_transport.go) interface. -- The hash function and replica count for the ring. - -**See what it's doing.** Group and cache stats are exposed as OpenTelemetry metrics, and the HTTP -transport can emit OpenTelemetry traces for peer calls. - -**Run a whole cluster in one process.** There is no global state, so you can start several instances -in a single process. The [`cluster`](cluster) package does this for tests. +**Each key has one owner at a time.** Peers sit on a [consistent hash ring](transport/peer/picker.go), +and a process finds a key's owner by hashing the key against its own copy of the ring. When a peer +joins or leaves, only the keys on its part of the ring change owner. + +**A miss goes to the owner.** `Get()` looks in the local caches first. On a miss, a process that +doesn't own the key asks the owner, and the owner runs your [`Getter`](group.go), the function that +fetches the value from your database or service, then caches the result. A process keeps a copy of +what it fetched from other peers in its hot cache, so popular keys are served locally. + +**Concurrent misses share one load.** Callers asking for the same missing key wait on a single +in-flight load through [`singleflight`](internal/singleflight/singleflight.go), and requests from +other peers join it. A million simultaneous `Get("FOO")` calls run your `Getter` once. If a peer +can't reach the owner, it runs the `Getter` itself instead of failing the request. + +**Writes reach the whole cluster.** `Set()`, `Remove()` and `RemoveKeys()` go to the owner and then to +every other peer. Delivery is best effort, so put an expiry on your values to cap how long a missed +update can live. Expiry is checked when a value is read, and needs no coordination between peers +beyond keeping their clocks in sync. + +**The parts are swappable.** The default cache is an LRU, and the [Otter cache](contrib/otter.go) uses +lock-free S3-FIFO eviction for machines with many cores (see +[Pluggable Internal Cache](#pluggable-internal-cache)). HTTP is the default transport, and you can +write your own against the [`transport.Transport`](transport/http_transport.go) interface. + +**It reports on itself.** Stats are exposed as OpenTelemetry metrics, and the HTTP transport can emit +OpenTelemetry traces for peer calls. + +**There is no global state.** You can run several instances in one process, and the +[`cluster`](cluster) package uses this to start a whole cluster inside a test. The sequence diagram under [The Cache Loading process](#the-cache-loading-process) walks one `Get()` through these steps.