Skip to content

feat!: add resolver-driven Memcached auto discovery - #43

Merged
yeqown merged 18 commits into
mainfrom
feat/40-auto-discovery
Oct 19, 2026
Merged

yeqown merged 18 commits into
mainfrom
feat/40-auto-discovery

Conversation

@yeqown

@yeqown yeqown commented Oct 6, 2026 •

Copy link
Copy Markdown
Owner

Merged and released: squash merged into main as 9984c7d; v1.5.0 is published with explicit breaking changes and migration instructions. The existing module import path is retained. All five CLI release assets built successfully, and the downloaded macOS ARM64 asset reports v1.5.0.

Main-library dependency update: PR #44 merged as aaf581f, updating CLI, GUI, example, and benchmark requirements to v1.5.0 with tidied module files. Its CI passed.

Static server lists cannot follow Memcached cluster membership changes. This PR implements opt-in discovery through config get cluster, scheduled topology refreshes, and per-node connection pools that are reused or drained as membership changes.

Related to #40. Review updated on 2026-10-19 for 800f365 against main / 9e03067.

Follow-up fixes are committed and pushed in 91e4f9a. Standalone CLI/example builds, the performance changes below, and the concurrent example race fix are included. Real integration tests and nine examples passed with the race detector. The deprecated SASL example was compiled only; live SASL authentication was not exercised because no SASL-enabled test server is available.

Implementation and discovery scope

  • Extract the public resolver and picker packages. Add NewWithContext, WithResolveTimeout, resolver-controlled refresh scheduling, and discovery/topology telemetry.
  • Resolve once during initialization, then refresh serially in the background. Publish routing inputs, nodes, and generation together; reuse pools for unchanged node identities, mark removed nodes draining before publishing membership, and close their pools outside the topology lock. Connections already borrowed may finish their requests.
  • Reserve connection capacity before dialing, wake pool waiters on state changes, and close idle sockets during pool shutdown. Data connections are opened on demand.
  • Migrate CLI routing, move the GUI into cmd/gui, update migration/lifecycle documentation, and make default tests self-contained.

Memcached automatic discovery is implemented, but must be enabled explicitly. The default remains resolver.NewStatic() with no refresh schedule.

client, err := memcached.New("configuration-endpoint:11211",
    memcached.WithResolver(resolver.NewAutoDiscovery(time.Minute)),
    memcached.WithResolveTimeout(5*time.Second),
    memcached.WithPicker(picker.NewRendezvousHashPicker(1)),
)

The resolver implements the ASCII config get cluster extension used by AWS ElastiCache and Google Memorystore. It reuses its discovery connection, reconnects after failed exchanges, validates configurations, rejects stale generations, sorts discovered nodes, and retains cached data on exchange timeout. Background errors retain the client's last usable topology and retry according to the returned schedule. Each resolver belongs to one client, which owns closing it.

Discovery accepts TCP/tcp4/tcp6 endpoints and uses TCP for discovered data nodes, preferring their advertised IPs. This does not provide DNS/SRV or Kubernetes discovery, legacy AWS discovery commands, TLS/authentication configuration for the discovery connection, data migration, or automatic request replay. Ordinary upstream Memcached does not expose this cluster discovery extension. AWS/GCP support was checked through protocol fixtures and topology integration tests; live cloud interoperability has not been exercised.

Breaking API and behavior changes

Previous API Migration / impact
Root Addr, NewAddr, Resolver, Picker, Builder Import resolver.Addr, resolver.Resolver, and picker.Picker; construct pickers directly. Addr contains only Network and Address; priority and metadata helpers are removed.
WithPickBuilder and root *PickBuilder constructors Use WithPicker(picker.NewCRC32HashPicker()), NewMurmur3HashPicker(seed), or NewRendezvousHashPicker(seed) from picker.
Resolve(string) ([]*Addr, error) Implement Resolve(context.Context, string) (resolver.ResolveResult, *time.Time, error) and Close() error. The result exposes Generation uint64; nil scheduling stops refreshes.
Public hash package / Murmur3 digest Removed; callers importing that package need an alternative.

New and the client command signatures remain available. Custom pickers must handle concurrent calls and changing address snapshots. Static resolution now checks syntax without performing DNS lookups and rejects duplicate identities and invalid address forms.

Rendezvous scoring now uses node identity rather than priority and changes existing key placement even with the same addresses and seed. Plan for cache misses during migration. CRC32/Murmur3 retain modulo routing, so membership changes can remap many keys. Retrieval commands now route by their first key, matching storage; multi-key Gets / GetAndTouches still require keys on the same node and do not scatter/gather.

Release target: v1.5.0, retaining the existing github.com/yeqown/memcached import path. The release will explicitly list these breaking changes and migration steps. External consumers have not been enumerated; compatibility conclusions cover the modules and examples checked here.

CLI/example build fixes and performance

The standalone CLI build failure at 2d985b2 is fixed in 91e4f9a: its nested module now requires published root pseudoversion v1.4.1-0.20261019055421-2d985b2ae2c6, which supplies the new picker package, and its module files have been tidied. CI adds GOWORK=off go build -mod=readonly ./... from cmd/memcached-cli, so workspace resolution cannot hide a stale dependency. No relative replace is needed. The standalone CLI currently uses the already-published 2d985b2 implementation; a later root release/dependency update is required to include the new performance optimizations in standalone CLI distributions.

The example module dependency files were also stale outside workspace mode and are now tidied. CI builds every standalone example with GOWORK=off and -mod=readonly. The concurrent example now keeps errors local to each goroutine, fixing a reproduced race caused by assigning to the outer err. Example execution and expected CAS/context-deadline output are documented in docs/testing.md.

The performance changes are internal and introduce no additional public API changes:

  • Allocate a pool notification channel only when borrowers actually wait for capacity. Registration and notification still use the same mutex, preserving capacity limits and wakeups on return, failure, and close.
  • Replace the per-borrow closure and separate synchronization object with one fresh caller-owned release handle. Keep sync.Once to preserve concurrent cleanup waiting, idempotence, and panic semantics. The handle still allocates once; it is never copied or reused.
  • Build default rendezvous hash input in a stack buffer for common keys, with a heap fallback for larger inputs. Keep identical length framing and key placement. Custom hashes continue to receive independent input buffers, now with one allocation per node instead of repeated buffer growth.
  • Add allocation regression tests, routing golden cases including long inputs, retained custom-hash-input checks, a multiple-waiter/cancellation test across repeated notifications, and reusable pool/rendezvous benchmarks.

The following medians are from five serially executed benchmark runs on Apple M2 / darwin/arm64, Go 1.26.0, GOMAXPROCS=8, with telemetry disabled. Temporary snapshots of main / 9e03067, pre-fix head / 2d985b2, and the optimized root implementation in 91e4f9a use identical HEAD dependency versions to isolate code changes. Runs used -benchmem -benchtime=500ms -count=5 -cpu=8. Get used a dedicated Memcached 1.6.37 with two server threads, a small seeded value, and warm connections; pool reuse used in-memory connections. Parallel cases used eight workers.

Benchmark main → pre-fix 2d985b2 → optimized 91e4f9a time/op Bytes/op Allocations/op
Pool reuse, serial 40.6 → 67.7 → 48.2 ns 0 → 112 → 0 0 → 1 → 0
Pool reuse, parallel 157.2 → 292.0 → 222.5 ns 0 → 112 → 0 0 → 1 → 0
Get, serial 26.22 → 27.24 → 29.09 µs 264 → 456 → 328 9 → 13 → 11
Get, parallel 10.21 → 9.24 → 9.16 µs 265 → 457 → 329 9 → 13 → 11
Rendezvous, 10 nodes 658.1 → 658.3 → 251.8 ns 720 → 800 → 0 20 → 30 → 0
Rendezvous, 50 nodes 3.029 → 3.319 → 1.211 µs 3600 → 4000 → 0 100 → 150 → 0

Compared with 2d985b2, pool reuse is approximately 24–29% faster in these runs, and default rendezvous selection is approximately 62–64% faster at 10/50 nodes. Get consistently saves 128 B and two allocations per operation. The first serial Get series was 6.8% slower; a follow-up of five alternating before/after pairs (-benchtime=750ms, alternating execution order) did not reproduce that regression: serial medians were 26.63 → 26.21 µs and parallel medians 9.64 → 9.34 µs. End-to-end timing remains noisy, so the supported conclusion is reduced allocation pressure rather than a production latency guarantee.

Some overhead remains relative to main: optimized pool reuse is still about 19% slower serially and 42% slower in this parallel microbenchmark, and Get still makes 11 rather than nine allocations. Rendezvous remains O(node count). Default rendezvous inputs larger than the stack buffer and custom hashes still allocate; custom hashes retain their ownership behavior.

Discovery I/O stays off the request path. Each auto-discovery client adds a refresh loop/timer and a discovery connection; unchanged identities retain their existing pools. Pool limits apply per node, so aggregate connection capacity grows with discovered node count. Production load and large-cluster behavior remain unmeasured.

The CLI release version injection is fixed in 800f365: the version is now a linker-overridable variable. A build-and-run check reproduced the previous failure, passed after the fix, and is now part of standalone CLI CI. Release builds can report the actual release tag.

Validation

Fresh local verification of 91e4f9a:

  • PASS: root module GOWORK=off go test -race -count=1 ./....
  • PASS: go vet ./..., golangci-lint run --timeout=5m ./... (v2.13.2, zero issues), and git diff --check.
  • PASS: standalone CLI GOWORK=off GOPROXY=off go build -mod=readonly ... and go test -mod=readonly -race -count=1 ./...; workspace CLI checks also pass.
  • PASS: GUI go test -race -count=1 ./service/... and benchmark-module compilation against the local library.
  • PASS: TestMemcachedIntegration with -race -count=1 against an isolated local Memcached 1.6.37, covering classic, meta, compression, concurrent reads, and concurrent meta operations.
  • PASS: all eight top-level example programs and otel ran with -race against an isolated Memcached 1.6.37 TCP/UDP daemon, both in workspace mode and with GOWORK=off / -mod=readonly. Only endpoint literals were redirected with temporary Go overlays; the examples otherwise ran unchanged. The concurrent example completed 10,000 Set/Get pairs without race reports in each mode.
  • PASS: every example standalone readonly build, including SASL and otel. SASL validation is compile-only, with no live authentication or daemon compatibility claim.
  • PASS: all three benchmark snapshots and five alternating Get pairs per mode. Allocation, routing, waiter, close, removal, and concurrent/panicking release regression tests pass under the race detector. Independent diff review found no blockers.

GitHub CI for 91e4f9a passed: Lint, Test (including standalone CLI and every example build), and Memcached compatibility all succeeded (run). The real example executions described above were verified locally in both workspace modes; SASL remains compile-only.

Release preparation re-ran the real Memcached 1.6.37 integration suite and all nine examples with the race detector; all passed. The CLI release-version check, standalone build, race invocation, vet, and root lint also passed after 800f365. SASL remains compile-only. Final CI for 800f365 passed (run); squash merge 9984c7d also passed main CI (run).

yeqown added 16 commits October 6, 2026 17:00
Move discovery, membership updates and source generations into topology, and delegate borrowing and draining to nodes. Reuse discovery connections and cached configurations on timeout, require resolvers to own cleanup, and route rendezvous hashing by node identity.

Update lifecycle, routing and telemetry tests for the new contracts.

BREAKING CHANGE: custom resolvers must implement Close, and rendezvous hashing no longer includes Priority in its scores.
Share protocol and metrics fixtures, separate daemon compatibility checks, and document test commands. Simplify config response parsing, preserve read errors, and handle cancellation and concurrent shutdown.
Use comparable Addr values containing only Network and Address for node identity. Remove priority and metadata, and move sorting into resolvers so topology preserves their order.

Extract the connection factory from node state and update tests and documentation.

BREAKING CHANGE: AddrKey, Priority, metadata helpers, and CanonicalAddress are removed. NewAddr now accepts only network and address. Topology preserves resolver order, which affects modulo routing.
Preserve resolver address identity, reject duplicate discovery nodes, and cancel refresh loops during shutdown. Drain removed nodes asynchronously and record routing and borrowing failures in request telemetry.
@yeqown yeqown changed the title feat: add resolver-driven Memcached auto discovery feat!: add resolver-driven Memcached auto discovery Oct 19, 2026
Update the CLI dependency to the published root commit containing picker and
refresh example dependency metadata for GOWORK=off. Cover standalone CLI and
example builds in CI.

Allocate pool wakeup channels only for capacity waiters, replace escaping
release closures with single-use handles, and keep built-in rendezvous inputs
on the stack. Preserve routing, custom hash ownership, and release semantics.

Fix the concurrent example's shared err data race and document example runs.

Validation:
- Full root tests and real Memcached 1.6.37 integration tests with -race.
- Nine examples run with -race against isolated TCP/UDP Memcached, both with
  workspace resolution and GOWORK=off; only endpoint literals were redirected
  through temporary Go overlays.
- CLI standalone build/checks, GUI service race tests, go vet, golangci-lint,
  standalone example builds, and git diff --check.
- SASL example compiled only; live SASL authentication was not tested
  because no SASL-enabled test server is available.
@yeqown
yeqown marked this pull request as ready for review October 19, 2026 08:59
@yeqown
yeqown merged commit 9984c7d into main Oct 19, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant