PMM-14807 Enable the global connection pool by default

[ademidoff]

PMM-14807

Addresses #1346.

Problem

The exporter created a fresh mongo.Client for every scrape and disconnected it immediately afterwards. Each scrape therefore cost the monitored mongod a TCP connect, a TLS handshake where configured, a SCRAM authentication and topology discovery — work paid for by the database, not by the exporter.

--mongodb.global-conn-pool already avoided this, but it defaulted to false, so nobody got the benefit by default.

Changes

main.go — --mongodb.global-conn-pool now defaults to true. --no-mongodb.global-conn-pool restores the previous per-scrape behaviour. (tagalign realigned the surrounding struct tag block as a side effect of the added default:"true".)

exporter/exporter.go — stop pinning a disconnected client in the cache. Ping failures were returned to the caller with the client left in place. That is correct for a transient error — the driver reconnects the pool on its own, and tearing it down would only add churn while MongoDB is already struggling — but a client that returns ErrClientDisconnected never recovers, and every later scrape would keep failing with it. That one is dropped so the next scrape builds a new client.

I deliberately did not reconnect inside the failing scrape. An earlier version did, and it cost the mongodb_up 0 datapoint during an outage: the in-scrape reconnect attempt ran past the Prometheus scrape timeout and returned no metrics at all, which makes "database down" indistinguishable from "exporter down".

Measurements

Single-node replica set, SCRAM-SHA-256 auth, 200 scrapes with diagnosticdata, dbstats, collstats, indexstats, top, currentop, replicasetstatus and replicasetconfig enabled. A third arm with no exporter running measured background mongod CPU, which is subtracted below. Three independent runs.

	pool off	pool on
connections created	604 – 606 (3.0/scrape)	4 total
SCRAM-SHA-256 handshakes	406 – 414 (2.0/scrape)	6 – 10
mongod CPU per scrape	6.4 – 11.2 ms	3.3 – 5.1 ms
distinct metric names exposed	5280	5280 (identical)
exporter RSS after 120 requests	76.6 MB	74.7 MB

Connections were attributed to the exporter via mongod's client metadata log lines (id 51800) filtered by appName, and corroborated by the serverStatus().connections.totalCreated delta.

The connection and handshake counts are stable to within a fraction of a percent across runs. The CPU figures are noisier — the reduction came out at 48%, 58% and 45% — because the test host was running other containers, so treat "roughly halves" as the claim rather than any single figure.

At PMM's 15s scrape interval this is roughly 17,400 connections and 11,700 SCRAM handshakes per target per day replaced by a handful that persist. The CPU figure is a floor: this mongod was on localhost with no TLS, and production adds network RTT plus a TLS handshake per connection, both of which pooling also eliminates.

Scrape budgets while MongoDB is unreachable

The review surfaced a second class of problem that none of the above covers: a scrape can be made to overrun its budget, so Prometheus receives nothing at all instead of mongodb_up 0 — which makes "database down" indistinguishable from "exporter down". Two causes, both fixed here, measured against a black-holed address (192.0.2.1, so TCP connect hangs rather than refuses) with a 1s scrape budget and the default 5s server-selection timeout:

	before	after
creating the pooled client inside the scrape	5.0s	1.0s
scrape arriving while another connect is in flight	5.0s	1.1s

These matter specifically because of this PR: with the pool off there is no shared client and no lock, so scrapes of one target could not affect each other. Enabling the pool by default introduces that coupling. getClient now holds the lock for the client pointer only, never across a command:

• The health-check Ping runs outside the lock. Under an exclusive lock, concurrent scrapes of one target — PMM scrapes each exporter at three resolutions with different collect[] sets — queued behind each other, so one slow scrape could push the next past its budget and report mongodb_up 0 for a perfectly healthy client.
• The connect that fills the cache gets its own budget, from ConnectTimeoutMS, rather than the caller's. Bounding it by the scrape context fixed the overrun but created the opposite failure: a scrape budget shorter than one connect cancelled every attempt, so the pool stayed permanently empty and every scrape kept paying for a connect that could never finish. Concurrent attempts now collapse onto one connect, and each caller gives up on its own deadline while it carries on in the background.
• ConnectTimeoutMS == 0 is still handled: connect would hand the driver a server-selection timeout of 0, which it reads as no timeout at all (topology.go:548 only arms a timer when the value is positive), so the budget falls back to a 30s default.

Both behaviours are covered by tests that fail against the previous shape.

Startup client leak without the pool

New()'s background connect discarded its client without disconnecting it, leaking a topology, its monitoring goroutines and a connection per exporter — times N under --split-cluster. This only affects --no-mongodb.global-conn-pool users and predates the PR, but the branch it lives in is one this PR rewrites. On a live mongod, with no scrape issued, connections.current after startup: +2 before, +0 after.

Behaviour verified

• Outage recovery — stopped mongod mid-run and restarted it. Both before and after this change the exporter reports mongodb_up 0 while it is down and recovers to 1 on the first scrape after restart.
• Multi-target — /metrics, /scrape?target= and /scrapeall all serve both targets with mongodb_up 1 and no errors in the exporter log.
• Memory — no penalty from holding persistent clients; RSS is marginally lower, since the unpooled path allocates a fresh pool per scrape for the GC to reclaim.
• Tests — go test ./... produces an identical set of failures on this branch and on main (12 tests, all requiring the full Docker cluster, which was not running locally). golangci-lint run --new-from-rev=main is clean.

All of the above was re-measured after the scrape-context fix, and the churn and recovery results are unchanged: the changed line runs at most once per process, and only when the client cache is empty.

New tests: main_test.go covers the flag default and its negation; exporter/exporter_test.go covers both halves of the cache contract. Both new exporter tests fail against the pre-change code and pass after it.

Notes for reviewers

• With the pool on, a few connections now stay open per target between scrapes rather than being torn down: measured at +1 after 8 concurrent scrapes, +3 with 120 collections in --discovering-mode. The driver never reaps idle connections (maxIdleTimeMS defaults to 0), so this is peak concurrency becoming steady state. It is bounded and small at these scales, but setting SetMaxConnIdleTime would make it explicit — happy to add it here or leave it as a follow-up.
• VERSION is bumped v0.53.0 → v0.54.0 in a separate commit. Minor rather than patch, since this flips a user-visible default. It was previously equal to the latest tag, so the next release would have collided.
• Go is bumped 1.26.2 → 1.26.7 in a separate commit, for the vulnerability fixes in the intervening patch releases. Only go.mod and tools/go.mod needed it — all four workflows resolve the toolchain through go-version-file: go.mod, and nothing in Dockerfile or .goreleaser.yml pins a Go version. go mod tidy on both modules produced no dependency churn.
• --collector.pbm remains unpooled: exporter/pbm_collector.go calls sdk.NewClient per Collect. That is PMM-13998 and is left alone here, but it is now the dominant remaining churn source when PBM metrics are enabled.
• The PMM-side leg of PMM-14807 — exposing the flag through pmm-admin add mongodb — is handled separately. With this default flip, PMM users get the benefit without needing that flag at all; the PMM work is only needed to let users turn pooling off.
• CHANGELOG is untouched; this repo only updates it at release time.

[Copilot]

Pull request overview

Enables MongoDB connection pooling by default while allowing operators to restore per-scrape connections.

Changes:

• Defaults --mongodb.global-conn-pool to enabled.
• Replaces permanently disconnected cached clients on the next scrape.
• Adds flag and connection-cache tests and updates documentation/versioning.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
VERSION	Bumps version to v0.54.0.
REFERENCE.md	Documents the new default and opt-out flag.
main.go	Enables global connection pooling by default.
main_test.go	Tests default, explicit, and negated flag behavior.
exporter/exporter.go	Handles disconnected cached clients.
exporter/exporter_test.go	Tests disconnected and transient-error cache behavior.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[JiriCtvrtka]

I compared my solution with yours and Claude says your one is in better shape. So lets proceed with yours.