fix: turn hyperdx usage stats off by default - #188
Merged
Merged
Conversation
Upstream HyperDX's usage-stats task runs on every API start and every 4h. It reads each connection with Connection.find(), which leaves out the password (the model marks it select: false), so it logs in to ClickHouse as the team's reader with no password and fails with a 516 twice per start. It also reports user counts and host details to in-otel.hyperdx.io, a third party. The Kubernetes hyperdx chart already sets USAGE_STATS_ENABLED=false unless a deployer opts in. Compose left it at upstream's default of on. It now matches the chart, with DFE_HYPERDX_USAGE_STATS_ENABLED=true as the opt-in.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every HyperDX API start on a Compose stack with an existing team logs two ClickHouse auth failures as dfe_query_reader. It is not a stale or rotated password. The stored connection logs in fine (HTTP 200).
The cause is upstream's usage-stats task (packages/api/src/tasks/usageStats.ts). It reads connections with Connection.find(), the model marks password select: false, so it queries system.parts with no password and gets a 516 once per (connection, database) pair -- two for the admin team (dfe and system). Upstream main still has the same code. The task also sends user counts and host details to in-otel.hyperdx.io every 4h.
The k8s hyperdx chart already defaults USAGE_STATS_ENABLED to false. Compose never set it, so it ran upstream's default of on. This sets it from DFE_HYPERDX_USAGE_STATS_ENABLED, default false.
Checked on a local rc.14 stack (hyperdx v0.2.7):
One more edit belongs with this and could not go in: env.example/hyperdx.env still says
# USAGE_STATS_ENABLED=false. The compose environment entry now beats that file, so the line should point at DFE_HYPERDX_USAGE_STATS_ENABLED in .env instead. Our tooling blocks agents from staging that path as a credential file, so it needs a human commit.