Stop the agent log file from growing without a limit - #5079
Draft
Vandit1604 wants to merge 1 commit into
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change Summary
What and Why:
The agent writes one log file for its whole lifetime, and nothing limits how large that file gets. A user in #5046 reached hundreds of GB.
The directory does get cleaned.
setupLogDirectory(agent/start.go:190) deletes any file nothing has touched for 24 hours. But it only runs when a daemon starts, throughcreateLogFileat agent/start.go:172, and a running agent keeps its own log's mtime current. So the one file that grows is the one file the cleanup can never reap.I confirmed this before changing anything. A 512MB log with a current mtime, placed next to a week old log, survives the prune while the old file goes.
The writer has no limit either.
setupLoggeropened the file withO_APPENDand nothing else (internal/command/agent/run.go:80).How:
openRotatingFilereplaces the plain open. It caps the file at 128MB, which is inside the 100MB to 500MB range the issue suggests. When the file is full, it renames it to<name>.prev.logand starts a new one. One earlier generation stays on disk, so you still have history when you go looking for it. The pair costs at most 256MB until the next daemon start prunes them.Three details worth pointing out:
.logending.fly doctor diagcollects agent logs with a*.logglob (internal/command/doctor/diag/diag.go:201), so the previous generation still ends up in a diagnostic bundle.I left the start up prune alone. It already bounds every file except the one in use, so replacing it with a keep the last ten rule would not fix anything. Happy to add that too if you would rather have both.
Testing:
internal/command/agent/logfile_test.gocovers four things: the file stays under the cap, an existing log resumes from its size on disk rather than getting a fresh budget, a log deleted underneath the agent gets replaced, and the rotated name keeps its.logending.I checked the tests fail without the fix. With the size check disabled, three of the four fail and the log reaches 402MB against the 128MB cap.
go build ./...,go vet,gofmtand golangci-lint v2.11.3 are all clean.Related to: #5046
Documentation