Skip to content

Stop the agent log file from growing without a limit - #5079

Draft
Vandit1604 wants to merge 1 commit into
superfly:masterfrom
Vandit1604:agent-log-rotation
Draft

Stop the agent log file from growing without a limit#5079
Vandit1604 wants to merge 1 commit into
superfly:masterfrom
Vandit1604:agent-log-rotation

Conversation

@Vandit1604

Copy link
Copy Markdown

Change Summary

What and Why:

The agent writes one log file for its whole lifetime, and nothing limits how large that file gets. A user in #5046 reached hundreds of GB.

The directory does get cleaned. setupLogDirectory (agent/start.go:190) deletes any file nothing has touched for 24 hours. But it only runs when a daemon starts, through createLogFile at agent/start.go:172, and a running agent keeps its own log's mtime current. So the one file that grows is the one file the cleanup can never reap.

I confirmed this before changing anything. A 512MB log with a current mtime, placed next to a week old log, survives the prune while the old file goes.

The writer has no limit either. setupLogger opened the file with O_APPEND and nothing else (internal/command/agent/run.go:80).

How:

openRotatingFile replaces the plain open. It caps the file at 128MB, which is inside the 100MB to 500MB range the issue suggests. When the file is full, it renames it to <name>.prev.log and starts a new one. One earlier generation stays on disk, so you still have history when you go looking for it. The pair costs at most 256MB until the next daemon start prunes them.

Three details worth pointing out:

  • It counts the bytes it writes instead of measuring the file. The reporter deleted the log directory and watched the agent keep filling the disk through the open descriptor. Counting bytes means a deleted log gets replaced at the next threshold instead of growing forever.
  • It closes the file before renaming it, because Windows will not rename an open file.
  • The rotated name keeps the .log ending. fly doctor diag collects agent logs with a *.log glob (internal/command/doctor/diag/diag.go:201), so the previous generation still ends up in a diagnostic bundle.

I left the start up prune alone. It already bounds every file except the one in use, so replacing it with a keep the last ten rule would not fix anything. Happy to add that too if you would rather have both.

Testing:

internal/command/agent/logfile_test.go covers four things: the file stays under the cap, an existing log resumes from its size on disk rather than getting a fresh budget, a log deleted underneath the agent gets replaced, and the rotated name keeps its .log ending.

I checked the tests fail without the fix. With the size check disabled, three of the four fail and the log reaches 402MB against the 128MB cap.

go build ./..., go vet, gofmt and golangci-lint v2.11.3 are all clean.

Related to: #5046


Documentation

  • Fresh Produce
  • In superfly/docs, or asked for help from docs team
  • n/a

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant