flyctl's ~/.fly/agent-logs grows without bounds (no rotation/garbage collection)

[adam-symbolica]

Summary

My system specifics for bug report
OS: macOS 26.6 (25G72), darwin/arm64
flyctl: v0.4.78 darwin/arm64 (commit e5cda84, 2026-08-03)

Issue

fly agent run writes stdout/stderr to ~/.fly/agent-logs/<id>.log for the agent's lifetime with no rotation, size cap, or GC.

• ~/.fly/logs already retains ~10 CLI debug runs (logging: log equiv. of LOG_LEVEL=debug to files in ~/.fly/logs #2536).
• agent-logs has no equivalent, so a long-lived agent under heavy WireGuard/SSH load can fill the local disk.

Repro:

1. concurrent fly ssh across many isolated-network apps grew a single agent log to hundreds of GB.
2. After rm of agent-logs/*, the still-running agent kept the unlinked FD open and continued growing (~37 GB until fly agent restart).

Proposal

1. On agent start: create the new log; prune old ~/.fly/agent-logs/*.log (keep last N=10, same idea as ~/.fly/logs).
2. While running: bound the active file (rotate/re-open at e.g. 100–500 MB and prune). File-count retention alone is not enough, one process == one log for hours/days.
3. Docs: note in fly agent help / doctor that these files are diagnostic, bounded, and safe to delete (rm + fly agent restart).

[Vandit1604]

I read through the agent code to see what was going on here, and I think the diagnosis needs a small correction before the fix makes sense.

There is a collector, it just can't ever reach the file you care about. setupLogDirectory (agent/start.go:190) deletes anything nothing has touched in 24 hours, and it only runs when a daemon starts, through createLogFile at agent/start.go:172. A running agent keeps writing to its own log, so that log's mtime is always fresh and it never qualifies. The one file that grows is the one file the cleanup skips.

I checked that rather than assuming it. I put a 512MB log with a current mtime next to a week old one and ran the prune. It reaped the old file and left the 512MB file exactly where it was.

Nothing bounds the file while it is open either. setupLogger does a plain O_APPEND open with no size limit (internal/command/agent/run.go:80).

That also explains the 37GB you saw after deleting the directory. Nothing counted how much had been written, so the unlinked descriptor had no threshold to hit. That one took seconds to reproduce: 134MB written to a descriptor with no file on disk to show for it.

I am raising a PR that caps the live file and keeps one previous generation, so you still have history when you go looking. I left the start up prune alone, since it already handles every file except the one in use.