compose gives the fetcher no writable cursor directory, so every restart re-reads the full lookback

[catinspace-au]

Seen on the rc.14 docker single stack, fetcher v1.4.24, at every fetcher start:

WARN dfe_fetcher: cursor.directory not set -- falling back to config file directory. Set cursor.directory to a PVC-backed path for pod restart persistence. fallback_dir=/etc/dfe/apps/dfe-fetcher
WARN dfe_fetcher::cursor::file: Cursor directory is not writable, operating in read-only mode directory="/etc/dfe/apps/dfe-fetcher"

The compose projection mounts the app-config volume read-only into the fetcher (as it should) and gives it no writable cursor directory, so the cursor store runs read-only and nothing survives a restart: every restart re-reads each source's full lookback (cursor.default_window_hours, 1 hour by default), which on CloudTrail is a 29 page burst that trips the LookupEvents limit (dfe-fetcher #120).

The k8s chart gives the fetcher a PVC-backed path for this. Compose needs the same: a named volume (the dlq-spool volume is already rw at /var/spool/dfe, or a cursor volume of its own) and cursor.directory pointing at it in config/fetcher/*.yaml.

Done when a fetcher restart on the docker single resumes from its cursor rather than re-reading the window, and the two warnings are gone from its start-up log.

[catinspace-au]

Two things from the rc.14 tree that make this smaller than it reads, and shift where the fix belongs.

The writable volume is already there. docker-compose.yml:476-478 gives the fetcher dlq-spool:/var/spool/dfe read-write, alongside the read-only config mount. So no new volume is needed -- this is a path, not plumbing.

But the path the warning names is not the one compose mounts. Compose mounts the config at /etc/dfe/fetcher.yaml:ro (:477), and the warning reports its fallback as /etc/dfe/apps/dfe-fetcher. That is the engine-rendered app-config directory, not the compose file. So the running fetcher is reading the config the ENGINE writes, and cursor.directory has to come from there rather than from config/fetcher/*.yaml in this repo.

If that is right, the fix is ours in dfe-engine or in dfe-infra's apps.yaml rather than a dfe-docker change, and editing the compose config file would leave the warning exactly where it is. Worth confirming which file the container actually loads before anyone changes either.

The consequence is worth restating because it is larger than a warning. With the cursor store read-only nothing survives a restart, so every restart re-reads each source's full lookback window. On CloudTrail that is a 29-page burst that trips the LookupEvents limit (dfe-fetcher#120). So this is a duplicate-ingest and rate-limit defect, not a tidiness one.

Verified against the rc.14 worktree: the volume mount, the read-only config mount and the two paths differing. NOT verified: which config file the fetcher process actually loads at runtime, which is the thing that decides ownership.

[catinspace-au]

Correcting myself: the fix belongs here after all, and my inference in the previous comment was wrong.

I reasoned from the warning's fallback_dir=/etc/dfe/apps/dfe-fetcher that the fetcher must be loading the engine-rendered config. Checked it properly instead:

• the image's default command is --config /etc/dfe/fetcher.yaml on both architectures (ENTRYPOINT ["dfe-fetcher"])
• docker-compose.yml:477 mounts ./config/${DFE_FETCHER_CONFIG:-fetcher/aws-kafka.yaml} to exactly that path
• nothing in docker-compose.yml or docker-compose.live.yml overrides the command, and /etc/dfe/apps appears nowhere in this repo's compose at all

So on the current tree the fetcher reads config/fetcher/*.yaml from this repo, and setting cursor.directory there is the fix -- which is what this issue said in the first place.

What I cannot reconcile, and it is worth someone checking before the fix lands. If the loaded config is /etc/dfe/fetcher.yaml, its directory is /etc/dfe, not /etc/dfe/apps/dfe-fetcher. So the fallback_dir in the captured warning does not correspond to anything this compose mounts. Either that observation came from a stack whose projection has since changed, or the fetcher derives the fallback from something other than the config file's own directory despite what the message says.

Worth resolving because it decides whether the fix is one line in config/fetcher/*.yaml or something that also has to reach the path the binary actually picks. Pointing cursor.directory at the already-mounted dlq-spool volume under /var/spool/dfe sidesteps the question entirely, since it names an absolute path rather than relying on any fallback.

Apologies for the churn -- I moved ownership on an inference I had not tested.