POP3 bounded sync needs durable UIDL progress instead of fixed tail-window polling

[seonghobae]

Current authority — 2026-09-18

This issue remains open, but its original source-state description is stale. Canonical email/POP3 owner #1195 has advanced from the fixed-tail predecessor to exact 630d5461d1eb8b59b392e4170dec66ce1edf034c on #1623. The durable UIDL progress repair is now present in source; remaining blockers are integration/evidence, migration reconciliation, and runtime acceptance rather than an absent implementation.

Verified source repair in #1195

The original problem was real: selecting the highest MAX_POP3_FETCH_MESSAGES message numbers on every poll could still starve older unseen messages, and a persistent RETR -ERR tail could monopolize every later poll.

#1195 now keeps provider UIDL as collection identity, separate from Message-ID/source fingerprint semantics, and persists owner-scoped collection state with:

• disposition observed | retryable;
• nullable retry_after;
• bounded selection that prioritizes never-attempted unseen UIDLs before due retries;
• successful persistence as the condition for moving a UIDL to observed;
• standard per-message -ERR/parse failures remaining retryable rather than silently completed;
• provider network I/O outside long-lived DB transaction/lock scope;
• no DELE during synchronization.

Source-order RED 5d40f83ff48e7dd9c3bf6deb93995130c5a1470a captured cross-poll starvation and retry-selection behavior. Repair 7406cb63923003df2c3a5d21a0104203f78b90bc extended the existing POP3 observed-UIDL state instead of creating a separate collection owner. Current #1195 exact head is 630d5461... after doctoring/authority follow-up.

Migration ownership boundary

The POP3 persistence change is still branch-local, not merge-authoritative. #1195 currently carries 0018_email_date_provenance → 0019_pop3_observed_uidl, while canonical workspace/database owner #1503 exact 9151c75568c582c8147cfee6757cd00a9b4d60b7 owns 0018_workspace_registry → 0019_email_read_state_repair → 0020_workspace_organization_binding → 0021_workspace_personal_owner_binding; both descend from historical 0017.

These are parallel Alembic heads and must not independently merge. Required order is #1503 protected integration first, then ordinary/non-force adoption by #1195 and rechain/renumber of the email/POP3 revisions behind the then-current protected head. Acceptance requires exactly one Alembic head plus fresh and historical PostgreSQL upgrade evidence.

Evidence boundary

Current exact #1195 630d5461... has zero repository-local pull-request workflow receipts. The latest formal CodeRabbit APPROVED review was submitted at 2026-09-17 05:43 UTC and predates the later durable retry/progress generation, so it is historical rather than qualifying current-head evidence. Do not infer PostgreSQL, security, coverage, restart durability, or warning-clean GREEN from source presence alone.

The issue therefore stays open until all of the following are proven on the integrated canonical lineage:

• fix: create workspace_entities/workspace_documents registry and provision Workspace rows #1503 migration reconciliation and a single Alembic head;
• fresh PostgreSQL migration/upgrade acceptance;
• multi-poll maildrop > cap eventual progress;
• restart durability and UIDL renumbering/deletion behavior;
• UIDL-unavailable fallback behavior with explicit correctness/resource trade-off;
• persistent/repeated negative RETR cases without lower-backlog starvation;
• duplicate-import accounting remains accurate;
• exact-current test/security/warning/coverage evidence is GREEN;
• qualifying post-last-push independent review exists.

Keep this issue open. The source repair is present, but closure requires integrated runtime evidence; no fixed message-number window, synthetic status, no-op wake commit, parallel migration head, or second POP3/dedupe implementation is acceptable.

[seonghobae]

Source repair is now present on canonical #1195 exact fcef1780986bdd0d0986342e71206b303e0dd27c and remains unaccepted pending exact-head evidence.

Implemented owner path now includes:

• durable unseen-UIDL progress, renumbering tolerance, explicit UIDL-unavailable fallback, and owner-scoped Pop3ObservedMessage / 0019_pop3_observed_uidl persistence;
• DB sessions outside provider network I/O and idempotent (tenant_config_id, provider_uidl) observation only after successful message persistence;
• QUIT recovery that preserves already retrieved bytes and explicitly closes the transport after graceful cleanup failure;
• partial RETR preservation plus a refined negative-response policy: source-order RED d9c87082355fd133a5f4ed9715c560eaee28b211, hardening RED 7606387c829e27845633d180642369547d23336b, final fix a5d7d21011cf96548cce18b8119a553123a25ab5, and doctoring fcef1780986bdd0d0986342e71206b303e0dd27c.

The RETR refinement matters because RFC 1939 permits repeated TRANSACTION-state commands and defines -ERR no such message as a valid RETR response. A standards-conforming per-message -ERR now leaves that identity retryable and continues to later selected messages; OSError or an unexpected non--ERR protocol response stops the rest of the bounded batch. No DELE is introduced.

Do not close this issue yet. Source-order RED/fix commits are not hosted GREEN evidence. Closure still requires exact-head stacked PostgreSQL migration/test/security/coverage receipts, post-last-push independent review, restart/renumbering/unsupported-UIDL acceptance, negative-RETR acceptance, and a large-mailbox provider-history profile on the exact integrated head.

[seonghobae]

Fresh current-head verification of #1195 fcef1780986bdd0d0986342e71206b303e0dd27c found one remaining durable-progress counterexample in the UIDL path.

Current selection filters persisted-success UIDLs, sorts the remaining identities by current message number, and chooses only the highest MAX_POP3_FETCH_MESSAGES candidates. RFC-style per-message -ERR is intentionally retryable and therefore is not written to pop3_observed_messages.

Counterexample: a static maildrop has >10 unseen UIDLs and the highest ten UIDLs each return a persistent standards-conforming -ERR to RETR. The worker correctly continues within that bounded batch, but records none of those ten as observed. The next poll selects the same ten again, so lower unseen UIDLs are starved indefinitely. The recent RETR repair therefore solves within-batch blocking but does not yet satisfy this issue's cross-poll eventual-progress invariant.

Do not "fix" this by marking every -ERR UIDL observed, because a transient rejection would then become silent permanent loss. A correct owner repair needs bounded durable retry/progress state (for example an owner-scoped retry/cursor/disposition contract with explicit retry policy) so failed identities cannot monopolize every poll while still remaining retryable. Because #1195's current POP3 schema revision also collides with #1503's canonical Alembic lineage, any new persistence revision must be designed/rechained only after the migration-owner ancestry is reconciled; do not add another parallel head.

Acceptance should include a >cap maildrop with at least cap persistent negative RETRs ahead of valid unseen messages, repeated polls/restart, and proof that valid backlog still advances while failed UIDLs remain bounded/retryable rather than silently discarded.

[seonghobae]

Canonical #1195 now contains a source-order repair for the remaining cross-poll starvation counterexample.

Exact lineage:

• RED 5d40f83ff48e7dd9c3bf6deb93995130c5a1470a adds selection contracts proving that never-attempted UIDLs must outrank due retries and that future retries/observed UIDLs are excluded.
• Fix 7406cb63923003df2c3a5d21a0104203f78b90bc extends the existing owner-scoped POP3 progress row with collection_disposition={observed,retryable} plus nullable retry_after. RFC-style per-message negative RETR and parse failures remain retryable; successful persistence transitions the UIDL to observed.
• The bounded selector now chooses never-attempted UIDLs before due retries. Thus ten persistent failed tail UIDLs cannot monopolize every poll while lower fresh backlog exists.
• Network RETR/UIDL work remains outside the DB session; collection-state writes occur only after provider I/O. No DELE is introduced.
• Doctoring 630d5461d1eb8b59b392e4170dec66ce1edf034c records the RFC 1939 basis, retry semantics and acceptance boundary.

The new regression models the issue's >cap persistent-negative-tail case and proves the next selection exposes lower fresh UIDLs while the failed identities remain retryable rather than silently observed.

This is not closure evidence. Exact 630d5461... still has zero repository-local PR workflow receipts, no post-last-push qualifying review, and the branch-local 0018/0019 migration lineage still must be ordinary-rechained after #1503 reaches protected ancestry. Keep #1717 open until the final migration-reconciled head has PostgreSQL/test/security/coverage receipts plus repeated-poll/restart/provider-history acceptance.