WDQS 0.3.156 updater backoff silently skips changes created during fresh-instance seed (one-time bootstrap flow)

[Ron-RONZZ-org]

Summary

The classic WDQS updater (RecentChangesPoller, version 0.3.156 — the same code is still present on master, see §Mechanism) can silently skip changes created while it is mid-catch-up on a fresh instance. The updater then logs Got no real changes indefinitely and the affected entities never reach Blazegraph — no error, no warning. We hit this deterministically in CI while bootstrapping a fresh instance (one-time seed), which is exactly the deployment flow wmde/wikibase-docker documents.

Mechanism

RecentChangesPoller.doFetchRecentChanges switches between two fetch strategies:

private static final Duration BACKOFF_TIME = Duration.ofSeconds(10);      // line 77
private static final Duration BACKOFF_THRESHOLD = Duration.ofMinutes(2); // line 81

private boolean changeIsRecent(Instant nextStartTime) {
    return nextStartTime.isAfter(Instant.now().minus(BACKOFF_THRESHOLD));
}

private RecentChangeResponse doFetchRecentChanges(Instant lastNextStartTime, Batch lastBatch) {
    if (useBackoff && changeIsRecent(lastNextStartTime)) {
        // fetch by TIME (rcstart = lastNextStartTime - 10s), no continuation token
        return wikibase.fetchRecentChangesByTime(lastNextStartTime.minus(BACKOFF_TIME), batchSize);
    } else {
        // fetch by CONTINUATION (rccontinue) — deterministic paging
        return wikibase.fetchRecentChanges(lastNextStartTime, lastBatch.getLastContinue(), batchSize);
    }
}

On a fresh instance the poller starts from now - maxDaysBack in continuation mode. As its cursor approaches "now" (within the 2-minute BACKOFF_THRESHOLD) it switches to time-based backoff fetching, and each cycle advances the time cursor by ~1 second (log: Backoff overflow, advancing next time to <T>). If a new entity revision is created while the cursor is advancing and its timestamp ends up before the already-advanced cursor, that change is never revisited — the poller simply never sees it.

Impact

Silent missing triples. The operator sees Got no real changes (the normal idle log line) and has no indication that data was skipped. This is the core problem: even if the backoff design is intentional (re-fetching near-now windows to catch unsettled transactions is sensible in steady state), skipping unseen changes without any warning is not. A fresh-instance bootstrap with concurrent edits — the documented one-time-seed flow — hits it deterministically.

Reproduction

Deterministic in our CI (we observed the same failure across ~10 runs):

• Stack: wikibase/wikibase:mw1.46.0 + wikibase/wdqs:wdqs0.3.156 + MariaDB (compose: dev/docker-compose.ci.yml in Ron-RONZZ-org/ronzz-wikibase).
• Step 1: create ~95 entities (11 properties, 4 classes, 80 items) via MediaWiki maintenance scripts (EntityStore).
• Step 2: create 5 more items via the API while the updater is still catching up.
• Result: the poller processes changes up to Q84@180|181 (Got 2 changes, from Q83@178 … to Q84@180), then Backoff overflow + Got no real changes forever — while action=query&list=recentchanges&rclimit=500&rcnamespace=0|120 demonstrably returns the skipped changes (rcid 186–189, Item:Q88/Q89). The 5 late entities never reach Blazegraph.

The updater log evidence is captured in the workflow logs of the linked repo's CI runs.

Suggested fix directions (not a patch)

1. Do not enter backoff mode until the continuation is exhausted (or carry the continuation token across the backoff transition), so unseen changes cannot fall behind the advancing time cursor.
2. At minimum: warn loudly when the backoff cursor advances past changes that were fetched but never processed, or when a time-based fetch excludes changes newer than the previously processed position. The silence is the actual bug.

Honest caveats

• We are on 0.3.156 because the pinned wmde docker images ship the classic updater; the newer streaming updater in this repo is not a drop-in for us. An upstream note on the classic updater's support/upgrade path would help.
• Our assessment that steady-state (already-caught-up) polling is unaffected is an inference from the source, not a formal proof; we have not stress-tested the steady-state path.
• The reproduction is deterministic but heavyweight (full wikibase-docker stack); we can provide the compose file and exact CI steps on request.

Environment

• WDQS 0.3.156 (wikibase/wdqs:wdqs0.3.156), MediaWiki 1.46.0 (Wikibase Suite image), MariaDB 11.
• The code's canonical tracker is Phabricator (project Wikidata Query Service) — the wikimedia/wikidata-query-rdf GitHub repo has issues disabled. Filing here (wmde/wikibase-suite) because your documented one-time-seed / quickstart flow is the affected deployment, and so the report reaches the WDQS maintainers for routing. The full ticket content above is ready to paste into a Phabricator task if preferred.