Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions .changeset/control-plane-classification.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
---
'@bugsee/core': minor
'@bugsee/browser-utils': minor
---

The control plane now reads the collector's own error code out of a FAILED response, and the browser /
worker dead-sibling queue gets an age bound.

Two halves of one asymmetry. The DATA plane classified server answers; the CONTROL plane did not, so a
`400` from `/v2/issues` was retried at every launch — bounded on node by the 7-day retention, and on
browser/worker not bounded at all.

**What is now classified.** `bugsee-api.ts` threw on a non-2xx status *before* parsing the body, so a
verdict the SDK honours perfectly on an HTTP 200 was discarded the moment the collector attached a status
to it: `14019 InvalidAppToken` delivered with a 400 was retried forever, and `99099 KillSdk` with a status
could not switch the SDK off at all. The failed body is now read for `{ ok: false, error: { code } }` and
the code goes on `BugseeError.serverCode`, where the existing (Android-drift-tested) `classifyServerErrorCode`
sees it. This adds no codes to the permanent set — it applies the set that was already there to a response
shape that was being thrown away. Android does the same on both endpoints.

**What is deliberately NOT classified.** A NAKED non-2xx — a status with no collector code behind it —
stays retryable. Android's own two endpoints disagree about the identical status, which is the tell that a
status is not a verdict about the payload; and on the web a bare 4xx is what an intermediary answers (a
captive portal, a MITM proxy, a WAF, a stale CDN route), none of which read the bytes. Pinned by tests, so
the decision is falsifiable rather than an omission.

**What bounds the rest.** `recoverSiblingBundleQueue` now applies `DEFAULT_DURABLE_RETENTION.maxAgeMs`
(7 days) — the same bound the same bytes already meet on the same tier when an instance recovers its own
queue, and the same TTL node's dead-subtree sweep uses. Only the age cap is mirrored, not the count/byte
caps: those evict the oldest *survivors*, which here would delete a report the collector was never asked
about. A frame with no staging timestamp (staged by an older SDK) is NEVER expired — reading unknown as the
epoch would delete every pending report on the launch after an upgrade.

`@bugsee/core` additionally exports `deserializeBundleFrame` and `DEFAULT_DURABLE_RETENTION`, so the bound
is read from one place rather than restated.
39 changes: 32 additions & 7 deletions docs/review/OPEN-FINDINGS.md
Original file line number Diff line number Diff line change
Expand Up @@ -660,12 +660,35 @@ get it wrong.

### New open items from round 4

- **The control plane never classifies at all** — asymmetry with the data plane. `bugsee-api.ts:82-84`
(`/v2/issues`) and `:107-109` (`/v2/sessions`) throw for every non-2xx and `upload-pipeline.ts:97-125`
defaults `permanent` to false, so a `400` from `/v2/issues` is retried forever. Fails SAFE (data kept),
bounded on node by the 7-day retention, **unbounded on browser/worker** (`recover-dead-instances.ts`
applies none). Deliberately NOT fixed: widening a deletion path unilaterally is what produced a new loss
path in each of the last three rounds. Needs a decision, and should land WITH its harness case.
- **The control plane never classifies at all** — **CLOSED**, on `fix/control-plane-classification`, in two
halves that are deliberately different kinds of thing.
- **What became a verdict:** the collector's OWN error code, when it arrives with a status attached.
`bugsee-api.ts` threw on `!isOk(status)` BEFORE parsing the body, so a code the SDK already honours on
an HTTP 200 was discarded the moment the collector also sent a status — `14019 InvalidAppToken` on a
400 retried forever, `99099 KillSdk` on a 400 unable to disable the SDK at all. The failed body is now
read (`httpFailure` → `serverErrorCodeOf`) and its code goes on `serverCode`, where the existing
drift-tested `classifyServerErrorCode` sees it. **No code was added to the permanent set** — the same
table now simply reaches a response shape that was being thrown away, which is what Android does on
both endpoints (`ReportUploadExecutor.java:468-484`, `CommunicationRequests.obtainSession`).
- **What did NOT become a verdict:** a naked non-2xx. Android's `/v2/issues` path falls back to
`classifyHttpStatus` (any non-401/408/425/429 4xx → PERMANENT → the bundle file is deleted), and this
SDK deliberately does not follow it, because Android's OWN `/v2/sessions` path falls back to
`classifyServerErrorCode(0)` → TRANSIENT for the identical status. Two adjacent calls to one collector
cannot both be right, and on the web a bare 4xx is what an intermediary answers. Pinned by tests
(`upload-pipeline.test.ts`, "a status is never a verdict here") so the decision is falsifiable.
- **What bounds the rest:** `recoverSiblingBundleQueue` now applies `DEFAULT_DURABLE_RETENTION.maxAgeMs`
(7 days) — the SAME bound these bytes already meet on this tier via the instance's own `recover()`, and
the TTL node's sweep uses. Age only: the count/byte caps evict the oldest SURVIVORS, which on this leg
would delete a report the collector was never asked about. A frame with no `firstSeenMs` is never
expired (the upgrade launch), and that rail is load-bearing — injecting `firstSeenMs ?? 0` is caught 23
times over by the pre-existing P2 in harness set H.
- **Harness:** set M gained 16 control cases (the same envelopes over a non-2xx, plus 6 naked statuses as
a fence), and a new set N drives the browser age bound across 4 launches over real IndexedDB. 369 cases,
0 violations. Reproduced first: pre-fix the new M cases give 7 violations (incl. `P7 the client KEPT
CAPTURING after the collector switched the SDK off`) and set N gives `it was still OFFERED 9x past the
age bound`. The §Evidence note that the harness oracle is flawed refers to the ROUND-1 copy in
`.session-artifacts/`; the committed `packages/instrumentation-tests/harness/invariants.mts` no longer
keys on `reportId ?? 'anon:'+summary` and imports no SDK predicate — verified before relying on it.
- **A doubly-403 signed PUT** returns `permanent: false` (`upload-pipeline.ts:169-186`, `renew_failed`)
where Android calls 403 PERMANENT. Fail-safe, and arguably right since each launch mints a fresh signed
URL. Left alone.
Expand Down Expand Up @@ -912,7 +935,9 @@ The live pipeline treats `permanent` as settled and frees the blob
(`recover-instances.ts:89`, `browser-utils/src/recover-dead-instances.ts:39`). So a 4xx-refused
bundle is re-uploaded every launch forever, and its marker is never retired either. Bounded at
7 days on node by `sweep-instances.ts:24`; **unbounded on browser/worker** —
`recoverSiblingBundleQueue` applies no retention and no `permanent` check.
`recoverSiblingBundleQueue` applies no retention and no `permanent` check. *(Both halves since fixed:
the `permanent` check by round 4's `isUploadSettled` unification, and the missing retention by the
7-day age bound on `recoverSiblingBundleQueue` — see §New open items from round 4.)*

### R2-4 · `skipReportIds.add` is not throw-safe — SEV3

Expand Down
72 changes: 70 additions & 2 deletions packages/browser-utils/src/coexistence.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -76,9 +76,15 @@ const okPipeline = (): UploadPipeline & { enqueue: ReturnType<typeof vi.fn> } =>
});

// Seed a bundle into another instance's prefix within the per-token bundle-queue database.
async function seedSibling(idb: IDBFactory, instanceId: string, bundleId: string, b: Bundle) {
async function seedSibling(
idb: IDBFactory,
instanceId: string,
bundleId: string,
b: Bundle,
firstSeenMs?: number,
) {
const shared = createIdbBlobStore({ databaseName: coexistenceDatabaseName(TOK), indexedDB: idb });
await shared.put(`${instanceId}/${bundleId}`, serializeBundle(b));
await shared.put(`${instanceId}/${bundleId}`, serializeBundle(b, firstSeenMs));
}
// Seed a report marker into a sibling's prefix within the per-token marker database. The bytes are a
// REAL serialized ReportMarker — coexistence now hydrates these to reconcile them against the bundle queue.
Expand Down Expand Up @@ -163,6 +169,68 @@ describe('createCoexistence — pass-through', () => {
});
});

describe('createCoexistence — the dead-sibling age bound, end to end', () => {
// The coordinator wires `recoverSiblingBundleQueue` with its DEFAULTS, so this is what actually
// decides how long a browser or worker keeps re-offering an undeliverable blob. Unit-testing the leg
// alone proves the arithmetic; only driving the real coordinator over real IndexedDB proves it is
// reached at all. On this tier an instance is dead the moment its tab closes, so before the bound a
// blob the collector never settled was replayed on every launch for the lifetime of the installation.
const DAY = 24 * 60 * 60 * 1000;

const unreachable = () => {
// Never settles: the collector is offline, or answering a bare 4xx this SDK keeps retrying.
const enqueue = vi.fn<UploadPipeline['enqueue']>(() => Promise.resolve({ ok: false }));
const drop = vi.fn<UploadPipeline['drop']>();
return { enqueue, drop, flush: () => Promise.resolve(true) } satisfies UploadPipeline;
};

const coexist = (idb: IDBFactory) =>
createCoexistence({
appToken: TOK,
persist: true,
indexedDB: idb,
locks: fakeLocks().manager,
});

it('gives up on a blob a dead sibling staged 90 days ago, and keeps yesterday’s', async () => {
const idb = new IDBFactory();
await seedSibling(idb, 'deadsib', 'ancient', aBundle('ancient'), Date.now() - 90 * DAY);
await seedSibling(idb, 'deadsib', 'recent', aBundle('recent'), Date.now() - DAY);
const pipeline = unreachable();

await coexist(idb).recoverDeadSiblings({ uploadPipeline: pipeline });

expect(pipeline.enqueue).toHaveBeenCalledTimes(1);
expect((pipeline.enqueue.mock.calls[0]?.[0] as Bundle).request.summary).toBe('recent');
expect(pipeline.drop).toHaveBeenCalledWith('retention_expired', 'issue');
const keys = await rawKeys(idb);
expect(keys).not.toContain('deadsib/ancient');
expect(keys).toContain('deadsib/recent'); // still pending — the next launch tries again
});

it('re-offers the SAME recent blob on launch after launch — the bound is age, not attempts', async () => {
const idb = new IDBFactory();
await seedSibling(idb, 'deadsib', 'b1', aBundle('kept'), Date.now() - DAY);
for (let launch = 0; launch < 3; launch += 1) {
const pipeline = unreachable();
await coexist(idb).recoverDeadSiblings({ uploadPipeline: pipeline });
expect(pipeline.enqueue).toHaveBeenCalledTimes(1);
expect(pipeline.drop).not.toHaveBeenCalled();
}
expect(await rawKeys(idb)).toContain('deadsib/b1');
});

it('never expires a blob staged by an older SDK, which carries no timestamp at all', async () => {
const idb = new IDBFactory();
await seedSibling(idb, 'deadsib', 'legacy', aBundle('legacy')); // no firstSeenMs
const pipeline = unreachable();
await coexist(idb).recoverDeadSiblings({ uploadPipeline: pipeline });
expect(pipeline.enqueue).toHaveBeenCalledTimes(1);
expect(pipeline.drop).not.toHaveBeenCalled();
expect(await rawKeys(idb)).toContain('deadsib/legacy');
});
});

describe('createCoexistence — bundle queue', () => {
it('writes under a per-token database + per-instance prefix and holds the instance lock', async () => {
const idb = new IDBFactory();
Expand Down
148 changes: 147 additions & 1 deletion packages/browser-utils/src/recover-dead-instances.test.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
import { type Bundle, serializeBundle, type UploadPipeline } from '@bugsee/core';
import {
type Bundle,
DEFAULT_DURABLE_RETENTION,
serializeBundle,
type UploadPipeline,
} from '@bugsee/core';
import { Severity } from '@bugsee/protocol';
import { describe, expect, it, vi } from 'vitest';
import type { AsyncBlobStore } from './idb';
Expand Down Expand Up @@ -128,3 +133,144 @@ describe('recoverSiblingBundleQueue', () => {
expect(pipeline.enqueue).not.toHaveBeenCalled();
});
});

// ══════════════════════════════════════════════════════════════════════════════════════════════════
// THE AGE BOUND — what stops "retried at every launch" from meaning "forever" on this tier
// ══════════════════════════════════════════════════════════════════════════════════════════════════
//
// Everything above is about a SETTLED answer. The dangerous case is the one that never settles: a
// collector that is simply unreachable, or a control-plane failure this SDK deliberately keeps retrying
// (a bare 4xx on `/v2/issues` — see `upload-pipeline.test.ts`, "a status is never a verdict here").
//
// On node such a blob is bounded twice over: `sweep-instances` reaps a dead instance's whole subtree at
// 7 days, and `recover()` applies the durable queue's own retention. This leg has neither — it reads a
// dead instance's prefix DIRECTLY — and on the web an instance is dead the moment its tab closes. So a
// blob that never settles was re-offered on every launch for the lifetime of the installation.
//
// The bound applied here is not a new policy: it is `DEFAULT_DURABLE_RETENTION.maxAgeMs`, the SAME
// 7 days already applied to the SAME bytes on the SAME tier by the instance's own `recover()`, and the
// same TTL node's sweep uses. Only the count and byte caps are deliberately left off — see below.
describe('recoverSiblingBundleQueue — the retention age bound', () => {
const DAY = 24 * 60 * 60 * 1000;
const NOW = 1_800_000_000_000;
const failing = () => {
// The collector is unreachable — never a refusal, so nothing here is ever "settled".
const enqueue = vi.fn<UploadPipeline['enqueue']>(() => Promise.resolve({ ok: false }));
const drop = vi.fn<UploadPipeline['drop']>();
return { enqueue, drop, flush: () => Promise.resolve(true) } satisfies UploadPipeline;
};

it('stops re-offering a blob past the age bound, and says so rather than deleting it silently', async () => {
const shared = memBlobWith([
[
'dead/old',
serializeBundle(aBundle('a crash from a tab closed months ago'), NOW - 90 * DAY),
],
]);
const pipeline = failing();
await recoverSiblingBundleQueue(shared.store, 'dead', pipeline, undefined, { now: () => NOW });
expect(pipeline.enqueue).not.toHaveBeenCalled();
expect(shared.map.has('dead/old')).toBe(false);
// Announced, exactly as `durable-upload-pipeline` announces its own evictions: a bundle that
// vanishes without an outcome is indistinguishable from one that was delivered.
expect(pipeline.drop).toHaveBeenCalledWith('retention_expired', 'issue');
});

it('keeps replaying a blob INSIDE the bound, launch after launch', async () => {
const shared = memBlobWith([['dead/recent', serializeBundle(aBundle('yesterday'), NOW - DAY)]]);
const pipeline = failing();
for (let launch = 0; launch < 3; launch += 1) {
await recoverSiblingBundleQueue(shared.store, 'dead', pipeline, undefined, {
now: () => NOW,
});
}
expect(pipeline.enqueue).toHaveBeenCalledTimes(3);
expect(shared.map.has('dead/recent')).toBe(true);
expect(pipeline.drop).not.toHaveBeenCalled();
});

it('is EXCLUSIVE at the bound — exactly maxAgeMs old is still replayed', async () => {
const shared = memBlobWith([
['dead/at', serializeBundle(aBundle('at'), NOW - DEFAULT_DURABLE_RETENTION.maxAgeMs)],
['dead/past', serializeBundle(aBundle('past'), NOW - DEFAULT_DURABLE_RETENTION.maxAgeMs - 1)],
]);
const pipeline = failing();
await recoverSiblingBundleQueue(shared.store, 'dead', pipeline, undefined, { now: () => NOW });
expect(pipeline.enqueue).toHaveBeenCalledTimes(1);
expect((pipeline.enqueue.mock.calls[0]?.[0] as Bundle).request.summary).toBe('at');
expect(shared.map.has('dead/at')).toBe(true);
expect(shared.map.has('dead/past')).toBe(false);
});

it('NEVER expires a frame that predates `firstSeenMs` — the upgrade launch', async () => {
// `serializeBundle(bundle)` with no timestamp is what every blob staged by an older SDK looks like.
// Reading "unknown" as the epoch would delete every pending crash report on the first launch after
// an upgrade — losing exactly the reports the upgrade was installed to deliver. Same rule as
// `durable-upload-pipeline.recoverPass`.
const shared = memBlobWith([['dead/legacy', serializeBundle(aBundle('legacy'))]]);
const pipeline = failing();
await recoverSiblingBundleQueue(shared.store, 'dead', pipeline, undefined, { now: () => NOW });
expect(pipeline.enqueue).toHaveBeenCalledTimes(1);
expect(shared.map.has('dead/legacy')).toBe(true);
expect(pipeline.drop).not.toHaveBeenCalled();
});

it('defaults to the durable queue’s own 7 days, not a second copy of the number', async () => {
const shared = memBlobWith([
[
'dead/old',
serializeBundle(aBundle('old'), Date.now() - DEFAULT_DURABLE_RETENTION.maxAgeMs - 60_000),
],
['dead/new', serializeBundle(aBundle('new'), Date.now() - 60_000)],
]);
const pipeline = failing();
await recoverSiblingBundleQueue(shared.store, 'dead', pipeline); // no clock, no bound: the defaults
expect(pipeline.enqueue).toHaveBeenCalledTimes(1);
expect(shared.map.has('dead/old')).toBe(false);
expect(shared.map.has('dead/new')).toBe(true);
});

it('honours an explicit maxAgeMs override', async () => {
const shared = memBlobWith([['dead/b', serializeBundle(aBundle('b'), NOW - 2 * DAY)]]);
const pipeline = failing();
await recoverSiblingBundleQueue(shared.store, 'dead', pipeline, undefined, {
now: () => NOW,
maxAgeMs: DAY,
});
expect(pipeline.enqueue).not.toHaveBeenCalled();
expect(shared.map.has('dead/b')).toBe(false);
});

it('reports a failed eviction to onError instead of throwing out of launch', async () => {
const onError = vi.fn();
const shared = memBlobWith([['dead/old', serializeBundle(aBundle('old'), NOW - 90 * DAY)]]);
const store: AsyncBlobStore = {
...shared.store,
remove: () => Promise.reject(new Error('idb gone')),
};
const pipeline = failing();
await expect(
recoverSiblingBundleQueue(store, 'dead', pipeline, onError, { now: () => NOW }),
).resolves.toBeUndefined();
expect(onError).toHaveBeenCalledTimes(1);
expect(pipeline.enqueue).not.toHaveBeenCalled(); // still withheld — the bound holds either way
});

it('bounds by AGE ONLY — a hundred fresh blobs are all still offered', async () => {
// The count and byte caps `recover()` also applies are deliberately NOT mirrored here. They evict
// the OLDEST survivors to make room, which on this leg would mean deleting a crash report the
// collector has never been asked about, purely because a burst arrived after it. Age is the one
// bound where "give up" and "this is worthless now" are the same statement.
const shared = memBlobWith(
Array.from({ length: 100 }, (_, i): [string, Uint8Array] => [
`dead/b${i}`,
serializeBundle(aBundle(`b${i}`), NOW - DAY),
]),
);
const pipeline = failing();
await recoverSiblingBundleQueue(shared.store, 'dead', pipeline, undefined, { now: () => NOW });
expect(pipeline.enqueue).toHaveBeenCalledTimes(100);
expect(shared.map.size).toBe(100);
expect(pipeline.drop).not.toHaveBeenCalled();
});
});
Loading
Loading