From 7f66dc80b3e61246ca3970d61736e772128d92f3 Mon Sep 17 00:00:00 2001 From: Alejandro Date: Sun, 27 Sep 2026 03:45:43 -0400 Subject: [PATCH 01/56] docs(spec): pipeline v2 campaign/claim/inventory redesign Design for #50 (folds in #49 streamer priority): platform-authoritative per-drop state, pure planner, session/claimer split, v1/v2 flag rollout. Co-Authored-By: Claude Opus 5.5 (1M context) --- .../specs/2026-09-27-pipeline-v2-design.md | 393 ++++++++++++++++++ 1 file changed, 393 insertions(+) create mode 100644 docs/superpowers/specs/2026-09-27-pipeline-v2-design.md diff --git a/docs/superpowers/specs/2026-09-27-pipeline-v2-design.md b/docs/superpowers/specs/2026-09-27-pipeline-v2-design.md new file mode 100644 index 0000000..0f5a416 --- /dev/null +++ b/docs/superpowers/specs/2026-09-27-pipeline-v2-design.md @@ -0,0 +1,393 @@ +# Pipeline v2: campaign / progress / claim / inventory redesign + +Issue: #50 (redesign), #49 (streamer priority, folded in). +Status: design approved in brainstorming 2026-09-27; spec pending review. + +## 1. Goal + +Replace the accreted watcher state machine (`internal/watcher/watcher.go`, +~2,050 LOC, ~20 layered special cases) with a pipeline built around one +authoritative per-drop state model reconciled from the platform. + +### Success criteria + +1. A drop claimed anywhere (in app, on the website, Kick sweep, or with a lost + claim response) ends up `claimed` in GrubDrops and the account moves on + without user action. +2. The inventory / drops page shows correct status without manual marking. +3. For any drop, one state + reason explains why it is or isn't being mined. +4. Streamer priority works: per-account priority streamers first, category + fallback after (#49). +5. No drop is stuck forever: every `blocked` state carries a `retry_after` and + a UI "retry now". + +### Scope + +Starting point: **the account already has a valid session.** + +In scope: campaign discovery + filtering, campaign/channel selection +(incl. #49), watcher lifecycle, accrual orchestration, progress checks, +claiming, drop/claim state + reconciliation, inventory page state, the +backend interface the pipeline calls. + +Out of scope: login / session acquisition, authcheck, #48, general web UI, +settings (except the streamer priority control), notifications, SSO, proxy, +sidecar internals (browser stack stays core, per CLAUDE.md). + +Invariants: +- Accrual mechanics unchanged: Twitch Spade beacon at a fixed 60s + `HeartbeatInterval`; Kick WS / IVS watch path as-is. Only orchestration + around them changes. +- Existing DB data migrates in place. No forced reset. +- Stay Go + html/template + HTMX. + +## 2. Problems being fixed (from code map, master @ efb8b39) + +| # | Problem | Current cause | +|---|---|---| +| P1 | Externally claimed Twitch drop shows unclaimed forever, wastes a watch | Reconcile reads only `dropCampaignsInProgress`; finished campaigns vanish from it; ghost-skip persists a skip but never a claim row | +| P2 | Lost claim response → same as P1 | Claim error path relies on reconcile that can't see finished campaigns | +| P3 | Endless claim retry loop | No failure counter; non-success statuses loop pick → watch → claim | +| P4 | Kick link-gated claim loops in the watcher | `needsLink` guards only the sweep; Kick forces `AccountLinked=true` | +| P5 | Watch continues after drop claimed mid-watch | `tickWatch` never reads `Progress.Claimed` | +| P6 | Kick minutes never reach required | Adapter maps 0 → 120 required; progress uses `required_units` | +| P7 | Kick ghost-skip deadlock | Reward listed only after accrual; skip blocks accrual; no UI to clear `skip_override` | +| P8 | Prune flip-flop deletes genuine claim rows | Prune on transient `IsClaimed=false` | +| P9 | Precondition never unlocks after manual mark | Gate reads inventory `claimed`, not local state | +| P10 | Manual mark restarts every watcher | `addClaim` calls full `loadAndStart` reload | +| P11 | Kick sweep claims without writing state | Sweep bypasses `claims` | +| P12 | Platform logic leaks into generic code | `Platform == "twitch"` checks, `oilrats`, `Game: "Rust"`, placeholder-ID heuristic | + +## 3. Architecture + +``` + ┌──────────────┐ state changes ┌────────────┐ + platform ───▶ │ Reconciler │ ───────────────▶ │ │ + (detail/ └──────────────┘ │ │ + progress) │ Account │──▶ planner.Plan(snapshot) + ┌──────────────┐ Progress/ │ loop │ │ + platform ◀──▶ │ Session │ StreamDown/ │ (1 gorout- │◀───────┘ Decision + (watch) └──────────────┘ Stalled ──────▶ │ ine owns │ + ┌──────────────┐ │ state) │ + platform ◀─── │ Claimer │ ◀── claimable ── │ │ + └──────────────┘ ── result ─────▶ │ │ + UI pings (mark / skip / retry / priority) ────▶ └────────────┘ + │ + drop_state (DB) ──▶ /drops, dashboard +``` + +New packages (names indicative, final in plan): + +- `internal/pipeline/dropstate`: state model, transitions, `DropStateStore`. +- `internal/pipeline/reconcile`: platform → drop_state sync. +- `internal/pipeline/planner`: pure `Plan(PlanInput) Decision`. +- `internal/pipeline/session`: one accrual run on one channel. +- `internal/pipeline/claimer`: claim policy + backoff. +- `internal/pipeline/loop`: per-account event loop wiring the above. + +v1 (`internal/watcher`) is untouched and stays selectable until v2 is +live-verified; then deleted in a follow-up release. + +## 4. Drop state model + reconciler + +### 4.1 `drop_state` table (new migration `0016_drop_state.sql`) + +One row per (account_id, drop_id). + +| column | type | meaning | +|---|---|---| +| `account_id` | TEXT | FK accounts | +| `drop_id` | TEXT | platform drop / reward id (Twitch timeBasedDrop id; Kick reward id) | +| `campaign_id` | TEXT | owning campaign | +| `platform` | TEXT | `twitch` / `kick` | +| `status` | TEXT | `eligible`, `accruing`, `claimable`, `claimed`, `blocked` | +| `block_reason` | TEXT NULL | `needs_link`, `claim_failed`, `sub_only`, `no_channels`, `expired`, `not_enrolled`, `user_skip` | +| `minutes` | INTEGER | platform-reported watched minutes | +| `required` | INTEGER | platform-reported required minutes | +| `source` | TEXT | `platform` or `user` (who last set status) | +| `fail_count` | INTEGER | consecutive claim failures | +| `retry_after` | TIMESTAMP NULL | when a `blocked` row is re-evaluated | +| `synced_at` | TIMESTAMP | last platform confirmation | +| `updated_at` | TIMESTAMP | | + +PK (account_id, drop_id). Index on (account_id, status). + +Status meaning: +- `eligible`: mineable, no progress yet. +- `accruing`: 0 < minutes < required. +- `claimable`: minutes ≥ required (platform-reported) and not claimed. +- `claimed`: platform says claimed, or our claim call succeeded, or user mark. +- `blocked`: not mineable now; `block_reason` says why, `retry_after` says when to look again. + +Multi-item Twitch drops (one timeBasedDrop, several rewards) are **one** +drop_state row keyed by the timeBasedDrop id; the reward names are display +data held on the benefits side, not separate state. + +### 4.2 Reconciler + +Per account, own goroutine. Emits `StateChanged` events to the loop; never +decides what to watch. + +**Triggers:** start-up; every `GRUB_RECONCILE_INTERVAL` (default 15 min); +immediately after any claim attempt; on `Stalled` from a session; on UI +"retry now". + +**Twitch:** +1. `ListActiveCampaigns`, filtered to whitelisted games for this account. +2. For each in-scope campaign, `DropCampaignDetails` (already wired, + `twitch/campaigns.go` with `detailsCache`). Per `timeBasedDrop.self`: + `isClaimed`, `currentMinutesWatched`, `dropInstanceID`. Details cache is + bypassed on post-claim reconciles. +3. Map to status. This works for campaigns that have left + `dropCampaignsInProgress`, which is the fix for P1/P2. + +Verification item (plan task 1): confirm the persisted query response +carries `self { isClaimed currentMinutesWatched dropInstanceID }` for an +enrolled but completed campaign. If `self` is absent, fall back to +`Inventory.dropCampaignsInProgress` + per-drop claim result, and the spec is +revisited before building on it. + +**Kick:** +1. `/drops/progress` (Bearer session) + `CampaignDetails`. +2. `required` always comes from the platform (`required_units`); the adapter + no longer invents 120 (fixes P6). A reward with no platform minutes and no + progress entry is `eligible` with `required` from campaign detail. +3. Verification item (plan task 1): does `/drops/progress` keep claimed + rewards listed with `claimed=true`, or drop them? + - If kept: straightforward mapping. + - If dropped: a reward that disappears is `claimed` **only** if our own + claim call returned success for it; otherwise `blocked:not_enrolled` + with `retry_after` (no inference from absence). + +**Rules:** +- `claimed` from the platform is monotonic: never demoted by a later sync. + No prune (fixes P8). +- A `user` mark stands while the platform is silent about the drop; a + definite platform answer overwrites it (`source` flips to `platform`). +- `blocked` rows are re-evaluated when `now ≥ retry_after`; there is no + permanent skip (replaces ghost-skip + self-heal, fixes P7). +- Every transition into `claimed` also appends to `claims` (history), via + `RecordClaimIfNew`, regardless of who claimed (fixes P11). + +### 4.3 Removed by this section (v2 path) + +`tracked` set + prune, ghost-skip + self-heal, vanish detection, +`skip_override:` / `collect_override:` kv reads, full reload on manual mark. + +### 4.4 Migration + +`0016_drop_state.sql` creates the table. A one-shot Go backfill at startup +(idempotent, guarded by a kv flag `drop_state_backfilled`): +- `claims` rows → `claimed`, `source=platform`. +- `collect_override:` → `claimed`, `source=user`. +- `skip_override:` → `blocked:not_enrolled`, `retry_after=now` (so they get + one fresh evaluation instead of staying dead). + +Old kv keys are left in place until v1 is deleted, so rollback to v1 is safe. + +sqlc: no `?` or parentheses in `queries/*.sql` comments. + +## 5. Planner + channel selection (#49) + +### 5.1 Contract + +```go +func Plan(in PlanInput) Decision + +type PlanInput struct { + Now time.Time + Drops []dropstate.Row // this account + Campaigns []platform.Campaign // in-scope, with windows + preconditions + Settings AccountSettings // game priority, streamer priority, force-watch + Live map[CampaignID][]Stream // candidate live channels per campaign + Cooldowns map[ChannelKey]time.Time // stalled channels + Current *Decision // for swap hysteresis + LastSwap time.Time +} + +type Decision struct { + Kind DecisionKind // Mine, ForceWatch, Idle + Channel Stream + Serves []DropID // all candidate drops this channel accrues + Reason string // human-readable, shown on dashboard +} +``` + +Pure: no I/O, no clock reads, no goroutines. + +### 5.2 Algorithm + +1. **Candidates:** drops in `eligible` / `accruing`, game whitelisted, now + inside campaign window, preconditions satisfied. Preconditions read + `drop_state` (a user-marked or platform-claimed precondition unlocks; + fixes P9). `sub_only` (0-minute Twitch) drops are `blocked:sub_only` + upstream and never candidates. +2. **Rank campaigns:** game priority → ends soonest → lowest-tier first. +3. **Channel for top campaign:** + 1. Streamer priority list (per account, ordered) ∩ live ∩ right category + ∩ campaign allow-list (if restricted) ∩ not on cooldown. + 2. Fallback: other eligible live channels by viewers. + If none, try next campaign. If no campaign yields a channel → force-watch + or idle. +4. **Serves:** every candidate drop (any campaign) that the chosen channel + accrues. Kick: typically several; Twitch: typically one. +5. **Hysteresis:** if `Current` is still valid (channel live, serves ≥1 + candidate) and `Now - LastSwap < 10m`, keep it. A dead/stalled current + channel bypasses the hold. +6. **Force-watch:** only when no Mine decision exists; first **live** + force-watch channel for the account. Idle otherwise, with reason + (`no_games`, `no_live_channels`, `all_claimed`, ...). + +### 5.3 Channel probing + +New optional backend capability: + +```go +type ChannelProber interface { + ProbeChannels(ctx context.Context, s Session, c Campaign, logins []string) ([]Stream, error) +} +``` + +The loop calls it for priority streamers so low-viewer favourites outside +Twitch's top-30 directory are still found. Kick implements it on top of the +existing `probeLive`. Twitch implements it with the existing per-login live ++ game check, taking logins from the argument (not the cached allow-list). + +### 5.4 Storage + UI + +New table `account_streamer_priority(account_id, platform, login, rank)`, +created in `0016` alongside drop_state. Existing `account_channels` +(null-game whitelist) rows are copied in, keeping order; the null-game gate +reads the new table. `account_channels` is dropped when v1 is deleted. + +UI: ordered streamer list per account next to game priority (drag-reorder +or up/down, reuse existing priority control pattern). Editing pings the +loop; no reload. + +### 5.5 Removed + +`Platform == "twitch"` check in pick, no-stream campaign set, `oilrats` +preference, scattered sorts in `pickCampaign`, force-watch without liveness. + +## 6. Session, claimer, loop, backend interface + +### 6.1 Session + +Wraps existing `StartWatch` / `Heartbeat` / `StopWatch`. Emits: +- `Progress{DropID, Minutes, Required, Claimed}` (Twitch PubSub or Kick poll) +- `StreamDown` +- `Stalled` (no minute gain for N consecutive polls; N from current freeze + detector) + +Reports only; makes no decisions. `Progress.Claimed=true` is honoured by the +loop immediately (fixes P5). + +### 6.2 Claimer + +Triggered when a drop enters `claimable`. + +| Outcome | Action | +|---|---| +| Success / Twitch `DROP_INSTANCE_ALREADY_CLAIMED` | → `claimed`; trigger reconcile | +| Link required (Kick connect_url, Twitch unlinked) | → `blocked:needs_link`, no retry; UI "link, then retry" (fixes P4) | +| Other error | `fail_count++`, backoff 1m → 5m → 30m; after 5 → `blocked:claim_failed`, `retry_after` 6h (fixes P3) | +| Lost response | post-claim reconcile reads `isClaimed` and corrects (fixes P2) | + +Kick sweep becomes claimer policy: on each Kick progress poll, claim +every drop in `claimable` for that account. Same state writes as a normal +claim, so no unrecorded claims. Hardcoded `Game: "Rust"` removed. + +### 6.3 Account loop + +```go +events := merge(reconciler.Events, session.Events, ticker.C, uiPings) +for ev := range events { + apply(ev) // mutate drop_state via DropStateStore + d := planner.Plan(snapshot()) + if !d.Equal(current) { swapSession(d) } +} +``` + +- Single goroutine owns all transitions. +- Live re-check ticker: 5 min (feeds `Live` for the planner, including + priority-streamer probes). +- UI actions (mark collected, skip, retry, priority edit) send a ping; no + `Reload` (fixes P10). +- Exposes a snapshot (status, decision reason, channel, serving drops) to the + scheduler for the dashboard, via the existing scheduler state API. + +### 6.4 Backend interface + +Split for the pipeline (login methods stay on the existing `Backend`, out of +the pipeline): + +```go +type CampaignSource interface { ListActiveCampaigns(ctx, s) ([]Campaign, error) } +type ProgressSource interface { DropProgress(ctx, s, campaigns []Campaign) ([]DropProgress, error) } +type ChannelSource interface { ListEligibleChannels(ctx, s, c) ([]Stream, error) } +type ChannelProber interface { ProbeChannels(ctx, s, c, logins) ([]Stream, error) } +type WatchSource interface { StartWatch(...); Heartbeat(...); StopWatch(...) } +type ClaimSink interface { Claim(ctx, s, DropProgress) (ClaimResult, error) } +``` + +`ClaimResult` carries typed outcomes (`Claimed`, `AlreadyClaimed`, +`NeedsLink`, `Failed`) instead of string-matched errors. + +A `pipeline.Platform` struct is assembled once per backend at construction, +holding whichever capabilities it has; no type assertions in the loop. Store +access through one `DropStateStore` interface (replaces anonymous +assertions `ClaimedBenefitIDs` / `RecordClaimIfNew` / `PruneClaim` / +`PersistAccountLinks`). + +Platform quirks stay in adapters: Kick required-minutes normalisation, +Twitch placeholder-ID handling, Kick channel preferences. + +### 6.5 Rollout + +- `GRUB_PIPELINE=v1|v2` (default `v1`), overridable per account (account + settings toggle) so v2 can run on one account first. +- `cmd/miner/main.go` `build()` branches on the selection; v1 wiring + unchanged. +- Default flips to `v2` only after live verification on a Twitch drop and a + Kick drop (CLAUDE.md accrual/claim gate). v1 deletion (incl. + `watcher.go`, `skip_override`/`collect_override` loaders, `account_channels`) + is a separate follow-up release. + +## 7. Error handling + +- Platform call errors inside reconcile: log, keep last known state, retry + next tick. Never demote state on error. +- Session start failure: channel onto cooldown, re-plan. +- DB write failure: loop logs and retries the apply on next event; planner + runs on in-memory snapshot so mining continues. +- `ErrIntegrityBlocked` and session-expiry surface to the scheduler as + today (auth is out of scope); v2 does not change their handling. + +## 8. Testing + +- **Planner:** table-driven tests. Cases: priority streamer live / offline / + not in allow-list, fallback, hysteresis hold and dead-channel bypass, + preconditions via user mark, force-watch liveness, idle reasons, Kick + multi-serve. +- **Claimer:** table tests over `ClaimResult` outcomes and backoff ladder. +- **Reconciler:** fake backend responses. Cases: finished Twitch campaign + with `isClaimed=true` (P1), lost claim response (P2), Kick claimed + retained vs. dropped, user mark overwritten by platform, `claimed` never + demoted, blocked retry_after expiry. +- **Loop:** event-sequence tests with fake clock. Cases: claimed on website + mid-watch (P5), stall → cooldown → swap, UI ping re-plans without reload. +- **Migration:** backfill from claims + kv overrides, idempotent on rerun. +- **Adapters:** Kick required-minutes from platform (P6); Twitch + `DropProgress` from details `self`. +- Every regression test must fail against a deliberate revert of its fix. + +Release gate: green build + unit tests + Kick WS canary, then live-drop +verification (Twitch and Kick) on staging before any tag that enables v2. + +## 9. Open verification items (plan task 1, before building on them) + +1. Twitch `DropCampaignDetails` returns `self` for enrolled completed + campaigns. +2. Kick `/drops/progress` retention of claimed rewards. + +Either answer may adjust §4.2 mapping; the rest of the design is unaffected. From 4a14b4b26e5e118adc99d613e210b5813d59bfaa Mon Sep 17 00:00:00 2001 From: Alejandro Date: Sun, 27 Sep 2026 04:05:24 -0400 Subject: [PATCH 02/56] docs(plan): pipeline v2 core implementation plan Co-Authored-By: Claude Opus 5.5 (1M context) --- .../plans/2026-09-27-pipeline-v2-core.md | 4298 +++++++++++++++++ 1 file changed, 4298 insertions(+) create mode 100644 docs/superpowers/plans/2026-09-27-pipeline-v2-core.md diff --git a/docs/superpowers/plans/2026-09-27-pipeline-v2-core.md b/docs/superpowers/plans/2026-09-27-pipeline-v2-core.md new file mode 100644 index 0000000..a8f63fb --- /dev/null +++ b/docs/superpowers/plans/2026-09-27-pipeline-v2-core.md @@ -0,0 +1,4298 @@ +# Pipeline v2 Core Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Build the headless pipeline v2 (platform-authoritative drop state, reconciler, pure planner with streamer priority, claimer, session, per-account loop), selectable per account behind `GRUB_PIPELINE=v2`, with v1 untouched. + +**Architecture:** One `drop_state` row per (account, drop) is the single mining truth. A reconciler folds platform per-campaign detail into it; a pure planner turns state + live channels into a decision (channel + drops served); a session runs accrual on that channel; a claimer turns claimable rows into claimed/blocked. One goroutine per account (the loop) owns every state transition. v1 (`internal/watcher`) is not modified except where noted (scheduler interfaces, a Twitch claim refactor that keeps behaviour identical). + +**Tech Stack:** Go 1.x, SQLite via sqlc + goose migrations, testify. No new dependencies. + +**Spec:** `docs/superpowers/specs/2026-09-27-pipeline-v2-design.md` + +**Scope split:** This is plan 1 of 2. Plan 2 (written after this lands) covers the UI: inventory page reading `drop_state`, mark/skip/retry actions via `Loop.Nudge`, streamer-priority editor, per-account pipeline toggle. Plan 1 is usable on its own: v2 mines headlessly, and the existing UI keeps working (manual marks reach v2 through a claims-table bridge on loop start, because the mark handler already triggers a reload). + +**Refinements to the spec made here (recorded in the spec in Task 13):** +- Reconcile runs serially on the loop goroutine (not its own goroutine). Single writer, no result races; session and PubSub events buffer meanwhile. +- Timestamps are INTEGER unix seconds (repo convention), `0` = unset. `block_reason` is `NOT NULL DEFAULT ''`. +- `user_skip` has no auto-expiry (`retry_after = 0`); it is explicit user intent and is cleared only by the user (Plan 2 "retry"). +- Platform interface names: `DropProgressSource`, `DropClaimer`, `ChannelProber` in `internal/platform`. + +## Global Constraints + +- Twitch heartbeat cadence is **fixed at 60s** (`BeatEvery = 60 * time.Second`, not configurable). Anything above 60s under-credits Twitch. +- Never send a `Client-Integrity` header (no Twitch HTTP changes in this plan add headers). +- **Never put `?` or parentheses in a `queries/*.sql` comment.** +- Do not delete or modify `internal/watcher`, the browser sidecar stack, `proto/`, `internal/dockerctl`, `internal/auth/browser`. +- `gofmt -w .` before every commit; `go build ./...` and `go test ./...` green before every commit. +- Every change gets a line in `docs/CHANGELOG.md` under `## [Unreleased]` (done in Task 13 for the whole feature). +- Default pipeline stays `v1`. v2 is enabled only by `GRUB_PIPELINE=v2` or kv `pipeline_override:` = `v2`. +- Accrual/claim change: **no version tag** until live-drop verification on staging (Task 14) passes on Twitch and Kick. +- Commits end with `Co-Authored-By: Claude Opus 5.5 (1M context) `. + +## Review Focus + +1. **Platform error mid-reconcile** (Twitch details 500, Kick progress 403): state must not change at all. Pinned in Task 7 (`TestRun_ProgressErrorChangesNothing`). +2. **Stale session events after a channel swap**: a `StreamDown` from the old channel must not put the new channel on cooldown. Pinned in Task 11 (`TestLoop_StaleStreamDownIgnored`). +3. **Twitch drop with `self: null`** (campaign not yet enrolled): must be `eligible`, never `claimed`. Pinned in Task 5 (`TestDropProgress_NullSelfIsUnclaimed`). +4. **Kick reward absent from `/drops/progress`**: never-watched stays `eligible`; absent after accruing goes `blocked:not_enrolled` then re-derives after 1h (no permanent skip). Pinned in Task 2 table cases. +5. **Multi-item Twitch drop** (one drop id repeated across benefit edges): exactly one state row and one claim. Pinned in Task 5 (`TestDropProgress_DedupesMultiItemDrop`) and Task 7 (`TestRun_MultiItemDropOneRow`). + +--- + +### Task 1: Live verification probes (gate) + +Answers spec §9 before anything builds on it. Probes are committed behind the `live` build tag so they never run in CI. + +**Files:** +- Create: `internal/store/live_session_test.go` +- Create: `internal/platform/twitch/live_probe_test.go` +- Create: `internal/platform/kick/live_probe_test.go` +- Modify: `docs/superpowers/specs/2026-09-27-pipeline-v2-design.md` (§9 results) + +**Interfaces:** +- Consumes: `store.Open`, `store.NewCryptor`, `store.NewSessionStore`, `(*SessionStore).Get`; twitch `New()`, `(*discovery).resolveCurrentLogin`, `(*client).gql`, `OpDropCampaignDetails`; kick `New(...)`, `b.api.d.do`, `dropsBase`. +- Produces: nothing used by later tasks except the recorded answers. + +- [ ] **Step 1: Session dump helper** + +```go +//go:build live + +package store + +import ( + "context" + "encoding/json" + "os" + "testing" + + "github.com/stretchr/testify/require" + + "github.com/aalejandrofer/grubdrops/internal/store/gen" +) + +// TestLive_DumpSession prints one account's decrypted session as JSON. +// Run against a COPY of the DB file: Open runs migrations. Output holds +// live tokens; do not paste it anywhere. +// +// GRUB_DB_PATH=/tmp/miner-copy.db GRUB_MASTER_KEY=... GRUB_LIVE_ACCOUNT=acc_x \ +// go test -tags live -run TestLive_DumpSession -v ./internal/store/ +func TestLive_DumpSession(t *testing.T) { + path, key, acct := os.Getenv("GRUB_DB_PATH"), os.Getenv("GRUB_MASTER_KEY"), os.Getenv("GRUB_LIVE_ACCOUNT") + if path == "" || key == "" || acct == "" { + t.Skip("set GRUB_DB_PATH, GRUB_MASTER_KEY, GRUB_LIVE_ACCOUNT") + } + ctx := context.Background() + db, err := Open(ctx, path) + require.NoError(t, err) + defer db.Close() + c, err := NewCryptor(key) + require.NoError(t, err) + s, ok, err := NewSessionStore(db, gen.New(db), c).Get(ctx, acct) + require.NoError(t, err) + require.True(t, ok, "no session for account") + b, err := json.Marshal(s) + require.NoError(t, err) + t.Logf("%s", b) +} +``` + +- [ ] **Step 2: Twitch probe** + +```go +//go:build live + +package twitch + +import ( + "context" + "encoding/json" + "os" + "testing" + + "github.com/stretchr/testify/require" + + "github.com/aalejandrofer/grubdrops/internal/platform" +) + +// TestLive_DetailsSelf dumps raw DropCampaignDetails for a campaign the +// account is enrolled in (ideally one fully claimed), to confirm +// timeBasedDrops[].self carries isClaimed + currentMinutesWatched. +// +// GRUB_LIVE_TWITCH_TOKEN=... GRUB_LIVE_CAMPAIGN_ID=... \ +// go test -tags live -run TestLive_DetailsSelf -v ./internal/platform/twitch/ +func TestLive_DetailsSelf(t *testing.T) { + tok, camp := os.Getenv("GRUB_LIVE_TWITCH_TOKEN"), os.Getenv("GRUB_LIVE_CAMPAIGN_ID") + if tok == "" || camp == "" { + t.Skip("set GRUB_LIVE_TWITCH_TOKEN, GRUB_LIVE_CAMPAIGN_ID") + } + ctx := context.Background() + b := New() + login, err := b.disc.resolveCurrentLogin(ctx, platform.Session{AccessToken: tok}) + require.NoError(t, err) + var raw json.RawMessage + require.NoError(t, b.c.gql(ctx, tok, OpDropCampaignDetails, + map[string]any{"dropID": camp, "channelLogin": login}, &raw)) + t.Logf("%s", raw) +} +``` + +- [ ] **Step 3: Kick probe** + +```go +//go:build live + +package kick + +import ( + "context" + "encoding/json" + "net/http" + "os" + "testing" + "time" + + "github.com/stretchr/testify/require" + + "github.com/aalejandrofer/grubdrops/internal/platform" +) + +// TestLive_ProgressRaw dumps raw /drops/progress for a Kick account that has +// at least one CLAIMED reward, to learn whether claimed rewards stay listed. +// +// GRUB_LIVE_KICK_SESSION_JSON='{"cookies":{...}}' \ +// go test -tags live -run TestLive_ProgressRaw -v ./internal/platform/kick/ +func TestLive_ProgressRaw(t *testing.T) { + js := os.Getenv("GRUB_LIVE_KICK_SESSION_JSON") + if js == "" { + t.Skip("set GRUB_LIVE_KICK_SESSION_JSON") + } + var s platform.Session + require.NoError(t, json.Unmarshal([]byte(js), &s)) + b := New(nil, nil, "grubdrops-browser-{slug}", 9090, 10*time.Minute) + body, status, err := b.api.d.do(context.Background(), s, http.MethodGet, dropsBase+"/api/v1/drops/progress", nil) + require.NoError(t, err) + t.Logf("status=%d body=%s", status, body) +} +``` + +- [ ] **Step 4: Confirm they compile and skip without env** + +Run: `go vet -tags live ./internal/store/ ./internal/platform/twitch/ ./internal/platform/kick/ && go test -tags live -run 'TestLive_' ./internal/store/ ./internal/platform/twitch/ ./internal/platform/kick/ -v` +Expected: three `--- SKIP` lines, exit 0. If `b.api.d` does not compile (field names differ), read `internal/platform/kick/api.go` for the doer field and adjust. + +- [ ] **Step 5: HUMAN step: run against real accounts** + +Ask the user to run Steps 1 to 3 with real values (DB copy from prod host, one Twitch account with a completed claimed campaign, one Kick account with a claimed reward). Do not run them yourself with prod secrets unless the user hands them over. + +- [ ] **Step 6: Record answers in spec §9** + +Replace §9 body with the two answers, for example: + +```markdown +## 9. Verification results (2026-09-XX) + +1. Twitch `DropCampaignDetails` for an enrolled completed campaign: `self` present = YES/NO. + Sample: `{"id":"...","self":{"isClaimed":true,"currentMinutesWatched":120,...}}` +2. Kick `/drops/progress` keeps claimed rewards listed: YES/NO. +``` + +If Twitch answer is NO: stop and tell the user; §4.2 needs revisiting before Task 5. If Kick answer is NO: no code change; the absent-reward rule in Task 2 already covers it. + +- [ ] **Step 7: Commit** + +```bash +gofmt -w internal/store internal/platform +git add internal/store/live_session_test.go internal/platform/twitch/live_probe_test.go internal/platform/kick/live_probe_test.go docs/superpowers/specs/2026-09-27-pipeline-v2-design.md +git commit -m "test(pipeline-v2): live probes for Twitch details self and Kick claimed retention + +Co-Authored-By: Claude Opus 5.5 (1M context) " +``` + +--- + +### Task 2: Drop state model (`dropstate`) + +**Files:** +- Create: `internal/pipeline/dropstate/dropstate.go` +- Test: `internal/pipeline/dropstate/dropstate_test.go` + +**Interfaces:** +- Consumes: nothing (stdlib `time` only). +- Produces: + - `type Status string` with `Eligible, Accruing, Claimable, Claimed, Blocked` + - `type Reason string` with `NoReason, NeedsLink, ClaimFailed, SubOnly, NoChannels, Expired, NotEnrolled, UserSkip` + - `type Source string` with `FromPlatform, FromUser` + - `type Row struct { AccountID, DropID, CampaignID, Platform string; Status Status; Reason Reason; Minutes, Required int; Source Source; FailCount int; RetryAfter, SyncedAt, UpdatedAt time.Time }`, `func (r Row) IsZero() bool` + - `type Observation struct { Known, Unmineable, Claimed bool; Minutes, Required int }` + - `func Apply(prev Row, obs Observation, now time.Time) Row` + - `func ClaimOK(prev Row, now time.Time) Row`, `func ClaimNeedsLink(prev Row, now time.Time) Row`, `func ClaimFailedAttempt(prev Row, now time.Time) Row` + - `func MarkCollected(prev Row, now time.Time) Row`, `func Skip(prev Row, now time.Time) Row`, `func Retry(prev Row, now time.Time) Row`, `func Expire(prev Row, now time.Time) Row`, `func BlockLink(prev Row, now time.Time) Row` + - `func Mineable(r Row) bool`, `func ReadyToClaim(r Row, now time.Time) bool` + - consts `RetryNeedsLink=24h, RetryClaimFailed=6h, RetryNotEnrolled=1h, RetrySubOnly=24h, RetryExpired=24h, MaxClaimFailures=5` + +- [ ] **Step 1: Write the failing tests** + +```go +package dropstate + +import ( + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +var t0 = time.Unix(1_000_000, 0) + +func base() Row { + return Row{AccountID: "a", DropID: "d", CampaignID: "c", Platform: "twitch", Source: FromPlatform} +} + +func row(s Status, r Reason, min, req int) Row { + x := base() + x.Status, x.Reason, x.Minutes, x.Required = s, r, min, req + return x +} + +func TestApply(t *testing.T) { + userClaimed := row(Claimed, NoReason, 0, 60) + userClaimed.Source = FromUser + notEnrolled := row(Blocked, NotEnrolled, 10, 60) + notEnrolled.RetryAfter = t0.Add(time.Hour) + skipped := row(Blocked, UserSkip, 0, 60) + skipped.Source = FromUser + needsLink := row(Blocked, NeedsLink, 60, 60) + needsLink.RetryAfter = t0.Add(RetryNeedsLink) + + cases := []struct { + name string + prev Row + obs Observation + at time.Time + want Status + reason Reason + src Source + }{ + {"new drop, platform silent -> eligible", base(), Observation{Required: 60}, t0, Eligible, NoReason, FromPlatform}, + {"new drop, platform silent, zero required -> sub_only", base(), Observation{}, t0, Blocked, SubOnly, FromPlatform}, + {"known progress -> accruing", row(Eligible, NoReason, 0, 60), Observation{Known: true, Minutes: 10, Required: 60}, t0, Accruing, NoReason, FromPlatform}, + {"known full -> claimable", row(Accruing, NoReason, 50, 60), Observation{Known: true, Minutes: 60, Required: 60}, t0, Claimable, NoReason, FromPlatform}, + {"known claimed -> claimed", row(Accruing, NoReason, 10, 60), Observation{Known: true, Claimed: true, Minutes: 60, Required: 60}, t0, Claimed, NoReason, FromPlatform}, + {"platform claimed never demoted", row(Claimed, NoReason, 60, 60), Observation{Known: true, Minutes: 0, Required: 60}, t0, Claimed, NoReason, FromPlatform}, + {"user mark kept while platform silent", userClaimed, Observation{Required: 60}, t0, Claimed, NoReason, FromUser}, + {"user mark overwritten by definite platform answer", userClaimed, Observation{Known: true, Minutes: 5, Required: 60}, t0, Accruing, NoReason, FromPlatform}, + {"accruing then platform silent -> not_enrolled", row(Accruing, NoReason, 10, 60), Observation{Required: 60}, t0, Blocked, NotEnrolled, FromPlatform}, + {"not_enrolled held inside window", notEnrolled, Observation{Required: 60}, t0, Blocked, NotEnrolled, FromPlatform}, + {"not_enrolled re-derived after window", notEnrolled, Observation{Required: 60}, t0.Add(2 * time.Hour), Accruing, NoReason, FromPlatform}, + {"unmineable placeholder -> not_enrolled", base(), Observation{Known: true, Unmineable: true}, t0, Blocked, NotEnrolled, FromPlatform}, + {"user skip survives progress", skipped, Observation{Known: true, Minutes: 30, Required: 60}, t0, Blocked, UserSkip, FromUser}, + {"claimed beats user skip", skipped, Observation{Known: true, Claimed: true, Minutes: 60, Required: 60}, t0, Claimed, NoReason, FromPlatform}, + {"needs_link held inside window", needsLink, Observation{Known: true, Minutes: 60, Required: 60}, t0, Blocked, NeedsLink, FromPlatform}, + {"needs_link re-derived after window", needsLink, Observation{Known: true, Minutes: 60, Required: 60}, t0.Add(25 * time.Hour), Claimable, NoReason, FromPlatform}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + got := Apply(tc.prev, tc.obs, tc.at) + assert.Equal(t, tc.want, got.Status) + assert.Equal(t, tc.reason, got.Reason) + assert.Equal(t, tc.src, got.Source) + assert.Equal(t, "d", got.DropID, "identity preserved") + }) + } +} + +func TestApply_PlatformClaimedKeepsHighestMinutes(t *testing.T) { + got := Apply(row(Claimed, NoReason, 60, 60), Observation{Known: true, Minutes: 0, Required: 60}, t0) + assert.Equal(t, 60, got.Minutes) +} + +func TestApply_KeepsClaimBackoff(t *testing.T) { + r := ClaimFailedAttempt(row(Claimable, NoReason, 60, 60), t0) + got := Apply(r, Observation{Known: true, Minutes: 60, Required: 60}, t0.Add(10*time.Second)) + assert.Equal(t, Claimable, got.Status) + assert.Equal(t, r.RetryAfter, got.RetryAfter, "sync must not reset claim backoff") + assert.Equal(t, 1, got.FailCount) +} + +func TestClaimFailedAttempt_Ladder(t *testing.T) { + r := row(Claimable, NoReason, 60, 60) + for _, wait := range []time.Duration{time.Minute, 5 * time.Minute, 30 * time.Minute, 30 * time.Minute} { + r = ClaimFailedAttempt(r, t0) + require.Equal(t, Claimable, r.Status) + require.Equal(t, t0.Add(wait), r.RetryAfter) + require.False(t, ReadyToClaim(r, t0)) + require.True(t, ReadyToClaim(r, t0.Add(wait))) + } + r = ClaimFailedAttempt(r, t0) + assert.Equal(t, Blocked, r.Status) + assert.Equal(t, ClaimFailed, r.Reason) + assert.Equal(t, t0.Add(RetryClaimFailed), r.RetryAfter) +} + +func TestClaimOutcomes(t *testing.T) { + ok := ClaimOK(row(Claimable, NoReason, 60, 60), t0) + assert.Equal(t, Claimed, ok.Status) + assert.Equal(t, FromPlatform, ok.Source) + link := ClaimNeedsLink(row(Claimable, NoReason, 60, 60), t0) + assert.Equal(t, Blocked, link.Status) + assert.Equal(t, NeedsLink, link.Reason) + assert.Equal(t, t0.Add(RetryNeedsLink), link.RetryAfter) +} + +func TestUserActions(t *testing.T) { + m := MarkCollected(row(Accruing, NoReason, 10, 60), t0) + assert.Equal(t, Claimed, m.Status) + assert.Equal(t, FromUser, m.Source) + + s := Skip(row(Accruing, NoReason, 10, 60), t0) + assert.Equal(t, Blocked, s.Status) + assert.Equal(t, UserSkip, s.Reason) + assert.True(t, s.RetryAfter.IsZero(), "user_skip never auto-expires") + + failed := row(Blocked, ClaimFailed, 60, 60) + failed.FailCount = 5 + r := Retry(failed, t0) + assert.Equal(t, Claimable, r.Status) + assert.Equal(t, 0, r.FailCount) +} + +func TestExpireAndBlockLink_LeaveClaimedAlone(t *testing.T) { + c := row(Claimed, NoReason, 60, 60) + assert.Equal(t, Claimed, Expire(c, t0).Status) + assert.Equal(t, Claimed, BlockLink(c, t0).Status) + assert.Equal(t, Expired, Expire(row(Accruing, NoReason, 10, 60), t0).Reason) + assert.Equal(t, NeedsLink, BlockLink(row(Eligible, NoReason, 0, 60), t0).Reason) + sk := row(Blocked, UserSkip, 0, 60) + assert.Equal(t, UserSkip, BlockLink(sk, t0).Reason) +} + +func TestMineable(t *testing.T) { + assert.True(t, Mineable(row(Eligible, NoReason, 0, 60))) + assert.True(t, Mineable(row(Accruing, NoReason, 1, 60))) + assert.False(t, Mineable(row(Claimable, NoReason, 60, 60))) + assert.False(t, Mineable(row(Blocked, NotEnrolled, 0, 60))) + assert.False(t, Mineable(Row{})) +} +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `go test ./internal/pipeline/dropstate/ -v` +Expected: FAIL, `undefined: Row` (package has no source yet). + +- [ ] **Step 3: Implement** + +```go +// Package dropstate is the per-(account, drop) state model for pipeline v2. +// Every transition is a pure function of (previous row, input, now) so the +// rules are table-testable and the account loop is the only writer. +package dropstate + +import "time" + +type Status string + +const ( + Eligible Status = "eligible" + Accruing Status = "accruing" + Claimable Status = "claimable" + Claimed Status = "claimed" + Blocked Status = "blocked" +) + +type Reason string + +const ( + NoReason Reason = "" + NeedsLink Reason = "needs_link" + ClaimFailed Reason = "claim_failed" + SubOnly Reason = "sub_only" + NoChannels Reason = "no_channels" + Expired Reason = "expired" + NotEnrolled Reason = "not_enrolled" + UserSkip Reason = "user_skip" +) + +type Source string + +const ( + FromPlatform Source = "platform" + FromUser Source = "user" +) + +// Retry windows for blocked rows. user_skip never auto-expires. +const ( + RetryNeedsLink = 24 * time.Hour + RetryClaimFailed = 6 * time.Hour + RetryNotEnrolled = time.Hour + RetrySubOnly = 24 * time.Hour + RetryExpired = 24 * time.Hour + MaxClaimFailures = 5 +) + +var claimBackoff = []time.Duration{time.Minute, 5 * time.Minute, 30 * time.Minute} + +type Row struct { + AccountID string + DropID string + CampaignID string + Platform string + Status Status + Reason Reason + Minutes int + Required int + Source Source + FailCount int + RetryAfter time.Time + SyncedAt time.Time + UpdatedAt time.Time +} + +// IsZero reports a row that has never been stored or derived. +func (r Row) IsZero() bool { return r.Status == "" } + +// Observation is what the platform said about one drop in one sync. +type Observation struct { + // Known is true when the platform returned a definite record for the + // drop. False means it said nothing, which proves nothing. + Known bool + // Unmineable marks drops the platform cannot track (scrape placeholders). + Unmineable bool + Claimed bool + Minutes int + Required int +} + +func derive(minutes, required int) (Status, Reason) { + switch { + case required <= 0: + return Blocked, SubOnly + case minutes >= required: + return Claimable, NoReason + case minutes > 0: + return Accruing, NoReason + default: + return Eligible, NoReason + } +} + +func retryFor(reason Reason, now time.Time) time.Time { + switch reason { + case NeedsLink: + return now.Add(RetryNeedsLink) + case ClaimFailed: + return now.Add(RetryClaimFailed) + case NotEnrolled, NoChannels: + return now.Add(RetryNotEnrolled) + case SubOnly: + return now.Add(RetrySubOnly) + case Expired: + return now.Add(RetryExpired) + } + return time.Time{} +} + +func (r Row) with(s Status, reason Reason, now time.Time) Row { + r.Status, r.Reason, r.UpdatedAt = s, reason, now + r.RetryAfter = retryFor(reason, now) + return r +} + +// Apply folds one platform observation into the previous row. The caller +// seeds identity fields (AccountID, DropID, CampaignID, Platform) on a new row. +func Apply(prev Row, obs Observation, now time.Time) Row { + r := prev + if obs.Known && !obs.Unmineable { + r.Minutes, r.Required, r.SyncedAt = obs.Minutes, obs.Required, now + } + // A platform-confirmed claim is final. + if prev.Status == Claimed && prev.Source == FromPlatform { + if r.Minutes < prev.Minutes { + r.Minutes = prev.Minutes + } + return r + } + if !obs.Known { + switch { + case prev.IsZero(): + r.Required = obs.Required + r.Source = FromPlatform + st, rs := derive(0, obs.Required) + return r.with(st, rs, now) + case (prev.Status == Accruing || prev.Status == Claimable) && prev.Source == FromPlatform: + return r.with(Blocked, NotEnrolled, now) + case prev.Status == Blocked && prev.Reason == NotEnrolled && !now.Before(prev.RetryAfter): + st, rs := derive(prev.Minutes, prev.Required) + return r.with(st, rs, now) + } + return r + } + if obs.Unmineable { + if prev.Status == Claimed { + return r + } + r.Source = FromPlatform + return r.with(Blocked, NotEnrolled, now) + } + if obs.Claimed { + r.Source, r.FailCount = FromPlatform, 0 + return r.with(Claimed, NoReason, now) + } + if prev.Status == Blocked && prev.Reason == UserSkip { + return r + } + if prev.Status == Blocked && (prev.Reason == NeedsLink || prev.Reason == ClaimFailed) && now.Before(prev.RetryAfter) { + return r + } + st, rs := derive(r.Minutes, r.Required) + if st == Claimable && prev.Status == Claimable && now.Before(prev.RetryAfter) { + return r // keep claim backoff + } + if prev.Status == Blocked { + r.FailCount = 0 + } + r.Source = FromPlatform + return r.with(st, rs, now) +} + +// ClaimOK records a successful or already-claimed claim. +func ClaimOK(prev Row, now time.Time) Row { + r := prev.with(Claimed, NoReason, now) + r.Source, r.FailCount = FromPlatform, 0 + return r +} + +// ClaimNeedsLink blocks a drop whose claim needs an external account link. +func ClaimNeedsLink(prev Row, now time.Time) Row { + r := prev.with(Blocked, NeedsLink, now) + r.Source = FromPlatform + return r +} + +// ClaimFailedAttempt counts a failed claim: backoff 1m, 5m, 30m, then +// blocked:claim_failed after MaxClaimFailures. +func ClaimFailedAttempt(prev Row, now time.Time) Row { + r := prev + r.FailCount++ + r.UpdatedAt = now + if r.FailCount >= MaxClaimFailures { + return r.with(Blocked, ClaimFailed, now) + } + i := r.FailCount - 1 + if i >= len(claimBackoff) { + i = len(claimBackoff) - 1 + } + r.RetryAfter = now.Add(claimBackoff[i]) + return r +} + +// MarkCollected is the user asserting the drop is claimed. +func MarkCollected(prev Row, now time.Time) Row { + r := prev.with(Claimed, NoReason, now) + r.Source = FromUser + return r +} + +// Skip is the user excluding a drop from mining until they retry it. +func Skip(prev Row, now time.Time) Row { + r := prev.with(Blocked, UserSkip, now) + r.Source = FromUser + return r +} + +// Retry clears any block and re-derives from the last known minutes. +func Retry(prev Row, now time.Time) Row { + st, rs := derive(prev.Minutes, prev.Required) + r := prev.with(st, rs, now) + r.FailCount, r.Source = 0, FromPlatform + return r +} + +// Expire blocks an unclaimed drop whose campaign has ended. +func Expire(prev Row, now time.Time) Row { + if prev.Status == Claimed { + return prev + } + return prev.with(Blocked, Expired, now) +} + +// BlockLink blocks a drop whose campaign needs an unlinked external account. +func BlockLink(prev Row, now time.Time) Row { + if prev.Status == Claimed || (prev.Status == Blocked && prev.Reason == UserSkip) { + return prev + } + return prev.with(Blocked, NeedsLink, now) +} + +// Mineable reports whether watching can still advance the drop. +func Mineable(r Row) bool { return r.Status == Eligible || r.Status == Accruing } + +// ReadyToClaim reports a claimable row whose backoff has elapsed. +func ReadyToClaim(r Row, now time.Time) bool { + return r.Status == Claimable && !now.Before(r.RetryAfter) +} +``` + +- [ ] **Step 4: Run tests** + +Run: `go test ./internal/pipeline/dropstate/ -v` +Expected: PASS, all cases. + +- [ ] **Step 5: Revert-proof check** + +Temporarily change the `prev.Status == Claimed && prev.Source == FromPlatform` early return to `false &&`. Run tests: `platform claimed never demoted` must FAIL. Restore. + +- [ ] **Step 6: Commit** + +```bash +gofmt -w internal/pipeline +git add internal/pipeline/dropstate +git commit -m "feat(pipeline-v2): per-drop state model with pure transitions + +Co-Authored-By: Claude Opus 5.5 (1M context) " +``` + +--- + +### Task 3: `drop_state` + streamer priority storage + +**Files:** +- Create: `internal/store/migrations/0016_pipeline_v2.sql` +- Create: `internal/store/queries/drop_state.sql` +- Create: `internal/store/queries/streamer_priority.sql` +- Regenerate: `internal/store/gen/*` (sqlc) +- Create: `internal/store/dropstate_store.go` +- Test: `internal/store/dropstate_store_test.go` + +**Interfaces:** +- Consumes: `dropstate.Row` and constants (Task 2). +- Produces: + - `type DropStateStore struct{ Q *gen.Queries }`, `func NewDropStateStore(q *gen.Queries) *DropStateStore` + - `func (s *DropStateStore) List(ctx context.Context, accountID string) ([]dropstate.Row, error)` + - `func (s *DropStateStore) Upsert(ctx context.Context, r dropstate.Row) error` + - `func (s *DropStateStore) StreamerPriority(ctx context.Context, accountID string) ([]string, error)` + - gen: `ListDropStates`, `UpsertDropState`, `ListStreamerPriority`, models `gen.DropState`, `gen.UpsertDropStateParams` + +- [ ] **Step 1: Migration** + +```sql +-- +goose Up +-- +goose StatementBegin +-- Pipeline v2: one authoritative mining state row per account and drop. +-- Timestamps are unix seconds, 0 means unset. +CREATE TABLE drop_state ( + account_id TEXT NOT NULL REFERENCES accounts(id) ON DELETE CASCADE, + drop_id TEXT NOT NULL, + campaign_id TEXT NOT NULL, + platform TEXT NOT NULL, + status TEXT NOT NULL, + block_reason TEXT NOT NULL DEFAULT '', + minutes INTEGER NOT NULL DEFAULT 0, + required INTEGER NOT NULL DEFAULT 0, + source TEXT NOT NULL DEFAULT 'platform', + fail_count INTEGER NOT NULL DEFAULT 0, + retry_after INTEGER NOT NULL DEFAULT 0, + synced_at INTEGER NOT NULL DEFAULT 0, + updated_at INTEGER NOT NULL, + PRIMARY KEY (account_id, drop_id) +); +CREATE INDEX idx_drop_state_status ON drop_state(account_id, status); + +-- Ordered per-account priority streamers. Seeded from account_channels, +-- the null-game channel whitelist, which is the same concept. +CREATE TABLE account_streamer_priority ( + account_id TEXT NOT NULL REFERENCES accounts(id) ON DELETE CASCADE, + platform TEXT NOT NULL, + login TEXT NOT NULL, + rank INTEGER NOT NULL, + PRIMARY KEY (account_id, login) +); +CREATE INDEX idx_account_streamer_priority_acct ON account_streamer_priority(account_id, rank); + +INSERT INTO account_streamer_priority (account_id, platform, login, rank) +SELECT ac.account_id, a.platform, ac.channel, ac.rank +FROM account_channels ac +JOIN accounts a ON a.id = ac.account_id; +-- +goose StatementEnd + +-- +goose Down +-- +goose StatementBegin +DROP INDEX IF EXISTS idx_account_streamer_priority_acct; +DROP TABLE IF EXISTS account_streamer_priority; +DROP INDEX IF EXISTS idx_drop_state_status; +DROP TABLE IF EXISTS drop_state; +-- +goose StatementEnd +``` + +- [ ] **Step 2: Queries** + +`internal/store/queries/drop_state.sql`: + +```sql +-- name: ListDropStates :many +SELECT * FROM drop_state WHERE account_id = ? ORDER BY campaign_id, drop_id; + +-- name: UpsertDropState :exec +INSERT INTO drop_state ( + account_id, drop_id, campaign_id, platform, status, block_reason, + minutes, required, source, fail_count, retry_after, synced_at, updated_at +) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) +ON CONFLICT(account_id, drop_id) DO UPDATE SET + campaign_id = excluded.campaign_id, + platform = excluded.platform, + status = excluded.status, + block_reason = excluded.block_reason, + minutes = excluded.minutes, + required = excluded.required, + source = excluded.source, + fail_count = excluded.fail_count, + retry_after = excluded.retry_after, + synced_at = excluded.synced_at, + updated_at = excluded.updated_at; +``` + +`internal/store/queries/streamer_priority.sql`: + +```sql +-- name: ListStreamerPriority :many +SELECT login, rank FROM account_streamer_priority +WHERE account_id = ? +ORDER BY rank ASC, login ASC; +``` + +- [ ] **Step 3: Generate** + +Run: `cd internal/store && sqlc generate && cd ../.. && go build ./...` +Expected: new files `internal/store/gen/drop_state.sql.go`, `streamer_priority.sql.go`; `models.go` gains `DropState` and `AccountStreamerPriority`; build OK. Open `gen/drop_state.sql.go` and confirm `UpsertDropStateParams` field names match Step 5 below (`AccountID, DropID, CampaignID, Platform, Status, BlockReason, Minutes, Required, Source, FailCount, RetryAfter, SyncedAt, UpdatedAt`, integers as `int64`). If sqlc named anything differently, use its names in Step 5. + +- [ ] **Step 4: Failing store test** + +```go +package store + +import ( + "context" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/aalejandrofer/grubdrops/internal/pipeline/dropstate" + "github.com/aalejandrofer/grubdrops/internal/store/gen" +) + +func seedAccount(t *testing.T, q *gen.Queries, id, platform string) { + t.Helper() + now := time.Now().Unix() + _, err := q.CreateAccount(context.Background(), gen.CreateAccountParams{ + ID: id, Platform: platform, DisplayName: id, + Status: "idle", FingerprintJson: "{}", Enabled: 1, + CreatedAt: now, UpdatedAt: now, + }) + require.NoError(t, err) +} + +func TestDropStateStore_RoundTrip(t *testing.T) { + db := openTest(t) + q := gen.New(db) + ctx := context.Background() + seedAccount(t, q, "acc-1", "twitch") + s := NewDropStateStore(q) + + at := time.Unix(1_700_000_000, 0) + in := dropstate.Row{ + AccountID: "acc-1", DropID: "d1", CampaignID: "c1", Platform: "twitch", + Status: dropstate.Blocked, Reason: dropstate.ClaimFailed, + Minutes: 60, Required: 60, Source: dropstate.FromPlatform, FailCount: 5, + RetryAfter: at.Add(time.Hour), SyncedAt: at, UpdatedAt: at, + } + require.NoError(t, s.Upsert(ctx, in)) + got, err := s.List(ctx, "acc-1") + require.NoError(t, err) + require.Len(t, got, 1) + assert.Equal(t, in, got[0]) + + in.Status, in.Reason, in.RetryAfter = dropstate.Claimed, dropstate.NoReason, time.Time{} + require.NoError(t, s.Upsert(ctx, in)) + got, err = s.List(ctx, "acc-1") + require.NoError(t, err) + require.Len(t, got, 1, "upsert must not duplicate") + assert.Equal(t, dropstate.Claimed, got[0].Status) + assert.True(t, got[0].RetryAfter.IsZero(), "0 maps back to zero time") +} + +func TestDropStateStore_StreamerPrioritySeededFromAccountChannels(t *testing.T) { + db := openTest(t) + q := gen.New(db) + ctx := context.Background() + seedAccount(t, q, "acc-1", "kick") + s := NewDropStateStore(q) + + // Migration already ran; insert straight into the new table like the + // seed does, then read ordered by rank. + _, err := db.ExecContext(ctx, `INSERT INTO account_streamer_priority (account_id, platform, login, rank) VALUES + ('acc-1','kick','zeta',1), ('acc-1','kick','alpha',0)`) + require.NoError(t, err) + got, err := s.StreamerPriority(ctx, "acc-1") + require.NoError(t, err) + assert.Equal(t, []string{"alpha", "zeta"}, got) +} +``` + +Run: `go test ./internal/store/ -run DropStateStore -v` +Expected: FAIL, `undefined: NewDropStateStore`. + +- [ ] **Step 5: Implement store** + +```go +package store + +import ( + "context" + "time" + + "github.com/aalejandrofer/grubdrops/internal/pipeline/dropstate" + "github.com/aalejandrofer/grubdrops/internal/store/gen" +) + +// DropStateStore persists pipeline v2 drop_state rows and reads the +// per-account streamer priority list. +type DropStateStore struct { + Q *gen.Queries +} + +func NewDropStateStore(q *gen.Queries) *DropStateStore { return &DropStateStore{Q: q} } + +func dsUnix(t time.Time) int64 { + if t.IsZero() { + return 0 + } + return t.Unix() +} + +func dsTime(v int64) time.Time { + if v == 0 { + return time.Time{} + } + return time.Unix(v, 0) +} + +func (s *DropStateStore) List(ctx context.Context, accountID string) ([]dropstate.Row, error) { + rows, err := s.Q.ListDropStates(ctx, accountID) + if err != nil { + return nil, err + } + out := make([]dropstate.Row, 0, len(rows)) + for _, r := range rows { + out = append(out, dropstate.Row{ + AccountID: r.AccountID, + DropID: r.DropID, + CampaignID: r.CampaignID, + Platform: r.Platform, + Status: dropstate.Status(r.Status), + Reason: dropstate.Reason(r.BlockReason), + Minutes: int(r.Minutes), + Required: int(r.Required), + Source: dropstate.Source(r.Source), + FailCount: int(r.FailCount), + RetryAfter: dsTime(r.RetryAfter), + SyncedAt: dsTime(r.SyncedAt), + UpdatedAt: dsTime(r.UpdatedAt), + }) + } + return out, nil +} + +func (s *DropStateStore) Upsert(ctx context.Context, r dropstate.Row) error { + return s.Q.UpsertDropState(ctx, gen.UpsertDropStateParams{ + AccountID: r.AccountID, + DropID: r.DropID, + CampaignID: r.CampaignID, + Platform: r.Platform, + Status: string(r.Status), + BlockReason: string(r.Reason), + Minutes: int64(r.Minutes), + Required: int64(r.Required), + Source: string(r.Source), + FailCount: int64(r.FailCount), + RetryAfter: dsUnix(r.RetryAfter), + SyncedAt: dsUnix(r.SyncedAt), + UpdatedAt: dsUnix(r.UpdatedAt), + }) +} + +func (s *DropStateStore) StreamerPriority(ctx context.Context, accountID string) ([]string, error) { + rows, err := s.Q.ListStreamerPriority(ctx, accountID) + if err != nil { + return nil, err + } + out := make([]string, 0, len(rows)) + for _, r := range rows { + out = append(out, r.Login) + } + return out, nil +} +``` + +- [ ] **Step 6: Migration seed test** + +Append to `dropstate_store_test.go`. It proves the `INSERT ... SELECT` seed ran by re-running goose Down/Up is overkill; instead assert on an account_channels row inserted BEFORE the migration is impossible here, so test the seed SQL directly: + +```go +func TestMigration0016_SeedCopiesAccountChannels(t *testing.T) { + db := openTest(t) + q := gen.New(db) + ctx := context.Background() + seedAccount(t, q, "acc-1", "kick") + require.NoError(t, q.AddAccountChannel(ctx, gen.AddAccountChannelParams{AccountID: "acc-1", Channel: "oilrats", Rank: 0})) + // Re-run the seed statement from 0016 exactly. + _, err := db.ExecContext(ctx, `INSERT INTO account_streamer_priority (account_id, platform, login, rank) + SELECT ac.account_id, a.platform, ac.channel, ac.rank FROM account_channels ac JOIN accounts a ON a.id = ac.account_id`) + require.NoError(t, err) + got, err := NewDropStateStore(q).StreamerPriority(ctx, "acc-1") + require.NoError(t, err) + assert.Equal(t, []string{"oilrats"}, got) +} +``` + +- [ ] **Step 7: Run all store tests** + +Run: `go test ./internal/store/ -v 2>&1 | tail -20` +Expected: PASS (existing tests unaffected by the new migration). + +- [ ] **Step 8: Commit** + +```bash +gofmt -w internal/store +git add internal/store +git commit -m "feat(pipeline-v2): drop_state and streamer priority tables + +Co-Authored-By: Claude Opus 5.5 (1M context) " +``` + +--- + +### Task 4: One-shot backfill from v1 state + +**Files:** +- Modify: `internal/store/queries/drop_state.sql` (append two queries) +- Regenerate: `internal/store/gen/*` +- Create: `internal/store/dropstate_backfill.go` +- Test: `internal/store/dropstate_backfill_test.go` + +**Interfaces:** +- Consumes: `DropStateStore` (Task 3), `SkipOverridePrefix`, `CollectOverridePrefix` (existing), gen `ListKVByPrefix(ctx, sql.NullString) ([]gen.Kv, error)`, `GetSettingString(ctx, key) ([]byte, error)`, `UpsertSettingString(ctx, gen.UpsertSettingStringParams{Key string; Value []byte})`. +- Produces: `func BackfillDropState(ctx context.Context, q *gen.Queries, now time.Time) (int, error)`, const `DropStateBackfilledKey = "drop_state_backfilled"`. + +- [ ] **Step 1: Queries** (append to `drop_state.sql`) + +```sql +-- name: ListClaimsForBackfill :many +SELECT c.account_id, c.benefit_id, b.campaign_id, a.platform +FROM claims c +JOIN benefits b ON b.id = c.benefit_id +JOIN accounts a ON a.id = c.account_id; + +-- name: GetBenefitCampaign :one +SELECT b.campaign_id, cp.platform +FROM benefits b +JOIN campaigns cp ON cp.id = b.campaign_id +WHERE b.id = ?; +``` + +Run: `cd internal/store && sqlc generate && cd ../.. && go build ./...` +Expected: `ListClaimsForBackfillRow{AccountID, BenefitID, CampaignID, Platform}` and `GetBenefitCampaignRow{CampaignID, Platform}` generated. + +- [ ] **Step 2: Failing test** + +```go +package store + +import ( + "context" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/aalejandrofer/grubdrops/internal/pipeline/dropstate" + "github.com/aalejandrofer/grubdrops/internal/platform" + "github.com/aalejandrofer/grubdrops/internal/store/gen" +) + +func TestBackfillDropState(t *testing.T) { + db := openTest(t) + q := gen.New(db) + ctx := context.Background() + seedAccount(t, q, "acc-1", "twitch") + require.NoError(t, NewCampaignPersister(q).PersistCampaigns(ctx, []platform.Campaign{{ + ID: "c1", Platform: "twitch", Game: "G", Name: "C", Status: "active", + Benefits: []platform.DropBenefit{ + {ID: "claimed1", CampaignID: "c1", Name: "A", RequiredMinutes: 60}, + {ID: "marked1", CampaignID: "c1", Name: "B", RequiredMinutes: 60}, + {ID: "skipped1", CampaignID: "c1", Name: "C", RequiredMinutes: 60}, + {ID: "both1", CampaignID: "c1", Name: "D", RequiredMinutes: 60}, + }, + }})) + rec := NewClaimRecorder(q) + require.NoError(t, rec.RecordClaim(ctx, "acc-1", platform.DropBenefit{ID: "claimed1"})) + require.NoError(t, rec.RecordClaim(ctx, "acc-1", platform.DropBenefit{ID: "both1"})) + for _, k := range []string{ + CollectOverridePrefix + "marked1:acc-1", + SkipOverridePrefix + "skipped1:acc-1", + SkipOverridePrefix + "both1:acc-1", // claim history must win over a skip + SkipOverridePrefix + "gone:acc-1", // benefit row missing: ignored + } { + require.NoError(t, q.UpsertSettingString(ctx, gen.UpsertSettingStringParams{Key: k, Value: []byte("1")})) + } + + now := time.Unix(1_700_000_000, 0) + n, err := BackfillDropState(ctx, q, now) + require.NoError(t, err) + assert.Equal(t, 4, n) + + rows, err := NewDropStateStore(q).List(ctx, "acc-1") + require.NoError(t, err) + by := map[string]dropstate.Row{} + for _, r := range rows { + by[r.DropID] = r + } + assert.Equal(t, dropstate.Claimed, by["claimed1"].Status) + assert.Equal(t, dropstate.FromPlatform, by["claimed1"].Source) + assert.Equal(t, dropstate.Claimed, by["marked1"].Status) + assert.Equal(t, dropstate.FromUser, by["marked1"].Source) + assert.Equal(t, dropstate.Blocked, by["skipped1"].Status) + assert.Equal(t, dropstate.NotEnrolled, by["skipped1"].Reason) + assert.Equal(t, now, by["skipped1"].RetryAfter, "skips get one fresh evaluation") + assert.Equal(t, dropstate.Claimed, by["both1"].Status) + assert.Equal(t, "c1", by["claimed1"].CampaignID) + + // Idempotent: second run is a no-op. + n, err = BackfillDropState(ctx, q, now) + require.NoError(t, err) + assert.Equal(t, 0, n) +} +``` + +Run: `go test ./internal/store/ -run TestBackfillDropState -v` +Expected: FAIL, `undefined: BackfillDropState`. + +- [ ] **Step 3: Implement** + +```go +package store + +import ( + "context" + "database/sql" + "fmt" + "strings" + "time" + + "github.com/aalejandrofer/grubdrops/internal/pipeline/dropstate" + "github.com/aalejandrofer/grubdrops/internal/store/gen" +) + +// DropStateBackfilledKey marks the one-shot v1 to v2 state backfill as done. +const DropStateBackfilledKey = "drop_state_backfilled" + +// BackfillDropState seeds drop_state from v1 state: ghost-skips become +// not_enrolled with an immediate retry, manual marks become user claims, and +// claim history becomes platform claims. Later sources win. Runs once; the v1 +// kv keys are left in place so rolling back to v1 is safe. +func BackfillDropState(ctx context.Context, q *gen.Queries, now time.Time) (int, error) { + if v, err := q.GetSettingString(ctx, DropStateBackfilledKey); err == nil && string(v) == "1" { + return 0, nil + } + rows := map[string]dropstate.Row{} + put := func(r dropstate.Row) { rows[r.AccountID+"|"+r.DropID] = r } + + fromKV := func(prefix string, mk func(acct, drop, campaign, plat string) dropstate.Row) error { + kvs, err := q.ListKVByPrefix(ctx, sql.NullString{String: prefix, Valid: true}) + if err != nil { + return fmt.Errorf("list %s: %w", prefix, err) + } + for _, kv := range kvs { + drop, acct, ok := splitOverrideKey(kv.Key, prefix) + if !ok { + continue + } + bc, err := q.GetBenefitCampaign(ctx, drop) + if err != nil { + continue // benefit row gone, nothing to seed + } + put(mk(acct, drop, bc.CampaignID, bc.Platform)) + } + return nil + } + if err := fromKV(SkipOverridePrefix, func(acct, drop, campaign, plat string) dropstate.Row { + return dropstate.Row{AccountID: acct, DropID: drop, CampaignID: campaign, Platform: plat, + Status: dropstate.Blocked, Reason: dropstate.NotEnrolled, Source: dropstate.FromPlatform, + RetryAfter: now, UpdatedAt: now} + }); err != nil { + return 0, err + } + if err := fromKV(CollectOverridePrefix, func(acct, drop, campaign, plat string) dropstate.Row { + return dropstate.Row{AccountID: acct, DropID: drop, CampaignID: campaign, Platform: plat, + Status: dropstate.Claimed, Source: dropstate.FromUser, UpdatedAt: now} + }); err != nil { + return 0, err + } + claims, err := q.ListClaimsForBackfill(ctx) + if err != nil { + return 0, fmt.Errorf("list claims: %w", err) + } + for _, c := range claims { + put(dropstate.Row{AccountID: c.AccountID, DropID: c.BenefitID, CampaignID: c.CampaignID, Platform: c.Platform, + Status: dropstate.Claimed, Source: dropstate.FromPlatform, UpdatedAt: now}) + } + + st := NewDropStateStore(q) + for _, r := range rows { + if err := st.Upsert(ctx, r); err != nil { + return 0, fmt.Errorf("upsert %s/%s: %w", r.AccountID, r.DropID, err) + } + } + if err := q.UpsertSettingString(ctx, gen.UpsertSettingStringParams{Key: DropStateBackfilledKey, Value: []byte("1")}); err != nil { + return 0, fmt.Errorf("mark backfilled: %w", err) + } + return len(rows), nil +} + +// splitOverrideKey parses prefix + dropID + ":" + accountID. +func splitOverrideKey(key, prefix string) (drop, acct string, ok bool) { + rest := strings.TrimPrefix(key, prefix) + i := strings.LastIndex(rest, ":") + if i <= 0 || i == len(rest)-1 { + return "", "", false + } + return rest[:i], rest[i+1:], true +} +``` + +- [ ] **Step 4: Run tests** + +Run: `go test ./internal/store/ -v 2>&1 | tail -20` +Expected: PASS. If `gen.Kv.Key` is not a `string`, adapt the field access. + +- [ ] **Step 5: Commit** + +```bash +gofmt -w internal/store +git add internal/store +git commit -m "feat(pipeline-v2): backfill drop_state from claims and v1 overrides + +Co-Authored-By: Claude Opus 5.5 (1M context) " +``` + +--- + +### Task 5: Platform capability types + Twitch adapter + +**Files:** +- Create: `internal/platform/pipeline.go` +- Modify: `internal/platform/twitch/campaigns.go` (add `Self` to details struct) +- Modify: `internal/platform/twitch/claim.go` (split out `claimStatus`, behaviour unchanged) +- Create: `internal/platform/twitch/pipeline.go` +- Test: `internal/platform/twitch/pipeline_test.go` + +**Interfaces:** +- Consumes: twitch `discovery`, `client.gql`, `OpDropCampaignDetails`, `(*discovery).inventory`, `(*channels).listEligible`, `(*watch).resolveUserID`, `claimer`. +- Produces (in `internal/platform`): + +```go +type DropProgress struct { + DropID, CampaignID string + Minutes, Required int + Claimed bool + InstanceID string + Known bool + Unmineable bool +} +type ClaimOutcome int // ClaimOK, ClaimAlready, ClaimNeedsLink, ClaimFailed; String() +type ClaimResult struct { Outcome ClaimOutcome; LinkURL, Detail string } +type DropProgressSource interface { DropProgress(ctx context.Context, s Session, camps []Campaign) ([]DropProgress, error) } +type DropClaimer interface { ClaimDrop(ctx context.Context, s Session, d DropProgress) ClaimResult } +type ChannelProber interface { ProbeChannels(ctx context.Context, s Session, c Campaign, logins []string) ([]Stream, error) } +``` +- Produces (twitch): `(*Backend).DropProgress`, `(*Backend).ClaimDrop`, `(*Backend).ProbeChannels`, `classifyClaimStatus(string) platform.ClaimResult`, `isSyntheticID(string) bool`. + +- [ ] **Step 1: Platform types** (`internal/platform/pipeline.go`) + +```go +package platform + +import "context" + +// DropProgress is one drop's per-account state as the platform reports it. +// Pipeline v2's reconciler folds these into drop_state. +type DropProgress struct { + DropID string + CampaignID string + Minutes int + Required int + Claimed bool + InstanceID string + // Known is true when the platform returned a definite record for this + // drop. False means the platform said nothing about it (Kick lists a + // reward only after it accrues), which is not evidence of anything. + Known bool + // Unmineable marks drops whose progress the platform cannot track, + // such as Twitch scrape placeholders. The reconciler blocks them. + Unmineable bool +} + +type ClaimOutcome int + +const ( + ClaimOK ClaimOutcome = iota + ClaimAlready + ClaimNeedsLink + ClaimFailed +) + +func (o ClaimOutcome) String() string { + switch o { + case ClaimOK: + return "ok" + case ClaimAlready: + return "already_claimed" + case ClaimNeedsLink: + return "needs_link" + default: + return "failed" + } +} + +// ClaimResult is a typed claim outcome, replacing string-matched errors. +type ClaimResult struct { + Outcome ClaimOutcome + LinkURL string + Detail string +} + +// DropProgressSource reports authoritative per-drop progress for the given +// campaigns. It returns an error rather than a partial result, so a failed +// sync never looks like "the platform forgot these drops". +type DropProgressSource interface { + DropProgress(ctx context.Context, s Session, camps []Campaign) ([]DropProgress, error) +} + +// DropClaimer claims one drop and classifies the outcome. +type DropClaimer interface { + ClaimDrop(ctx context.Context, s Session, d DropProgress) ClaimResult +} + +// ChannelProber checks specific channels for liveness and, when c.Game is +// set, that they stream the campaign's game. Used for priority streamers +// that platform directories may not list. +type ChannelProber interface { + ProbeChannels(ctx context.Context, s Session, c Campaign, logins []string) ([]Stream, error) +} +``` + +- [ ] **Step 2: Add `Self` to the details struct** (`campaigns.go`, inside `TimeBasedDrops []struct { ... }`, after `PreconditionDrops`) + +```go + // self is the viewer's own progress on this drop. Null when + // the account is not enrolled in the campaign yet. + Self *struct { + CurrentMinutesWatched int `json:"currentMinutesWatched"` + IsClaimed bool `json:"isClaimed"` + DropInstanceID string `json:"dropInstanceID"` + } `json:"self"` +``` + +- [ ] **Step 3: Split `claimStatus` out of `claim`** (`claim.go`, replace the `claim` function) + +```go +func (cl *claimer) claim(ctx context.Context, sess platform.Session, b platform.DropBenefit, userID int64) error { + status, err := cl.claimStatus(ctx, sess, b, userID) + if err != nil { + return err + } + switch status { + case "ELIGIBLE_FOR_ALL", "DROP_INSTANCE_ALREADY_CLAIMED", "": + return nil + default: + return fmt.Errorf("claim status: %s", status) + } +} + +// claimStatus sends the claim mutation and returns Twitch's raw status. +func (cl *claimer) claimStatus(ctx context.Context, sess platform.Session, b platform.DropBenefit, userID int64) (string, error) { + // Prefer the per-account instance id captured at progress time. + // When it's missing, construct DevilXD's synthetic instance id + // `userID#campaignID#dropID` (inventory.py generate_claim) — Twitch + // accepts it and rejects the bare drop-template id with + // INVALID_DROP_INSTANCE. Only fall back to the template id as a last + // resort when we couldn't resolve the user id. + id := b.InstanceID + if id == "" && userID > 0 && b.CampaignID != "" && b.ID != "" { + id = fmt.Sprintf("%d#%s#%s", userID, b.CampaignID, b.ID) + } + if id == "" { + id = b.ID + } + var out claimResult + if err := cl.c.gql(ctx, sess.AccessToken, OpClaimDrop, + map[string]any{"input": map[string]any{"dropInstanceID": id}}, &out); err != nil { + return "", fmt.Errorf("claim %s: %w", id, err) + } + return out.ClaimDropRewards.Status, nil +} +``` + +Run: `go test ./internal/platform/twitch/ -run Claim -v` +Expected: PASS (existing claim tests prove the refactor is behaviour-neutral). + +- [ ] **Step 4: Failing adapter tests** (`pipeline_test.go`) + +```go +package twitch + +import ( + "context" + "encoding/json" + "net/http" + "net/http/httptest" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/aalejandrofer/grubdrops/internal/platform" +) + +// pipelineTestBackend serves DropCampaignDetails + Inventory from fixtures. +func pipelineTestBackend(t *testing.T, details map[string]string, inventory string, detailsStatus int) *Backend { + t.Helper() + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + var req gqlRequest + require.NoError(t, json.NewDecoder(r.Body).Decode(&req)) + switch req.OperationName { + case OpDropCampaignDetails.Name: + if detailsStatus != 0 { + w.WriteHeader(detailsStatus) + return + } + id, _ := req.Variables["dropID"].(string) + _, _ = w.Write([]byte(details[id])) + case OpInventory.Name: + _, _ = w.Write([]byte(inventory)) + default: + t.Fatalf("unexpected op %q", req.OperationName) + } + })) + t.Cleanup(srv.Close) + return &Backend{disc: &discovery{c: newTestClient(srv.URL), userLogin: "testuser"}} +} + +const emptyInventory = `{"data":{"currentUser":{"inventory":{"dropCampaignsInProgress":[]}}}}` + +func TestDropProgress_ReadsSelfForCompletedCampaign(t *testing.T) { + // Campaign left dropCampaignsInProgress (fully claimed), but details self + // still reports isClaimed. This is the P1 fix. + b := pipelineTestBackend(t, map[string]string{"c1": `{"data":{"user":{"dropCampaign":{"timeBasedDrops":[ + {"id":"d1","requiredMinutesWatched":60,"benefitEdges":[],"self":{"currentMinutesWatched":60,"isClaimed":true,"dropInstanceID":"i1"}} + ]}}}}`}, emptyInventory, 0) + got, err := b.DropProgress(context.Background(), platform.Session{AccessToken: "t"}, + []platform.Campaign{{ID: "c1", Platform: "twitch"}}) + require.NoError(t, err) + require.Len(t, got, 1) + assert.Equal(t, platform.DropProgress{DropID: "d1", CampaignID: "c1", Minutes: 60, Required: 60, Claimed: true, InstanceID: "i1", Known: true}, got[0]) +} + +func TestDropProgress_NullSelfIsUnclaimed(t *testing.T) { + b := pipelineTestBackend(t, map[string]string{"c1": `{"data":{"user":{"dropCampaign":{"timeBasedDrops":[ + {"id":"d1","requiredMinutesWatched":60,"benefitEdges":[],"self":null} + ]}}}}`}, emptyInventory, 0) + got, err := b.DropProgress(context.Background(), platform.Session{AccessToken: "t"}, + []platform.Campaign{{ID: "c1", Platform: "twitch"}}) + require.NoError(t, err) + require.Len(t, got, 1) + assert.False(t, got[0].Claimed) + assert.Equal(t, 0, got[0].Minutes) + assert.True(t, got[0].Known) +} + +func TestDropProgress_DedupesMultiItemDrop(t *testing.T) { + b := pipelineTestBackend(t, map[string]string{"c1": `{"data":{"user":{"dropCampaign":{"timeBasedDrops":[ + {"id":"d1","requiredMinutesWatched":60,"benefitEdges":[{"benefit":{"id":"b1","name":"A"}},{"benefit":{"id":"b2","name":"B"}}],"self":null}, + {"id":"d1","requiredMinutesWatched":60,"benefitEdges":[],"self":null} + ]}}}}`}, emptyInventory, 0) + got, err := b.DropProgress(context.Background(), platform.Session{AccessToken: "t"}, + []platform.Campaign{{ID: "c1", Platform: "twitch"}}) + require.NoError(t, err) + assert.Len(t, got, 1) +} + +func TestDropProgress_InventoryFillsInstanceAndMinutes(t *testing.T) { + b := pipelineTestBackend(t, map[string]string{"c1": `{"data":{"user":{"dropCampaign":{"timeBasedDrops":[ + {"id":"d1","requiredMinutesWatched":60,"benefitEdges":[],"self":{"currentMinutesWatched":10,"isClaimed":false,"dropInstanceID":""}} + ]}}}}`}, `{"data":{"currentUser":{"inventory":{"dropCampaignsInProgress":[ + {"id":"c1","timeBasedDrops":[{"id":"d1","self":{"currentMinutesWatched":25,"isClaimed":false,"dropInstanceID":"inst"}}]} + ]}}}}`, 0) + got, err := b.DropProgress(context.Background(), platform.Session{AccessToken: "t"}, + []platform.Campaign{{ID: "c1", Platform: "twitch"}}) + require.NoError(t, err) + require.Len(t, got, 1) + assert.Equal(t, 25, got[0].Minutes) + assert.Equal(t, "inst", got[0].InstanceID) +} + +func TestDropProgress_SyntheticCampaignIsUnmineable(t *testing.T) { + b := pipelineTestBackend(t, nil, emptyInventory, 0) + got, err := b.DropProgress(context.Background(), platform.Session{AccessToken: "t"}, + []platform.Campaign{{ID: "Game|Name", Platform: "twitch", Benefits: []platform.DropBenefit{{ID: "Game|Name_default"}}}}) + require.NoError(t, err) + require.Len(t, got, 1) + assert.True(t, got[0].Unmineable) + assert.True(t, got[0].Known) +} + +func TestDropProgress_DetailsErrorFailsWhole(t *testing.T) { + b := pipelineTestBackend(t, nil, emptyInventory, http.StatusInternalServerError) + _, err := b.DropProgress(context.Background(), platform.Session{AccessToken: "t"}, + []platform.Campaign{{ID: "c1", Platform: "twitch"}}) + require.Error(t, err) +} + +func TestClassifyClaimStatus(t *testing.T) { + assert.Equal(t, platform.ClaimOK, classifyClaimStatus("ELIGIBLE_FOR_ALL").Outcome) + assert.Equal(t, platform.ClaimOK, classifyClaimStatus("").Outcome) + assert.Equal(t, platform.ClaimAlready, classifyClaimStatus("DROP_INSTANCE_ALREADY_CLAIMED").Outcome) + r := classifyClaimStatus("DROP_INSTANCE_ALREADY_EXPIRED") + assert.Equal(t, platform.ClaimFailed, r.Outcome) + assert.Contains(t, r.Detail, "DROP_INSTANCE_ALREADY_EXPIRED") +} + +func TestBackend_SatisfiesPipelineInterfaces(t *testing.T) { + var _ platform.DropProgressSource = (*Backend)(nil) + var _ platform.DropClaimer = (*Backend)(nil) + var _ platform.ChannelProber = (*Backend)(nil) +} +``` + +Run: `go test ./internal/platform/twitch/ -run 'DropProgress|ClassifyClaim|PipelineInterfaces' -v` +Expected: FAIL, `b.DropProgress undefined`. + +- [ ] **Step 5: Implement** (`internal/platform/twitch/pipeline.go`) + +```go +package twitch + +import ( + "context" + "fmt" + "strings" + + "github.com/aalejandrofer/grubdrops/internal/platform" +) + +// isSyntheticID reports scrape-fallback placeholder ids ("Game|Name"). +// Twitch can't report progress for them. +func isSyntheticID(id string) bool { return strings.ContainsAny(id, "| ") } + +// dropProgress reads DropCampaignDetails uncached, because self changes as +// the viewer watches and claims. One entry per drop id. +func (d *discovery) dropProgress(ctx context.Context, sess platform.Session, campaignID string) ([]platform.DropProgress, error) { + channelLogin := d.userLogin + if channelLogin == "" { + channelLogin = "twitch" + } + var det campaignDetailsData + if err := d.c.gql(ctx, sess.AccessToken, OpDropCampaignDetails, + map[string]any{"dropID": campaignID, "channelLogin": channelLogin}, &det); err != nil { + return nil, fmt.Errorf("drop progress %s: %w", campaignID, err) + } + seen := map[string]bool{} + var out []platform.DropProgress + for _, td := range det.User.DropCampaign.TimeBasedDrops { + if td.ID == "" || seen[td.ID] { + continue + } + seen[td.ID] = true + dp := platform.DropProgress{DropID: td.ID, CampaignID: campaignID, Required: td.RequiredMinutesWatched, Known: true} + if td.Self != nil { + dp.Minutes = td.Self.CurrentMinutesWatched + dp.Claimed = td.Self.IsClaimed + dp.InstanceID = td.Self.DropInstanceID + } + out = append(out, dp) + } + return out, nil +} + +// DropProgress satisfies platform.DropProgressSource. Per-campaign details +// are the truth (they survive the campaign leaving the in-progress +// inventory); the in-progress inventory only fills fresher minutes and the +// instance id. +func (b *Backend) DropProgress(ctx context.Context, s platform.Session, camps []platform.Campaign) ([]platform.DropProgress, error) { + inv, err := b.disc.inventory(ctx, s) + if err != nil { + return nil, err + } + byID := make(map[string]platform.Progress, len(inv)) + for _, p := range inv { + byID[p.BenefitID] = p + } + var out []platform.DropProgress + for _, c := range camps { + if isSyntheticID(c.ID) { + for _, bf := range c.Benefits { + out = append(out, platform.DropProgress{DropID: bf.ID, CampaignID: c.ID, Known: true, Unmineable: true}) + } + continue + } + dps, err := b.disc.dropProgress(ctx, s, c.ID) + if err != nil { + return nil, err + } + for i := range dps { + p, ok := byID[dps[i].DropID] + if !ok { + continue + } + if p.MinutesWatched > dps[i].Minutes { + dps[i].Minutes = p.MinutesWatched + } + dps[i].Claimed = dps[i].Claimed || p.Claimed + if dps[i].InstanceID == "" { + dps[i].InstanceID = p.InstanceID + } + } + out = append(out, dps...) + } + return out, nil +} + +// classifyClaimStatus maps Twitch's claimDropRewards.status to an outcome. +func classifyClaimStatus(status string) platform.ClaimResult { + switch status { + case "ELIGIBLE_FOR_ALL", "": + return platform.ClaimResult{Outcome: platform.ClaimOK} + case "DROP_INSTANCE_ALREADY_CLAIMED": + return platform.ClaimResult{Outcome: platform.ClaimAlready} + default: + return platform.ClaimResult{Outcome: platform.ClaimFailed, Detail: "claim status: " + status} + } +} + +// ClaimDrop satisfies platform.DropClaimer. Unlinked campaigns are blocked +// before claiming (reconciler reads self.isAccountConnected), so Twitch has +// no link outcome here. +func (b *Backend) ClaimDrop(ctx context.Context, s platform.Session, d platform.DropProgress) platform.ClaimResult { + userID, _ := b.watch.resolveUserID(ctx, s) + status, err := b.claim.claimStatus(ctx, s, platform.DropBenefit{ID: d.DropID, CampaignID: d.CampaignID, InstanceID: d.InstanceID}, userID) + if err != nil { + return platform.ClaimResult{Outcome: platform.ClaimFailed, Detail: err.Error()} + } + return classifyClaimStatus(status) +} + +// ProbeChannels satisfies platform.ChannelProber: parallel live check of the +// given logins, filtered to c.Game when set. +func (b *Backend) ProbeChannels(ctx context.Context, s platform.Session, c platform.Campaign, logins []string) ([]platform.Stream, error) { + return b.chans.listEligible(ctx, s, c, logins) +} + +var ( + _ platform.DropProgressSource = (*Backend)(nil) + _ platform.DropClaimer = (*Backend)(nil) + _ platform.ChannelProber = (*Backend)(nil) +) +``` + +- [ ] **Step 6: Run tests** + +Run: `go test ./internal/platform/... -v 2>&1 | tail -30` +Expected: PASS. If the 500 test does not error, check `client.gql` treats non-2xx as an error; if it doesn't, make the fixture return `{"errors":[{"message":"boom"}]}` instead, whichever `gql` rejects. + +- [ ] **Step 7: Commit** + +```bash +gofmt -w internal/platform +git add internal/platform +git commit -m "feat(pipeline-v2): platform progress/claim/probe capabilities, Twitch adapter + +Co-Authored-By: Claude Opus 5.5 (1M context) " +``` + +--- + +### Task 6: Kick adapter + +**Files:** +- Modify: `internal/platform/kick/api.go` (add `Required` to `progressReward`) +- Create: `internal/platform/kick/pipeline.go` +- Test: `internal/platform/kick/pipeline_test.go` + +**Interfaces:** +- Consumes: `platform.DropProgress`, `ClaimResult`, capability interfaces (Task 5); kick `api.progressDetail`, `api.Claim`, `ClaimNeedsLinkError`, `probeLive`, `kickChannel`. +- Produces: `(*kick.Backend).DropProgress`, `ClaimDrop`, `ProbeChannels`. + +- [ ] **Step 1: Add `Required` to `progressReward`** + +In `api.go`, add field `Required int // required_units` to `progressReward`, and in `progressDetail`'s append add: + +```go + Required: mnum(rm, "required_units", "required_minutes", "requiredMinutes", "minutes"), +``` + +- [ ] **Step 2: Failing tests** (`pipeline_test.go`) + +```go +package kick + +import ( + "context" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/aalejandrofer/grubdrops/internal/platform" +) + +const progressURL = "https://web.kick.com/api/v1/drops/progress" + +func TestKickDropProgress_RequiredFromPlatform(t *testing.T) { + f := &fakeDoer{resp: map[string]fakeResp{progressURL: {200, `{"data":[ + {"id":"c1","rewards":[{"id":"r1","claimed":false,"progress":0.5,"required_units":30}]} + ]}`}}} + b := withFake(f) + got, err := b.DropProgress(context.Background(), sess("acc1"), []platform.Campaign{{ + ID: "c1", Benefits: []platform.DropBenefit{{ID: "r1", CampaignID: "c1", RequiredMinutes: 120}}, + }}) + require.NoError(t, err) + require.Len(t, got, 1) + // Platform says 30, adapter's invented 120 must not win (P6). + assert.Equal(t, platform.DropProgress{DropID: "r1", CampaignID: "c1", Minutes: 15, Required: 30, Known: true}, got[0]) +} + +func TestKickDropProgress_AbsentRewardIsUnknown(t *testing.T) { + f := &fakeDoer{resp: map[string]fakeResp{progressURL: {200, `{"data":[]}`}}} + b := withFake(f) + got, err := b.DropProgress(context.Background(), sess("acc1"), []platform.Campaign{{ + ID: "c1", Benefits: []platform.DropBenefit{{ID: "r1", CampaignID: "c1", RequiredMinutes: 120}}, + }}) + require.NoError(t, err) + require.Len(t, got, 1) + assert.False(t, got[0].Known, "absence proves nothing") + assert.Equal(t, 120, got[0].Required) +} + +func TestKickDropProgress_ErrorFailsWhole(t *testing.T) { + f := &fakeDoer{resp: map[string]fakeResp{progressURL: {403, `{}`}}} + _, err := withFake(f).DropProgress(context.Background(), sess("acc1"), []platform.Campaign{{ID: "c1"}}) + require.Error(t, err) +} + +func TestKickClaimDrop_Outcomes(t *testing.T) { + claimURL := "https://web.kick.com/api/v1/drops/claim" + ok := withFake(&fakeDoer{resp: map[string]fakeResp{claimURL: {200, `{}`}}}) + assert.Equal(t, platform.ClaimOK, ok.ClaimDrop(context.Background(), sess("a"), platform.DropProgress{DropID: "r1", CampaignID: "c1"}).Outcome) + + link := withFake(&fakeDoer{resp: map[string]fakeResp{claimURL: {400, `{"connect_url":"https://kick.com/connect/riot"}`}}}) + r := link.ClaimDrop(context.Background(), sess("a"), platform.DropProgress{DropID: "r1", CampaignID: "c1"}) + assert.Equal(t, platform.ClaimNeedsLink, r.Outcome) + assert.Equal(t, "https://kick.com/connect/riot", r.LinkURL) + + fail := withFake(&fakeDoer{resp: map[string]fakeResp{claimURL: {500, `{}`}}}) + assert.Equal(t, platform.ClaimFailed, fail.ClaimDrop(context.Background(), sess("a"), platform.DropProgress{DropID: "r1", CampaignID: "c1"}).Outcome) +} + +func TestKickProbeChannels(t *testing.T) { + f := &fakeDoer{resp: map[string]fakeResp{ + "https://kick.com/api/v2/channels/fav/livestream": {200, `{"data":{"id":7,"viewer_count":3,"categories":[{"name":"Rust","slug":"rust"}]}}`}, + "https://kick.com/api/v2/channels/off/livestream": {200, `{"data":null}`}, + "https://kick.com/api/v2/channels/other/livestream": {200, `{"data":{"id":8,"viewer_count":9,"categories":[{"name":"Slots","slug":"slots"}]}}`}, + }} + got, err := withFake(f).ProbeChannels(context.Background(), sess("a"), platform.Campaign{Game: "Rust"}, []string{"Fav", "off", "other"}) + require.NoError(t, err) + require.Len(t, got, 1) + assert.Equal(t, "fav", got[0].Channel) +} + +func TestKickBackend_SatisfiesPipelineInterfaces(t *testing.T) { + var _ platform.DropProgressSource = (*Backend)(nil) + var _ platform.DropClaimer = (*Backend)(nil) + var _ platform.ChannelProber = (*Backend)(nil) +} +``` + +Run: `go test ./internal/platform/kick/ -run 'KickDropProgress|KickClaimDrop|KickProbe|PipelineInterfaces' -v` +Expected: FAIL, `b.DropProgress undefined`. + +- [ ] **Step 3: Implement** (`internal/platform/kick/pipeline.go`) + +```go +package kick + +import ( + "context" + "errors" + "fmt" + "math" + "strings" + + "github.com/aalejandrofer/grubdrops/internal/platform" +) + +// DropProgress satisfies platform.DropProgressSource. Required minutes come +// from /drops/progress required_units whenever Kick lists the reward. Kick +// lists a reward only after it accrues, so an unlisted reward is reported +// with Known=false instead of guessing. +func (b *Backend) DropProgress(ctx context.Context, s platform.Session, camps []platform.Campaign) ([]platform.DropProgress, error) { + rewards, err := b.api.progressDetail(ctx, s) + if err != nil { + return nil, fmt.Errorf("kick drop progress: %w", err) + } + byID := make(map[string]progressReward, len(rewards)) + for _, r := range rewards { + byID[r.RewardID] = r + } + var out []platform.DropProgress + for _, c := range camps { + for _, bf := range c.Benefits { + r, ok := byID[bf.ID] + if !ok { + out = append(out, platform.DropProgress{DropID: bf.ID, CampaignID: c.ID, Required: bf.RequiredMinutes}) + continue + } + req := r.Required + if req <= 0 { + req = bf.RequiredMinutes + } + minutes := 0 + if req > 0 && r.Fraction > 0 { + minutes = int(math.Round(r.Fraction * float64(req))) + } + out = append(out, platform.DropProgress{ + DropID: bf.ID, CampaignID: c.ID, Minutes: minutes, Required: req, + Claimed: r.Claimed, Known: true, + }) + } + } + return out, nil +} + +// ClaimDrop satisfies platform.DropClaimer. A claim error on an already +// auto-granted reward shows up as ClaimFailed; the post-claim reconcile +// then sees claimed=true and corrects the state. +func (b *Backend) ClaimDrop(ctx context.Context, s platform.Session, d platform.DropProgress) platform.ClaimResult { + err := b.api.Claim(ctx, s, d.DropID, d.CampaignID) + if err == nil { + return platform.ClaimResult{Outcome: platform.ClaimOK} + } + var linkErr *ClaimNeedsLinkError + if errors.As(err, &linkErr) { + return platform.ClaimResult{Outcome: platform.ClaimNeedsLink, LinkURL: linkErr.ConnectURL, Detail: err.Error()} + } + return platform.ClaimResult{Outcome: platform.ClaimFailed, Detail: err.Error()} +} + +// ProbeChannels satisfies platform.ChannelProber on top of probeLive. +func (b *Backend) ProbeChannels(ctx context.Context, s platform.Session, c platform.Campaign, logins []string) ([]platform.Stream, error) { + pool := make([]kickChannel, 0, len(logins)) + for _, l := range logins { + if l = strings.ToLower(strings.TrimSpace(l)); l != "" { + pool = append(pool, kickChannel{Slug: l}) + } + } + return b.probeLive(ctx, s, c, pool), nil +} + +var ( + _ platform.DropProgressSource = (*Backend)(nil) + _ platform.DropClaimer = (*Backend)(nil) + _ platform.ChannelProber = (*Backend)(nil) +) +``` + +- [ ] **Step 4: Run tests** + +Run: `go test ./internal/platform/kick/ -v 2>&1 | tail -30` +Expected: PASS (including existing sweep tests, since `progressReward` only gained a field). + +- [ ] **Step 5: Commit** + +```bash +gofmt -w internal/platform +git add internal/platform/kick +git commit -m "feat(pipeline-v2): Kick adapter with platform-sourced required minutes + +Co-Authored-By: Claude Opus 5.5 (1M context) " +``` + +--- + +### Task 7: Reconciler + +**Files:** +- Create: `internal/pipeline/reconcile/reconcile.go` +- Test: `internal/pipeline/reconcile/reconcile_test.go` + +**Interfaces:** +- Consumes: `dropstate` (Task 2), `platform.Backend.ListActiveCampaigns`, `platform.DropProgressSource` (Task 5). +- Produces: + +```go +type CampaignPersister interface { PersistCampaigns(ctx context.Context, camps []platform.Campaign) error } +type Config struct { + AccountID, Platform string + Backend platform.Backend + Progress platform.DropProgressSource + Session platform.Session + AllowGame func(game string) bool + AllowChannel func(channels []string) bool + ForceLinked func(campaignID string) bool + Persister CampaignPersister +} +type Result struct { + Campaigns []platform.Campaign // active, in scope + Rows []dropstate.Row // every row touched this sync + Progress map[string]platform.DropProgress // by drop id +} +func Run(ctx context.Context, cfg Config, prev map[string]dropstate.Row, now time.Time) (Result, error) +func InScope(camps []platform.Campaign, allowGame func(string) bool, allowChannel func([]string) bool) []platform.Campaign +``` + +- [ ] **Step 1: Failing tests** + +```go +package reconcile + +import ( + "context" + "errors" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/aalejandrofer/grubdrops/internal/pipeline/dropstate" + "github.com/aalejandrofer/grubdrops/internal/platform" +) + +type fakeBackend struct { + platform.Backend // nil: unimplemented methods panic + camps []platform.Campaign + progress []platform.DropProgress + progErr error +} + +func (f *fakeBackend) ListActiveCampaigns(context.Context, platform.Session) ([]platform.Campaign, error) { + return f.camps, nil +} + +func (f *fakeBackend) DropProgress(context.Context, platform.Session, []platform.Campaign) ([]platform.DropProgress, error) { + return f.progress, f.progErr +} + +type fakePersister struct{ got []platform.Campaign } + +func (p *fakePersister) PersistCampaigns(_ context.Context, c []platform.Campaign) error { + p.got = append(p.got, c...) + return nil +} + +var now = time.Unix(1_700_000_000, 0) + +func camp(id, game string, drops ...string) platform.Campaign { + c := platform.Campaign{ID: id, Platform: "twitch", Game: game, Status: "active", AccountLinked: true, AccountLinkChecked: true} + for _, d := range drops { + c.Benefits = append(c.Benefits, platform.DropBenefit{ID: d, CampaignID: id, RequiredMinutes: 60}) + } + return c +} + +func cfg(f *fakeBackend, p *fakePersister) Config { + return Config{AccountID: "a", Platform: "twitch", Backend: f, Progress: f, Persister: p, + AllowGame: func(g string) bool { return g == "G" }} +} + +func byDrop(rows []dropstate.Row) map[string]dropstate.Row { + m := map[string]dropstate.Row{} + for _, r := range rows { + m[r.DropID] = r + } + return m +} + +func TestRun_ClaimedAfterLeavingInventory(t *testing.T) { + f := &fakeBackend{camps: []platform.Campaign{camp("c1", "G", "d1")}, + progress: []platform.DropProgress{{DropID: "d1", CampaignID: "c1", Minutes: 60, Required: 60, Claimed: true, Known: true}}} + prev := map[string]dropstate.Row{"d1": {AccountID: "a", DropID: "d1", CampaignID: "c1", Platform: "twitch", Status: dropstate.Accruing, Minutes: 30, Required: 60, Source: dropstate.FromPlatform}} + res, err := Run(context.Background(), cfg(f, &fakePersister{}), prev, now) + require.NoError(t, err) + assert.Equal(t, dropstate.Claimed, byDrop(res.Rows)["d1"].Status) +} + +func TestRun_NewDropSeedsIdentity(t *testing.T) { + f := &fakeBackend{camps: []platform.Campaign{camp("c1", "G", "d1")}, + progress: []platform.DropProgress{{DropID: "d1", CampaignID: "c1", Minutes: 5, Required: 60, Known: true}}} + res, err := Run(context.Background(), cfg(f, &fakePersister{}), nil, now) + require.NoError(t, err) + r := byDrop(res.Rows)["d1"] + assert.Equal(t, "a", r.AccountID) + assert.Equal(t, "c1", r.CampaignID) + assert.Equal(t, "twitch", r.Platform) + assert.Equal(t, dropstate.Accruing, r.Status) +} + +func TestRun_ProgressErrorChangesNothing(t *testing.T) { + f := &fakeBackend{camps: []platform.Campaign{camp("c1", "G", "d1")}, progErr: errors.New("details 500")} + prev := map[string]dropstate.Row{"d1": {DropID: "d1", Status: dropstate.Accruing, Minutes: 30, Required: 60}} + res, err := Run(context.Background(), cfg(f, &fakePersister{}), prev, now) + require.Error(t, err) + assert.Empty(t, res.Rows) +} + +func TestRun_MultiItemDropOneRow(t *testing.T) { + c := camp("c1", "G", "d1", "d1") + f := &fakeBackend{camps: []platform.Campaign{c}, + progress: []platform.DropProgress{{DropID: "d1", CampaignID: "c1", Required: 60, Known: true}}} + res, err := Run(context.Background(), cfg(f, &fakePersister{}), nil, now) + require.NoError(t, err) + assert.Len(t, res.Rows, 1) +} + +func TestRun_UnlinkedCampaignBlocked_UnlessForced(t *testing.T) { + c := camp("c1", "G", "d1") + c.AccountLinked = false + f := &fakeBackend{camps: []platform.Campaign{c}, + progress: []platform.DropProgress{{DropID: "d1", CampaignID: "c1", Required: 60, Known: true}}} + res, err := Run(context.Background(), cfg(f, &fakePersister{}), nil, now) + require.NoError(t, err) + assert.Equal(t, dropstate.NeedsLink, byDrop(res.Rows)["d1"].Reason) + + cf := cfg(f, &fakePersister{}) + cf.ForceLinked = func(id string) bool { return id == "c1" } + res, err = Run(context.Background(), cf, nil, now) + require.NoError(t, err) + assert.Equal(t, dropstate.Eligible, byDrop(res.Rows)["d1"].Status) +} + +func TestRun_ExpiredCampaign(t *testing.T) { + c := camp("c1", "G", "d1") + c.Status = "expired" + f := &fakeBackend{camps: []platform.Campaign{c}} + prev := map[string]dropstate.Row{"d1": {DropID: "d1", Status: dropstate.Accruing, Minutes: 30, Required: 60}} + res, err := Run(context.Background(), cfg(f, &fakePersister{}), prev, now) + require.NoError(t, err) + assert.Equal(t, dropstate.Expired, byDrop(res.Rows)["d1"].Reason) + assert.Empty(t, res.Campaigns, "expired campaigns are not mining candidates") +} + +func TestRun_ScopeAndPersist(t *testing.T) { + nullGame := camp("c2", "", "d2") + nullGame.AllowedChannels = []string{"fav"} + reward := camp("c3", "G", "d3") + reward.Kind = "reward" + f := &fakeBackend{camps: []platform.Campaign{camp("c1", "Other", "d1"), nullGame, reward}, + progress: []platform.DropProgress{{DropID: "d2", CampaignID: "c2", Required: 60, Known: true}}} + p := &fakePersister{} + cf := cfg(f, p) + cf.AllowChannel = func(chs []string) bool { return len(chs) > 0 && chs[0] == "fav" } + res, err := Run(context.Background(), cf, nil, now) + require.NoError(t, err) + got := byDrop(res.Rows) + assert.NotContains(t, got, "d1", "non-whitelisted game") + assert.Contains(t, got, "d2", "null-game campaign allowed by channel") + assert.NotContains(t, got, "d3", "reward kind has no watch time") + require.Len(t, p.got, 1) + assert.Equal(t, "c2", p.got[0].ID) +} +``` + +Run: `go test ./internal/pipeline/reconcile/ -v` +Expected: FAIL, `undefined: Run`. + +- [ ] **Step 2: Implement** + +```go +// Package reconcile folds platform per-campaign progress into drop_state. +// It never decides what to watch. +package reconcile + +import ( + "context" + "fmt" + "time" + + "github.com/aalejandrofer/grubdrops/internal/pipeline/dropstate" + "github.com/aalejandrofer/grubdrops/internal/platform" +) + +type CampaignPersister interface { + PersistCampaigns(ctx context.Context, camps []platform.Campaign) error +} + +type Config struct { + AccountID string + Platform string + Backend platform.Backend + Progress platform.DropProgressSource + Session platform.Session + AllowGame func(game string) bool + AllowChannel func(channels []string) bool + ForceLinked func(campaignID string) bool + Persister CampaignPersister +} + +type Result struct { + Campaigns []platform.Campaign + Rows []dropstate.Row + Progress map[string]platform.DropProgress +} + +// InScope keeps campaigns the account mines: whitelisted game, or a +// campaign whose channels match the account's priority streamers +// (null-game drops). One-click reward campaigns have no watch time. +func InScope(camps []platform.Campaign, allowGame func(string) bool, allowChannel func([]string) bool) []platform.Campaign { + var out []platform.Campaign + for _, c := range camps { + if c.Kind == "reward" { + continue + } + ok := allowGame == nil || allowGame(c.Game) + if !ok && allowChannel != nil && allowChannel(c.AllowedChannels) { + ok = true + } + if ok { + out = append(out, c) + } + } + return out +} + +func isActive(c platform.Campaign, now time.Time) bool { + if c.Status != "" && c.Status != "active" { + return false + } + return c.EndsAt.IsZero() || now.Before(c.EndsAt) +} + +// Run lists campaigns, persists the in-scope ones, reads their progress and +// returns every drop row it touched. On any platform error it returns the +// error and no rows, so a failed sync never changes state. +func Run(ctx context.Context, cfg Config, prev map[string]dropstate.Row, now time.Time) (Result, error) { + camps, err := cfg.Backend.ListActiveCampaigns(ctx, cfg.Session) + if err != nil { + return Result{}, fmt.Errorf("list campaigns: %w", err) + } + scope := InScope(camps, cfg.AllowGame, cfg.AllowChannel) + if cfg.Persister != nil && len(scope) > 0 { + if err := cfg.Persister.PersistCampaigns(ctx, scope); err != nil { + return Result{}, fmt.Errorf("persist campaigns: %w", err) + } + } + var active []platform.Campaign + for _, c := range scope { + if isActive(c, now) { + active = append(active, c) + } + } + byDrop := map[string]platform.DropProgress{} + if len(active) > 0 { + obs, err := cfg.Progress.DropProgress(ctx, cfg.Session, active) + if err != nil { + return Result{}, fmt.Errorf("drop progress: %w", err) + } + for _, o := range obs { + byDrop[o.DropID] = o + } + } + + res := Result{Campaigns: active, Progress: byDrop} + seen := map[string]bool{} + for _, c := range scope { + if c.Status == "upcoming" { + continue + } + expired := !isActive(c, now) + linked := c.AccountLinked || !c.AccountLinkChecked || (cfg.ForceLinked != nil && cfg.ForceLinked(c.ID)) + for _, b := range c.Benefits { + if b.ID == "" || seen[b.ID] { + continue + } + seen[b.ID] = true + p, ok := prev[b.ID] + if !ok || p.IsZero() { + p = dropstate.Row{AccountID: cfg.AccountID, DropID: b.ID, CampaignID: c.ID, Platform: cfg.Platform, Source: dropstate.FromPlatform} + } + var next dropstate.Row + if expired { + if p.IsZero() { + continue // never tracked, nothing to expire + } + next = dropstate.Expire(p, now) + } else { + o, known := byDrop[b.ID] + req := b.RequiredMinutes + if known { + req = o.Required + } + next = dropstate.Apply(p, dropstate.Observation{ + Known: known && o.Known, Unmineable: o.Unmineable, Claimed: o.Claimed, + Minutes: o.Minutes, Required: req, + }, now) + if !linked { + next = dropstate.BlockLink(next, now) + } + } + res.Rows = append(res.Rows, next) + } + } + return res, nil +} +``` + +- [ ] **Step 3: Run tests** + +Run: `go test ./internal/pipeline/... -v 2>&1 | tail -30` +Expected: PASS. + +- [ ] **Step 4: Commit** + +```bash +gofmt -w internal/pipeline +git add internal/pipeline/reconcile +git commit -m "feat(pipeline-v2): reconciler folds platform progress into drop state + +Co-Authored-By: Claude Opus 5.5 (1M context) " +``` + +--- + +### Task 8: Planner (incl. streamer priority, #49) + +**Files:** +- Create: `internal/pipeline/planner/planner.go` +- Test: `internal/pipeline/planner/planner_test.go` + +**Interfaces:** +- Consumes: `dropstate.Row`, `dropstate.Mineable`, `dropstate.Claimed`; `platform.Campaign`, `platform.Stream`. +- Produces: + +```go +type Kind int // Idle, Mine, ForceWatch; String() "idle" | "mine" | "force_watch" +type Settings struct { + GameRank func(game string) int + PriorityMode string // "ordered" | "ending_soonest" + StreamerPriority []string + ForceWatch []platform.Stream // live force-watch channels, rank order +} +type Input struct { + Now time.Time + Rows map[string]dropstate.Row + Campaigns []platform.Campaign + Settings Settings + Live map[string][]platform.Stream // campaign id -> live eligible channels + Cooldowns map[string]time.Time // lowercased channel -> until + Current *Decision + LastSwap time.Time +} +type Decision struct { + Kind Kind + CampaignID string + Channel platform.Stream + Serves []string + Reason string +} +func (d Decision) Same(o Decision) bool +func Plan(in Input) Decision +func Candidates(in Input) []platform.Campaign +const SwapHold = 10 * time.Minute +const ReasonNothingToMine = "nothing_to_mine"; const ReasonNoLiveChannels = "no_live_channels" +``` + +- [ ] **Step 1: Failing tests** + +```go +package planner + +import ( + "testing" + "time" + + "github.com/stretchr/testify/assert" + + "github.com/aalejandrofer/grubdrops/internal/pipeline/dropstate" + "github.com/aalejandrofer/grubdrops/internal/platform" +) + +var now = time.Unix(1_700_000_000, 0) + +func camp(id, game string, ends time.Time, drops ...platform.DropBenefit) platform.Campaign { + return platform.Campaign{ID: id, Game: game, Status: "active", EndsAt: ends, Benefits: drops} +} + +func drop(id, campaign string, req int, pre ...string) platform.DropBenefit { + return platform.DropBenefit{ID: id, CampaignID: campaign, RequiredMinutes: req, Preconditions: pre} +} + +func rows(rs ...dropstate.Row) map[string]dropstate.Row { + m := map[string]dropstate.Row{} + for _, r := range rs { + m[r.DropID] = r + } + return m +} + +func st(id string, s dropstate.Status) dropstate.Row { return dropstate.Row{DropID: id, Status: s} } + +func stream(ch string, viewers int) platform.Stream { + return platform.Stream{Channel: ch, ViewerCount: viewers} +} + +func TestPlan_PicksHighestViewerByDefault(t *testing.T) { + in := Input{Now: now, + Campaigns: []platform.Campaign{camp("c1", "G", time.Time{}, drop("d1", "c1", 60))}, + Rows: rows(st("d1", dropstate.Eligible)), + Live: map[string][]platform.Stream{"c1": {stream("small", 5), stream("big", 500)}}, + } + d := Plan(in) + assert.Equal(t, Mine, d.Kind) + assert.Equal(t, "big", d.Channel.Channel) + assert.Equal(t, []string{"d1"}, d.Serves) +} + +func TestPlan_PriorityStreamerFirst(t *testing.T) { + in := Input{Now: now, + Campaigns: []platform.Campaign{camp("c1", "G", time.Time{}, drop("d1", "c1", 60))}, + Rows: rows(st("d1", dropstate.Eligible)), + Live: map[string][]platform.Stream{"c1": {stream("big", 500), stream("Fav2", 1), stream("fav1", 2)}}, + Settings: Settings{StreamerPriority: []string{"fav1", "fav2"}}, + } + assert.Equal(t, "fav1", Plan(in).Channel.Channel) + in.Live["c1"] = []platform.Stream{stream("big", 500), stream("Fav2", 1)} + assert.Equal(t, "Fav2", Plan(in).Channel.Channel, "case-insensitive, next priority") + in.Live["c1"] = []platform.Stream{stream("big", 500)} + assert.Equal(t, "big", Plan(in).Channel.Channel, "fallback to category") +} + +func TestPlan_GameRankThenEndingSoonest(t *testing.T) { + soon, late := now.Add(time.Hour), now.Add(48*time.Hour) + in := Input{Now: now, + Campaigns: []platform.Campaign{camp("late", "A", late, drop("d1", "late", 60)), camp("soon", "B", soon, drop("d2", "soon", 60))}, + Rows: rows(st("d1", dropstate.Eligible), st("d2", dropstate.Eligible)), + Live: map[string][]platform.Stream{"late": {stream("x", 1)}, "soon": {stream("y", 1)}}, + Settings: Settings{GameRank: func(g string) int { return map[string]int{"A": 0, "B": 1}[g] }}, + } + assert.Equal(t, "late", Plan(in).CampaignID, "ordered: game rank wins") + in.Settings.PriorityMode = "ending_soonest" + assert.Equal(t, "soon", Plan(in).CampaignID) +} + +func TestPlan_SkipsNonMineableAndLockedPreconditions(t *testing.T) { + in := Input{Now: now, + Campaigns: []platform.Campaign{camp("c1", "G", time.Time{}, + drop("claimed", "c1", 30), drop("locked", "c1", 60, "gate"), drop("gate", "c1", 10))}, + Rows: rows(st("claimed", dropstate.Claimed), st("locked", dropstate.Eligible), st("gate", dropstate.Blocked)), + Live: map[string][]platform.Stream{"c1": {stream("x", 1)}}, + } + assert.Equal(t, Idle, Plan(in).Kind, "gate is blocked, locked waits on it") + + in.Rows["gate"] = dropstate.Row{DropID: "gate", Status: dropstate.Claimed, Source: dropstate.FromUser} + d := Plan(in) + assert.Equal(t, Mine, d.Kind, "user-marked precondition unlocks (P9)") + assert.Equal(t, []string{"locked"}, d.Serves) +} + +func TestPlan_ServesDropsAcrossCampaignsOnSameChannel(t *testing.T) { + in := Input{Now: now, + Campaigns: []platform.Campaign{ + camp("c1", "Rust", time.Time{}, drop("r1", "c1", 60), drop("r2", "c1", 30)), + camp("c2", "Rust", time.Time{}, drop("r3", "c2", 120)), + }, + Rows: rows(st("r1", dropstate.Eligible), st("r2", dropstate.Accruing), st("r3", dropstate.Eligible)), + Live: map[string][]platform.Stream{"c1": {stream("oilrats", 9)}, "c2": {stream("OILRATS", 9)}}, + } + d := Plan(in) + assert.Equal(t, []string{"r2", "r1", "r3"}, d.Serves, "lowest tier first, then other campaigns") +} + +func TestPlan_CooldownSkipsChannel(t *testing.T) { + in := Input{Now: now, + Campaigns: []platform.Campaign{camp("c1", "G", time.Time{}, drop("d1", "c1", 60))}, + Rows: rows(st("d1", dropstate.Eligible)), + Live: map[string][]platform.Stream{"c1": {stream("big", 500), stream("small", 5)}}, + Cooldowns: map[string]time.Time{"big": now.Add(time.Minute)}, + } + assert.Equal(t, "small", Plan(in).Channel.Channel) +} + +func TestPlan_HysteresisHoldsThenBypassesDeadChannel(t *testing.T) { + cur := Decision{Kind: Mine, CampaignID: "c1", Channel: stream("big", 500), Serves: []string{"d1"}} + in := Input{Now: now, + Campaigns: []platform.Campaign{camp("c1", "G", time.Time{}, drop("d1", "c1", 60))}, + Rows: rows(st("d1", dropstate.Eligible)), + Live: map[string][]platform.Stream{"c1": {stream("big", 500), stream("fav", 1)}}, + Settings: Settings{StreamerPriority: []string{"fav"}}, + Current: &cur, LastSwap: now.Add(-time.Minute), + } + assert.Equal(t, "big", Plan(in).Channel.Channel, "inside hold: keep current") + in.LastSwap = now.Add(-SwapHold - time.Second) + assert.Equal(t, "fav", Plan(in).Channel.Channel, "hold over: priority streamer wins") + in.LastSwap = now.Add(-time.Minute) + in.Live["c1"] = []platform.Stream{stream("fav", 1)} + assert.Equal(t, "fav", Plan(in).Channel.Channel, "current went offline: bypass hold") +} + +func TestPlan_ForceWatchOnlyWhenNothingToMine(t *testing.T) { + in := Input{Now: now, Settings: Settings{ForceWatch: []platform.Stream{stream("pts", 1)}}} + d := Plan(in) + assert.Equal(t, ForceWatch, d.Kind) + assert.Equal(t, "pts", d.Channel.Channel) + + in.Settings.ForceWatch = nil + d = Plan(in) + assert.Equal(t, Idle, d.Kind) + assert.Equal(t, ReasonNothingToMine, d.Reason) +} + +func TestPlan_NoLiveChannelsIdleReason(t *testing.T) { + in := Input{Now: now, + Campaigns: []platform.Campaign{camp("c1", "G", time.Time{}, drop("d1", "c1", 60))}, + Rows: rows(st("d1", dropstate.Eligible)), + } + assert.Equal(t, ReasonNoLiveChannels, Plan(in).Reason) +} + +func TestPlan_OutsideWindowIgnored(t *testing.T) { + c := camp("c1", "G", now.Add(-time.Minute), drop("d1", "c1", 60)) + in := Input{Now: now, Campaigns: []platform.Campaign{c}, Rows: rows(st("d1", dropstate.Eligible)), + Live: map[string][]platform.Stream{"c1": {stream("x", 1)}}} + assert.Equal(t, Idle, Plan(in).Kind) +} + +func TestDecision_Same(t *testing.T) { + a := Decision{Kind: Mine, Channel: stream("Big", 1)} + assert.True(t, a.Same(Decision{Kind: Mine, Channel: stream("big", 9)})) + assert.False(t, a.Same(Decision{Kind: ForceWatch, Channel: stream("big", 9)})) + assert.True(t, Decision{Kind: Idle}.Same(Decision{Kind: Idle, Reason: "x"})) +} +``` + +Run: `go test ./internal/pipeline/planner/ -v` +Expected: FAIL, `undefined: Input`. + +- [ ] **Step 2: Implement** + +```go +// Package planner turns drop state, campaigns and live channels into one +// decision: which channel to watch and which drops that serves. Pure: no +// I/O, no clock reads, no goroutines. +package planner + +import ( + "sort" + "strings" + "time" + + "github.com/aalejandrofer/grubdrops/internal/pipeline/dropstate" + "github.com/aalejandrofer/grubdrops/internal/platform" +) + +type Kind int + +const ( + Idle Kind = iota + Mine + ForceWatch +) + +func (k Kind) String() string { + switch k { + case Mine: + return "mine" + case ForceWatch: + return "force_watch" + default: + return "idle" + } +} + +// SwapHold is the minimum time on a channel before the planner switches to +// a better one, unless the current channel is dead or cooling down. +const SwapHold = 10 * time.Minute + +const ( + ReasonNothingToMine = "nothing_to_mine" + ReasonNoLiveChannels = "no_live_channels" +) + +type Settings struct { + GameRank func(game string) int + PriorityMode string + StreamerPriority []string + ForceWatch []platform.Stream +} + +type Input struct { + Now time.Time + Rows map[string]dropstate.Row + Campaigns []platform.Campaign + Settings Settings + Live map[string][]platform.Stream + Cooldowns map[string]time.Time + Current *Decision + LastSwap time.Time +} + +type Decision struct { + Kind Kind + CampaignID string + Channel platform.Stream + Serves []string + Reason string +} + +// Same reports whether two decisions watch the same thing. +func (d Decision) Same(o Decision) bool { + return d.Kind == o.Kind && strings.EqualFold(d.Channel.Channel, o.Channel.Channel) +} + +func key(ch string) string { return strings.ToLower(ch) } + +func cooling(in Input, ch string) bool { + until, ok := in.Cooldowns[key(ch)] + return ok && in.Now.Before(until) +} + +func inWindow(c platform.Campaign, now time.Time) bool { + if c.Status != "" && c.Status != "active" { + return false + } + if !c.StartsAt.IsZero() && now.Before(c.StartsAt) { + return false + } + return c.EndsAt.IsZero() || now.Before(c.EndsAt) +} + +// mineable returns the campaign's mineable drop ids, lowest tier first. +func mineable(in Input, c platform.Campaign) []string { + type d struct { + id string + req int + } + seen := map[string]bool{} + var ds []d + for _, b := range c.Benefits { + if seen[b.ID] || !dropstate.Mineable(in.Rows[b.ID]) { + continue + } + ok := true + for _, p := range b.Preconditions { + if in.Rows[p].Status != dropstate.Claimed { + ok = false + break + } + } + if !ok { + continue + } + seen[b.ID] = true + ds = append(ds, d{b.ID, b.RequiredMinutes}) + } + sort.SliceStable(ds, func(i, j int) bool { + if ds[i].req != ds[j].req { + return ds[i].req < ds[j].req + } + return ds[i].id < ds[j].id + }) + out := make([]string, len(ds)) + for i, x := range ds { + out[i] = x.id + } + return out +} + +func ends(c platform.Campaign) time.Time { + if c.EndsAt.IsZero() { + return time.Unix(1<<62, 0) + } + return c.EndsAt +} + +// Candidates returns in-window campaigns with at least one mineable drop, +// best first: game rank then ending soonest (ending soonest first when +// PriorityMode is "ending_soonest"). +func Candidates(in Input) []platform.Campaign { + var out []platform.Campaign + for _, c := range in.Campaigns { + if inWindow(c, in.Now) && len(mineable(in, c)) > 0 { + out = append(out, c) + } + } + rank := func(g string) int { + if in.Settings.GameRank == nil { + return 0 + } + return in.Settings.GameRank(g) + } + sort.SliceStable(out, func(i, j int) bool { + a, b := out[i], out[j] + if in.Settings.PriorityMode == "ending_soonest" && !ends(a).Equal(ends(b)) { + return ends(a).Before(ends(b)) + } + if ra, rb := rank(a.Game), rank(b.Game); ra != rb { + return ra < rb + } + if !ends(a).Equal(ends(b)) { + return ends(a).Before(ends(b)) + } + return a.ID < b.ID + }) + return out +} + +func hasChannel(streams []platform.Stream, ch string) bool { + for _, s := range streams { + if strings.EqualFold(s.Channel, ch) { + return true + } + } + return false +} + +func servedBy(in Input, cands []platform.Campaign, ch string) []string { + var out []string + for _, c := range cands { + if hasChannel(in.Live[c.ID], ch) { + out = append(out, mineable(in, c)...) + } + } + return out +} + +func pickChannel(in Input, c platform.Campaign) (platform.Stream, bool) { + prio := map[string]int{} + for i, l := range in.Settings.StreamerPriority { + if _, dup := prio[key(l)]; !dup { + prio[key(l)] = i + } + } + live := append([]platform.Stream(nil), in.Live[c.ID]...) + sort.SliceStable(live, func(i, j int) bool { + pi, iok := prio[key(live[i].Channel)] + pj, jok := prio[key(live[j].Channel)] + if iok != jok { + return iok + } + if iok && pi != pj { + return pi < pj + } + return live[i].ViewerCount > live[j].ViewerCount + }) + for _, s := range live { + if !cooling(in, s.Channel) { + return s, true + } + } + return platform.Stream{}, false +} + +func forceOrIdle(in Input, reason string) Decision { + for _, s := range in.Settings.ForceWatch { + if !cooling(in, s.Channel) { + return Decision{Kind: ForceWatch, Channel: s, Reason: "force-watch " + s.Channel} + } + } + return Decision{Kind: Idle, Reason: reason} +} + +// Plan decides what the account should watch now. +func Plan(in Input) Decision { + cands := Candidates(in) + if len(cands) == 0 { + return forceOrIdle(in, ReasonNothingToMine) + } + if cur := in.Current; cur != nil && cur.Kind == Mine && in.Now.Sub(in.LastSwap) < SwapHold && !cooling(in, cur.Channel.Channel) { + if serves := servedBy(in, cands, cur.Channel.Channel); len(serves) > 0 { + d := *cur + d.Serves, d.Reason = serves, "holding current channel" + return d + } + } + for _, c := range cands { + ch, ok := pickChannel(in, c) + if !ok { + continue + } + reason := "top viewers" + for _, l := range in.Settings.StreamerPriority { + if strings.EqualFold(l, ch.Channel) { + reason = "priority streamer" + break + } + } + return Decision{Kind: Mine, CampaignID: c.ID, Channel: ch, Serves: servedBy(in, cands, ch.Channel), Reason: reason} + } + return forceOrIdle(in, ReasonNoLiveChannels) +} +``` + +- [ ] **Step 3: Run tests** + +Run: `go test ./internal/pipeline/planner/ -v` +Expected: PASS. + +- [ ] **Step 4: Commit** + +```bash +gofmt -w internal/pipeline +git add internal/pipeline/planner +git commit -m "feat(pipeline-v2): pure planner with streamer priority and swap hysteresis + +Refs #49 + +Co-Authored-By: Claude Opus 5.5 (1M context) " +``` + +--- + +### Task 9: Claimer + +**Files:** +- Create: `internal/pipeline/claimer/claimer.go` +- Test: `internal/pipeline/claimer/claimer_test.go` + +**Interfaces:** +- Consumes: `platform.DropClaimer`, `platform.ClaimResult` (Task 5); `dropstate.ClaimOK/ClaimNeedsLink/ClaimFailedAttempt` (Task 2). +- Produces: `func Attempt(ctx context.Context, c platform.DropClaimer, s platform.Session, row dropstate.Row, dp platform.DropProgress, now time.Time) (dropstate.Row, platform.ClaimResult)` + +- [ ] **Step 1: Failing test** + +```go +package claimer + +import ( + "context" + "testing" + "time" + + "github.com/stretchr/testify/assert" + + "github.com/aalejandrofer/grubdrops/internal/pipeline/dropstate" + "github.com/aalejandrofer/grubdrops/internal/platform" +) + +type fakeClaimer struct { + res platform.ClaimResult + got platform.DropProgress +} + +func (f *fakeClaimer) ClaimDrop(_ context.Context, _ platform.Session, d platform.DropProgress) platform.ClaimResult { + f.got = d + return f.res +} + +func TestAttempt(t *testing.T) { + now := time.Unix(1_700_000_000, 0) + row := dropstate.Row{DropID: "d1", CampaignID: "c1", Status: dropstate.Claimable, Minutes: 60, Required: 60} + cases := []struct { + out platform.ClaimOutcome + status dropstate.Status + reason dropstate.Reason + }{ + {platform.ClaimOK, dropstate.Claimed, dropstate.NoReason}, + {platform.ClaimAlready, dropstate.Claimed, dropstate.NoReason}, + {platform.ClaimNeedsLink, dropstate.Blocked, dropstate.NeedsLink}, + {platform.ClaimFailed, dropstate.Claimable, dropstate.NoReason}, + } + for _, tc := range cases { + t.Run(tc.out.String(), func(t *testing.T) { + fc := &fakeClaimer{res: platform.ClaimResult{Outcome: tc.out}} + got, res := Attempt(context.Background(), fc, platform.Session{}, row, platform.DropProgress{DropID: "d1", InstanceID: "i"}, now) + assert.Equal(t, tc.status, got.Status) + assert.Equal(t, tc.reason, got.Reason) + assert.Equal(t, tc.out, res.Outcome) + assert.Equal(t, "i", fc.got.InstanceID, "instance id forwarded") + }) + } + fc := &fakeClaimer{res: platform.ClaimResult{Outcome: platform.ClaimFailed}} + got, _ := Attempt(context.Background(), fc, platform.Session{}, row, platform.DropProgress{DropID: "d1"}, now) + assert.Equal(t, 1, got.FailCount) + assert.Equal(t, now.Add(time.Minute), got.RetryAfter) +} +``` + +Run: `go test ./internal/pipeline/claimer/ -v` +Expected: FAIL, `undefined: Attempt`. + +- [ ] **Step 2: Implement** + +```go +// Package claimer turns a claimable drop into claimed or blocked. +package claimer + +import ( + "context" + "time" + + "github.com/aalejandrofer/grubdrops/internal/pipeline/dropstate" + "github.com/aalejandrofer/grubdrops/internal/platform" +) + +// Attempt claims one drop and returns its next state. The caller must +// reconcile afterwards: a lost response shows up as ClaimFailed here and is +// corrected when the platform reports isClaimed. +func Attempt(ctx context.Context, c platform.DropClaimer, s platform.Session, row dropstate.Row, dp platform.DropProgress, now time.Time) (dropstate.Row, platform.ClaimResult) { + res := c.ClaimDrop(ctx, s, dp) + switch res.Outcome { + case platform.ClaimOK, platform.ClaimAlready: + return dropstate.ClaimOK(row, now), res + case platform.ClaimNeedsLink: + return dropstate.ClaimNeedsLink(row, now), res + default: + return dropstate.ClaimFailedAttempt(row, now), res + } +} +``` + +- [ ] **Step 3: Run tests and commit** + +Run: `go test ./internal/pipeline/claimer/ -v` +Expected: PASS. + +```bash +gofmt -w internal/pipeline +git add internal/pipeline/claimer +git commit -m "feat(pipeline-v2): claimer maps typed claim outcomes to drop state + +Co-Authored-By: Claude Opus 5.5 (1M context) " +``` + +--- + +### Task 10: Session + +**Files:** +- Create: `internal/pipeline/session/session.go` +- Test: `internal/pipeline/session/session_test.go` + +**Interfaces:** +- Consumes: `platform.Backend` (`StartWatch`, `Heartbeat`, `StopWatch`, `InventoryProgress`). +- Produces: + +```go +type Kind int // Progress, StreamDown, Stalled +type Event struct { Kind Kind; Channel string; Progress []platform.Progress; Err error } +type Config struct { + Backend platform.Backend + Session platform.Session + Stream platform.Stream + Serves []string + Ticks <-chan time.Time // 60s ticker in production + StallPolls int // consecutive polls with no gain before Stalled +} +func Run(ctx context.Context, cfg Config, out chan<- Event) +``` + +- [ ] **Step 1: Failing tests** + +```go +package session + +import ( + "context" + "errors" + "sync" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/aalejandrofer/grubdrops/internal/platform" +) + +type fakeBackend struct { + platform.Backend + mu sync.Mutex + startErr error + beatErr error + minutes []int // minutes for d1 per poll; last value repeats + polls int + beats int + stopped bool +} + +func (f *fakeBackend) StartWatch(_ context.Context, _ platform.Session, s platform.Stream) (platform.WatchHandle, error) { + return platform.WatchHandle{Channel: s.Channel}, f.startErr +} +func (f *fakeBackend) Heartbeat(context.Context, platform.WatchHandle) error { + f.mu.Lock() + defer f.mu.Unlock() + f.beats++ + return f.beatErr +} +func (f *fakeBackend) StopWatch(context.Context, platform.WatchHandle) error { + f.mu.Lock() + defer f.mu.Unlock() + f.stopped = true + return nil +} +func (f *fakeBackend) InventoryProgress(context.Context, platform.Session) ([]platform.Progress, error) { + f.mu.Lock() + defer f.mu.Unlock() + i := f.polls + if i >= len(f.minutes) { + i = len(f.minutes) - 1 + } + f.polls++ + return []platform.Progress{{BenefitID: "d1", MinutesWatched: f.minutes[i]}}, nil +} + +func recv(t *testing.T, ch <-chan Event) Event { + t.Helper() + select { + case e := <-ch: + return e + case <-time.After(2 * time.Second): + t.Fatal("no event") + return Event{} + } +} + +func TestRun_ProgressEachTickAndStopOnCancel(t *testing.T) { + f := &fakeBackend{minutes: []int{1, 2}} + ticks := make(chan time.Time) + out := make(chan Event, 8) + ctx, cancel := context.WithCancel(context.Background()) + done := make(chan struct{}) + go func() { Run(ctx, Config{Backend: f, Stream: platform.Stream{Channel: "ch"}, Serves: []string{"d1"}, Ticks: ticks, StallPolls: 5}, out); close(done) }() + + e := recv(t, out) + assert.Equal(t, Progress, e.Kind, "first beat is immediate") + assert.Equal(t, 1, e.Progress[0].MinutesWatched) + ticks <- time.Now() + assert.Equal(t, 2, recv(t, out).Progress[0].MinutesWatched) + cancel() + <-done + assert.True(t, f.stopped) +} + +func TestRun_HeartbeatErrorIsStreamDown(t *testing.T) { + f := &fakeBackend{minutes: []int{1}, beatErr: errors.New("offline")} + out := make(chan Event, 8) + Run(context.Background(), Config{Backend: f, Stream: platform.Stream{Channel: "ch"}, Ticks: make(chan time.Time)}, out) + e := recv(t, out) + assert.Equal(t, StreamDown, e.Kind) + assert.Equal(t, "ch", e.Channel) + assert.True(t, f.stopped) +} + +func TestRun_StartErrorIsStreamDown(t *testing.T) { + f := &fakeBackend{startErr: errors.New("sidecar")} + out := make(chan Event, 8) + Run(context.Background(), Config{Backend: f, Stream: platform.Stream{Channel: "ch"}, Ticks: make(chan time.Time)}, out) + assert.Equal(t, StreamDown, recv(t, out).Kind) +} + +func TestRun_StalledOnceAfterNoGain(t *testing.T) { + f := &fakeBackend{minutes: []int{5}} + ticks := make(chan time.Time) + out := make(chan Event, 32) + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + go Run(ctx, Config{Backend: f, Stream: platform.Stream{Channel: "ch"}, Serves: []string{"d1"}, Ticks: ticks, StallPolls: 2}, out) + require.Equal(t, Progress, recv(t, out).Kind) // poll 1 sets baseline + ticks <- time.Now() + require.Equal(t, Progress, recv(t, out).Kind) // no gain 1 + ticks <- time.Now() + require.Equal(t, Progress, recv(t, out).Kind) // no gain 2 + assert.Equal(t, Stalled, recv(t, out).Kind) + ticks <- time.Now() + assert.Equal(t, Progress, recv(t, out).Kind) + select { + case e := <-out: + t.Fatalf("unexpected second event %v", e.Kind) + case <-time.After(50 * time.Millisecond): + } +} +``` + +Run: `go test ./internal/pipeline/session/ -v` +Expected: FAIL, `undefined: Run`. + +- [ ] **Step 2: Implement** + +```go +// Package session runs one accrual session on one channel and reports what +// it sees. It makes no decisions. +package session + +import ( + "context" + "time" + + "github.com/aalejandrofer/grubdrops/internal/platform" +) + +type Kind int + +const ( + Progress Kind = iota + StreamDown + Stalled +) + +type Event struct { + Kind Kind + Channel string + Progress []platform.Progress + Err error +} + +type Config struct { + Backend platform.Backend + Session platform.Session + Stream platform.Stream + Serves []string + // Ticks drives beats. Production passes a 60s ticker: Twitch credits one + // minute per beacon, so a slower cadence under-credits. + Ticks <-chan time.Time + StallPolls int +} + +// Run starts the watch, beats immediately and on every tick, and stops the +// watch when ctx ends or the stream goes down. +func Run(ctx context.Context, cfg Config, out chan<- Event) { + ch := cfg.Stream.Channel + send := func(e Event) { + e.Channel = ch + select { + case out <- e: + case <-ctx.Done(): + } + } + h, err := cfg.Backend.StartWatch(ctx, cfg.Session, cfg.Stream) + if err != nil { + send(Event{Kind: StreamDown, Err: err}) + return + } + defer func() { + stopCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + _ = cfg.Backend.StopWatch(stopCtx, h) + }() + + serves := make(map[string]bool, len(cfg.Serves)) + for _, id := range cfg.Serves { + serves[id] = true + } + best, still, stalled := -1, 0, false + beat := func() bool { + if err := cfg.Backend.Heartbeat(ctx, h); err != nil { + if ctx.Err() == nil { + send(Event{Kind: StreamDown, Err: err}) + } + return false + } + prog, err := cfg.Backend.InventoryProgress(ctx, cfg.Session) + if err != nil { + return true // transient; the next tick retries + } + send(Event{Kind: Progress, Progress: prog}) + total := 0 + for _, p := range prog { + if serves[p.BenefitID] { + total += p.MinutesWatched + } + } + if total > best { + best, still = total, 0 + } else { + still++ + } + if !stalled && cfg.StallPolls > 0 && still >= cfg.StallPolls { + stalled = true + send(Event{Kind: Stalled}) + } + return true + } + if !beat() { + return + } + for { + select { + case <-ctx.Done(): + return + case <-cfg.Ticks: + if !beat() { + return + } + } + } +} +``` + +- [ ] **Step 3: Run tests and commit** + +Run: `go test ./internal/pipeline/session/ -race -v` +Expected: PASS, no races. + +```bash +gofmt -w internal/pipeline +git add internal/pipeline/session +git commit -m "feat(pipeline-v2): accrual session reports progress, stream-down, stall + +Co-Authored-By: Claude Opus 5.5 (1M context) " +``` + +--- + +### Task 11: Account loop + +**Files:** +- Create: `internal/pipeline/loop/loop.go` +- Test: `internal/pipeline/loop/loop_test.go` + +**Interfaces:** +- Consumes: everything from Tasks 2, 5, 7 to 10; `watcher.Snapshot` (existing, for dashboard compatibility); `platform.PubSubAware`, `platform.PubSubHooks`, `platform.ChannelSubscriber` (existing). +- Produces: + +```go +type Notifier interface { Notify(ctx context.Context, event string, fields map[string]any) error } +type Store interface { + List(ctx context.Context, accountID string) ([]dropstate.Row, error) + Upsert(ctx context.Context, r dropstate.Row) error +} +type ClaimHistory interface { + RecordClaimIfNew(ctx context.Context, accountID string, b platform.DropBenefit) (bool, error) + ClaimedBenefitIDs(ctx context.Context, accountID string) (map[string]bool, error) +} +type Config struct { ... see code ... } +func New(cfg Config) (*Loop, error) +func (l *Loop) Run(ctx context.Context) error +func (l *Loop) Snapshot() watcher.Snapshot +func (l *Loop) LastDiscovery() ([]platform.Campaign, time.Time) +func (l *Loop) AllowGame() func(game string) bool +func (l *Loop) Nudge() +``` + +`*store.DropStateStore` satisfies `Store`; `*store.ClaimRecorder` satisfies `ClaimHistory`; `*store.CampaignPersister` satisfies `reconcile.CampaignPersister`. + +- [ ] **Step 1: Failing tests** + +```go +package loop + +import ( + "context" + "strings" + "sync" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/aalejandrofer/grubdrops/internal/pipeline/dropstate" + "github.com/aalejandrofer/grubdrops/internal/platform" +) + +// fakeBackend: one game "G", campaign c1 with drop d1 (60 min). Tests +// mutate the fields under mu to script the platform. +type fakeBackend struct { + platform.Backend + mu sync.Mutex + camps []platform.Campaign + progress []platform.DropProgress + inventory []platform.Progress + live []platform.Stream + claimRes platform.ClaimResult + claims int + watching []string + stops int + beatFailOn string +} + +func (f *fakeBackend) ListActiveCampaigns(context.Context, platform.Session) ([]platform.Campaign, error) { + f.mu.Lock() + defer f.mu.Unlock() + return f.camps, nil +} +func (f *fakeBackend) DropProgress(context.Context, platform.Session, []platform.Campaign) ([]platform.DropProgress, error) { + f.mu.Lock() + defer f.mu.Unlock() + return append([]platform.DropProgress(nil), f.progress...), nil +} +func (f *fakeBackend) ListEligibleChannels(context.Context, platform.Session, platform.Campaign) ([]platform.Stream, error) { + f.mu.Lock() + defer f.mu.Unlock() + return append([]platform.Stream(nil), f.live...), nil +} +func (f *fakeBackend) InventoryProgress(context.Context, platform.Session) ([]platform.Progress, error) { + f.mu.Lock() + defer f.mu.Unlock() + return append([]platform.Progress(nil), f.inventory...), nil +} +func (f *fakeBackend) StartWatch(_ context.Context, _ platform.Session, s platform.Stream) (platform.WatchHandle, error) { + f.mu.Lock() + defer f.mu.Unlock() + f.watching = append(f.watching, s.Channel) + return platform.WatchHandle{Channel: s.Channel}, nil +} +func (f *fakeBackend) Heartbeat(_ context.Context, h platform.WatchHandle) error { + f.mu.Lock() + defer f.mu.Unlock() + if f.beatFailOn != "" && strings.EqualFold(h.Channel, f.beatFailOn) { + return context.DeadlineExceeded + } + return nil +} +func (f *fakeBackend) StopWatch(context.Context, platform.WatchHandle) error { + f.mu.Lock() + defer f.mu.Unlock() + f.stops++ + return nil +} +func (f *fakeBackend) ClaimDrop(context.Context, platform.Session, platform.DropProgress) platform.ClaimResult { + f.mu.Lock() + defer f.mu.Unlock() + f.claims++ + return f.claimRes +} + +type memStore struct { + mu sync.Mutex + rows map[string]dropstate.Row +} + +func (m *memStore) List(context.Context, string) ([]dropstate.Row, error) { + m.mu.Lock() + defer m.mu.Unlock() + var out []dropstate.Row + for _, r := range m.rows { + out = append(out, r) + } + return out, nil +} +func (m *memStore) Upsert(_ context.Context, r dropstate.Row) error { + m.mu.Lock() + defer m.mu.Unlock() + m.rows[r.DropID] = r + return nil +} +func (m *memStore) get(id string) dropstate.Row { + m.mu.Lock() + defer m.mu.Unlock() + return m.rows[id] +} + +type memHistory struct { + mu sync.Mutex + recorded []string + claimed map[string]bool +} + +func (h *memHistory) RecordClaimIfNew(_ context.Context, _ string, b platform.DropBenefit) (bool, error) { + h.mu.Lock() + defer h.mu.Unlock() + h.recorded = append(h.recorded, b.ID) + return true, nil +} +func (h *memHistory) ClaimedBenefitIDs(context.Context, string) (map[string]bool, error) { + return h.claimed, nil +} + +func setup(t *testing.T) (*fakeBackend, *memStore, *memHistory, Config) { + f := &fakeBackend{ + camps: []platform.Campaign{{ID: "c1", Platform: "twitch", Game: "G", Name: "Camp", Status: "active", + AccountLinked: true, AccountLinkChecked: true, + Benefits: []platform.DropBenefit{{ID: "d1", CampaignID: "c1", Name: "Drop", RequiredMinutes: 60}}}}, + progress: []platform.DropProgress{{DropID: "d1", CampaignID: "c1", Minutes: 10, Required: 60, Known: true}}, + inventory: []platform.Progress{{BenefitID: "d1", MinutesWatched: 10}}, + live: []platform.Stream{{Channel: "ch1", ViewerCount: 10}, {Channel: "ch2", ViewerCount: 5}}, + claimRes: platform.ClaimResult{Outcome: platform.ClaimOK}, + } + st := &memStore{rows: map[string]dropstate.Row{}} + h := &memHistory{} + cfg := Config{ + AccountID: "acc", Platform: "twitch", Backend: f, Store: st, History: h, + AllowGame: func(g string) bool { return g == "G" }, + ReconcileEvery: time.Hour, LiveEvery: time.Hour, + BeatEvery: 10 * time.Millisecond, StallPolls: 1000, // tests opt into stalls + } + return f, st, h, cfg +} + +func run(t *testing.T, cfg Config) (*Loop, context.CancelFunc) { + l, err := New(cfg) + require.NoError(t, err) + ctx, cancel := context.WithCancel(context.Background()) + done := make(chan struct{}) + go func() { _ = l.Run(ctx); close(done) }() + t.Cleanup(func() { cancel(); <-done }) + return l, cancel +} + +func TestLoop_ClaimedOnWebsiteMidWatch(t *testing.T) { + f, st, h, cfg := setup(t) + l, _ := run(t, cfg) + require.Eventually(t, func() bool { return l.Snapshot().Channel == "ch1" }, 2*time.Second, 5*time.Millisecond) + + f.mu.Lock() + f.inventory = []platform.Progress{{BenefitID: "d1", MinutesWatched: 20, Claimed: true}} + f.mu.Unlock() + + require.Eventually(t, func() bool { return st.get("d1").Status == dropstate.Claimed }, 2*time.Second, 5*time.Millisecond) + require.Eventually(t, func() bool { return l.Snapshot().State == "idle" }, 2*time.Second, 5*time.Millisecond) + h.mu.Lock() + assert.Contains(t, h.recorded, "d1") + h.mu.Unlock() + f.mu.Lock() + assert.Equal(t, 0, f.claims, "no claim call for an already-claimed drop") + assert.GreaterOrEqual(t, f.stops, 1) + f.mu.Unlock() +} + +func TestLoop_ClaimsWhenClaimable(t *testing.T) { + f, st, _, cfg := setup(t) + f.progress[0].Minutes = 60 + f.inventory[0].MinutesWatched = 60 + run(t, cfg) + require.Eventually(t, func() bool { return st.get("d1").Status == dropstate.Claimed }, 2*time.Second, 5*time.Millisecond) + f.mu.Lock() + assert.Equal(t, 1, f.claims) + f.mu.Unlock() +} + +func TestLoop_NeedsLinkBlocksWithoutLooping(t *testing.T) { + f, st, _, cfg := setup(t) + f.progress[0].Minutes = 60 + f.inventory[0].MinutesWatched = 60 + f.claimRes = platform.ClaimResult{Outcome: platform.ClaimNeedsLink} + run(t, cfg) + require.Eventually(t, func() bool { return st.get("d1").Reason == dropstate.NeedsLink }, 2*time.Second, 5*time.Millisecond) + time.Sleep(100 * time.Millisecond) + f.mu.Lock() + assert.Equal(t, 1, f.claims, "needs_link is not retried") + f.mu.Unlock() +} + +func TestLoop_StallCoolsChannelAndSwaps(t *testing.T) { + _, _, _, cfg := setup(t) + cfg.StallPolls = 3 + l, _ := run(t, cfg) + // Minutes never move, so after StallPolls the loop cools ch1 and moves on. + require.Eventually(t, func() bool { return l.Snapshot().Channel == "ch2" }, 3*time.Second, 5*time.Millisecond) +} + +func TestLoop_StaleStreamDownIgnored(t *testing.T) { + f, _, _, cfg := setup(t) + l, err := New(cfg) + require.NoError(t, err) + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + // Drive the loop manually: load, reconcile, refresh, plan. + require.NoError(t, l.load(ctx)) + l.reconcile(ctx) + l.refreshLive(ctx) + l.replan(ctx) + require.Equal(t, "ch1", l.current.Channel.Channel) + l.onSession(ctx, sessionEvent("ch-old-down")) + assert.Empty(t, l.cooldowns, "event from another channel must not cool anything") + l.stopSession() + _ = f +} + +func TestLoop_BridgesManualMarksOnLoad(t *testing.T) { + _, st, h, cfg := setup(t) + st.rows["d1"] = dropstate.Row{AccountID: "acc", DropID: "d1", CampaignID: "c1", Platform: "twitch", Status: dropstate.Accruing, Minutes: 10, Required: 60, Source: dropstate.FromPlatform} + h.claimed = map[string]bool{"d1": true} + l, err := New(cfg) + require.NoError(t, err) + require.NoError(t, l.load(context.Background())) + assert.Equal(t, dropstate.Claimed, st.get("d1").Status) + assert.Equal(t, dropstate.FromUser, st.get("d1").Source) +} + +func TestNew_RejectsBackendWithoutCapabilities(t *testing.T) { + type bare struct{ platform.Backend } + _, err := New(Config{AccountID: "a", Backend: bare{}}) + require.Error(t, err) +} +``` + +Add this helper at the bottom of the test file (it builds a StreamDown for a given channel): + +```go +func sessionEvent(ch string) session.Event { return session.Event{Kind: session.StreamDown, Channel: ch} } +``` + +and add `"github.com/aalejandrofer/grubdrops/internal/pipeline/session"` to the test imports. + +Run: `go test ./internal/pipeline/loop/ -v` +Expected: FAIL, `undefined: New`. + +- [ ] **Step 2: Implement** + +```go +// Package loop is the per-account pipeline v2 runner. One goroutine owns +// every drop state transition: it reconciles, plans, runs one session at a +// time, and claims. +package loop + +import ( + "context" + "fmt" + "log/slog" + "strings" + "sync" + "time" + + "github.com/aalejandrofer/grubdrops/internal/pipeline/claimer" + "github.com/aalejandrofer/grubdrops/internal/pipeline/dropstate" + "github.com/aalejandrofer/grubdrops/internal/pipeline/planner" + "github.com/aalejandrofer/grubdrops/internal/pipeline/reconcile" + "github.com/aalejandrofer/grubdrops/internal/pipeline/session" + "github.com/aalejandrofer/grubdrops/internal/platform" + "github.com/aalejandrofer/grubdrops/internal/watcher" +) + +type Notifier interface { + Notify(ctx context.Context, event string, fields map[string]any) error +} + +type Store interface { + List(ctx context.Context, accountID string) ([]dropstate.Row, error) + Upsert(ctx context.Context, r dropstate.Row) error +} + +type ClaimHistory interface { + RecordClaimIfNew(ctx context.Context, accountID string, b platform.DropBenefit) (bool, error) + ClaimedBenefitIDs(ctx context.Context, accountID string) (map[string]bool, error) +} + +type Config struct { + AccountID string + AccountLabel string + Platform string + Backend platform.Backend + Session platform.Session + Store Store + History ClaimHistory + Persister reconcile.CampaignPersister + Notifier Notifier + + AllowGame func(game string) bool + AllowChannel func(channels []string) bool + GameRank func(game string) int + PriorityMode string + ForceLinked func(campaignID string) bool + StreamerPriority []string + ForceWatch []string + ProgressNotifyStepPct int + + Now func() time.Time + ReconcileEvery time.Duration // default 15m + LiveEvery time.Duration // default 5m + BeatEvery time.Duration // default 60s; production never overrides (Twitch credit gate) + StallPolls int // default 5 + StallCooldown time.Duration // default 30m + DownCooldown time.Duration // default 2m +} + +// maxLiveCampaigns bounds channel lookups per refresh. +const maxLiveCampaigns = 8 + +type pubsubEvent struct { + kind string // "progress" | "claimable" | "down" + drop string + instance string + minutes int + required int + channel string +} + +type Loop struct { + cfg Config + prog platform.DropProgressSource + claimer platform.DropClaimer + prober platform.ChannelProber + subs platform.ChannelSubscriber + + nudge chan struct{} + pubsub chan pubsubEvent + sessEvents chan session.Event + + mu sync.Mutex + snap watcher.Snapshot + discovery []platform.Campaign + discoverAt time.Time + + // Owned by the Run goroutine. + rows map[string]dropstate.Row + camps []platform.Campaign + progress map[string]platform.DropProgress + live map[string][]platform.Stream + forceLive []platform.Stream + cooldowns map[string]time.Time + current *planner.Decision + lastSwap time.Time + sessCancel context.CancelFunc + subscribed string + milestones map[string]int +} + +// New assembles the loop, checking the backend's pipeline capabilities once. +func New(cfg Config) (*Loop, error) { + prog, ok := cfg.Backend.(platform.DropProgressSource) + if !ok { + return nil, fmt.Errorf("backend %T has no DropProgress: pipeline v2 unsupported", cfg.Backend) + } + cl, ok := cfg.Backend.(platform.DropClaimer) + if !ok { + return nil, fmt.Errorf("backend %T has no ClaimDrop: pipeline v2 unsupported", cfg.Backend) + } + if cfg.Now == nil { + cfg.Now = time.Now + } + if cfg.ReconcileEvery <= 0 { + cfg.ReconcileEvery = 15 * time.Minute + } + if cfg.LiveEvery <= 0 { + cfg.LiveEvery = 5 * time.Minute + } + if cfg.BeatEvery <= 0 { + cfg.BeatEvery = 60 * time.Second + } + if cfg.StallPolls <= 0 { + cfg.StallPolls = 5 + } + if cfg.StallCooldown <= 0 { + cfg.StallCooldown = 30 * time.Minute + } + if cfg.DownCooldown <= 0 { + cfg.DownCooldown = 2 * time.Minute + } + cfg.Session.AccountID = cfg.AccountID + if cfg.AllowGame != nil { + cfg.Session.GameFilter = cfg.AllowGame + } + l := &Loop{ + cfg: cfg, prog: prog, claimer: cl, + nudge: make(chan struct{}, 1), + pubsub: make(chan pubsubEvent, 64), + sessEvents: make(chan session.Event, 64), + rows: map[string]dropstate.Row{}, + progress: map[string]platform.DropProgress{}, + live: map[string][]platform.Stream{}, + cooldowns: map[string]time.Time{}, + milestones: map[string]int{}, + } + l.prober, _ = cfg.Backend.(platform.ChannelProber) + l.subs, _ = cfg.Backend.(platform.ChannelSubscriber) + l.snap = watcher.Snapshot{AccountID: cfg.AccountID, State: "pick_campaign"} + if ps, ok := cfg.Backend.(platform.PubSubAware); ok { + ps.SetAccountPubSubHooks(cfg.AccountID, l.hooks()) + } + return l, nil +} + +func (l *Loop) hooks() platform.PubSubHooks { + push := func(e pubsubEvent) { + select { + case l.pubsub <- e: + default: // never block the PubSub reader + } + } + return platform.PubSubHooks{ + OnDropProgress: func(dropID string, cur, req int64) { + push(pubsubEvent{kind: "progress", drop: dropID, minutes: int(cur), required: int(req)}) + }, + OnDropClaim: func(dropID, instanceID string) { + push(pubsubEvent{kind: "claimable", drop: dropID, instance: instanceID}) + }, + OnStreamDown: func(channelID string) { + push(pubsubEvent{kind: "down", channel: channelID}) + }, + } +} + +// Nudge asks the loop to reconcile and refresh channels now. +func (l *Loop) Nudge() { + select { + case l.nudge <- struct{}{}: + default: + } +} + +func (l *Loop) Snapshot() watcher.Snapshot { + l.mu.Lock() + defer l.mu.Unlock() + return l.snap +} + +func (l *Loop) LastDiscovery() ([]platform.Campaign, time.Time) { + l.mu.Lock() + defer l.mu.Unlock() + return append([]platform.Campaign(nil), l.discovery...), l.discoverAt +} + +func (l *Loop) AllowGame() func(game string) bool { return l.cfg.AllowGame } + +func (l *Loop) Run(ctx context.Context) error { + if err := l.load(ctx); err != nil { + return err + } + l.reconcile(ctx) + l.refreshLive(ctx) + l.claimReady(ctx) + l.replan(ctx) + recT := time.NewTicker(l.cfg.ReconcileEvery) + liveT := time.NewTicker(l.cfg.LiveEvery) + defer recT.Stop() + defer liveT.Stop() + defer l.stopSession() + for { + select { + case <-ctx.Done(): + return ctx.Err() + case <-recT.C: + l.reconcile(ctx) + case <-liveT.C: + l.refreshLive(ctx) + case <-l.nudge: + l.reconcile(ctx) + l.refreshLive(ctx) + case ev := <-l.sessEvents: + l.onSession(ctx, ev) + case pe := <-l.pubsub: + l.onPubSub(ctx, pe) + } + l.claimReady(ctx) + l.replan(ctx) + } +} + +// load reads stored state, then bridges claim rows written by the v1-era +// mark-collected UI (that handler reloads the scheduler, so this runs). +func (l *Loop) load(ctx context.Context) error { + rows, err := l.cfg.Store.List(ctx, l.cfg.AccountID) + if err != nil { + return fmt.Errorf("load drop state: %w", err) + } + for _, r := range rows { + l.rows[r.DropID] = r + } + if l.cfg.History == nil { + return nil + } + ids, err := l.cfg.History.ClaimedBenefitIDs(ctx, l.cfg.AccountID) + if err != nil { + slog.Warn("pipeline: read claim history failed", "kind", "error", "account", l.cfg.AccountID, "err", err) + return nil + } + now := l.cfg.Now() + for id := range ids { + if r, ok := l.rows[id]; ok && r.Status != dropstate.Claimed { + next := dropstate.MarkCollected(r, now) + l.rows[id] = next + if err := l.cfg.Store.Upsert(ctx, next); err != nil { + slog.Warn("pipeline: persist bridged mark failed", "kind", "error", "account", l.cfg.AccountID, "drop", id, "err", err) + } + } + } + return nil +} + +func (l *Loop) reconcile(ctx context.Context) { + res, err := reconcile.Run(ctx, reconcile.Config{ + AccountID: l.cfg.AccountID, Platform: l.cfg.Platform, + Backend: l.cfg.Backend, Progress: l.prog, Session: l.cfg.Session, + AllowGame: l.cfg.AllowGame, AllowChannel: l.cfg.AllowChannel, + ForceLinked: l.cfg.ForceLinked, Persister: l.cfg.Persister, + }, l.rows, l.cfg.Now()) + if err != nil { + slog.Warn("pipeline reconcile failed; keeping last state", "kind", "error", "account", l.cfg.AccountID, "err", err) + return + } + l.camps = res.Campaigns + for id, dp := range res.Progress { + l.progress[id] = dp + } + l.mu.Lock() + l.discovery, l.discoverAt = res.Campaigns, l.cfg.Now() + l.mu.Unlock() + for _, r := range res.Rows { + l.commit(ctx, r) + } +} + +func (l *Loop) benefit(dropID string) (platform.DropBenefit, platform.Campaign) { + for _, c := range l.camps { + for _, b := range c.Benefits { + if b.ID == dropID { + return b, c + } + } + } + return platform.DropBenefit{ID: dropID}, platform.Campaign{} +} + +// commit stores a row and handles the transition into claimed. +func (l *Loop) commit(ctx context.Context, next dropstate.Row) { + prev := l.rows[next.DropID] + l.rows[next.DropID] = next + if err := l.cfg.Store.Upsert(ctx, next); err != nil { + slog.Warn("pipeline: persist drop state failed", "kind", "error", "account", l.cfg.AccountID, "drop", next.DropID, "err", err) + } + if next.Status != dropstate.Claimed || prev.Status == dropstate.Claimed { + return + } + b, c := l.benefit(next.DropID) + if l.cfg.History != nil { + if _, err := l.cfg.History.RecordClaimIfNew(ctx, l.cfg.AccountID, b); err != nil { + slog.Warn("pipeline: record claim history failed", "kind", "error", "account", l.cfg.AccountID, "drop", next.DropID, "err", err) + } + } + slog.Info("pipeline drop claimed", "kind", "claim", "account", l.cfg.AccountID, "drop", next.DropID, "source", string(next.Source)) + // Only notify live transitions, not historical claims seen on first sync. + if !prev.IsZero() && next.Source == dropstate.FromPlatform { + l.notify(ctx, "claim", c, b, nil) + } +} + +func (l *Loop) notify(ctx context.Context, event string, c platform.Campaign, b platform.DropBenefit, extra map[string]any) { + if l.cfg.Notifier == nil { + return + } + f := map[string]any{"account": l.cfg.AccountID} + if l.cfg.AccountLabel != "" { + f["account_label"] = l.cfg.AccountLabel + } + if l.cfg.Platform != "" { + f["platform"] = l.cfg.Platform + } + if c.Game != "" { + f["game"] = c.Game + } + if c.Name != "" { + f["campaign"] = c.Name + } + if b.Name != "" { + f["drop"] = b.Name + } + for k, v := range extra { + f[k] = v + } + _ = l.cfg.Notifier.Notify(ctx, event, f) +} + +func (l *Loop) plannerInput() planner.Input { + return planner.Input{ + Now: l.cfg.Now(), Rows: l.rows, Campaigns: l.camps, + Settings: planner.Settings{ + GameRank: l.cfg.GameRank, PriorityMode: l.cfg.PriorityMode, + StreamerPriority: l.cfg.StreamerPriority, ForceWatch: l.forceLive, + }, + Live: l.live, Cooldowns: l.cooldowns, Current: l.current, LastSwap: l.lastSwap, + } +} + +// allowedPriority returns the priority logins a campaign accepts. +func allowedPriority(c platform.Campaign, prio []string) []string { + if len(c.AllowedChannels) == 0 { + return prio + } + allowed := map[string]bool{} + for _, a := range c.AllowedChannels { + allowed[strings.ToLower(a)] = true + } + var out []string + for _, p := range prio { + if allowed[strings.ToLower(p)] { + out = append(out, p) + } + } + return out +} + +func mergeStreams(first, rest []platform.Stream) []platform.Stream { + seen := map[string]bool{} + var out []platform.Stream + for _, s := range append(append([]platform.Stream(nil), first...), rest...) { + k := strings.ToLower(s.Channel) + if !seen[k] { + seen[k] = true + out = append(out, s) + } + } + return out +} + +func (l *Loop) refreshLive(ctx context.Context) { + live := map[string][]platform.Stream{} + for i, c := range planner.Candidates(l.plannerInput()) { + if i >= maxLiveCampaigns { + break + } + streams, err := l.cfg.Backend.ListEligibleChannels(ctx, l.cfg.Session, c) + if err != nil { + slog.Debug("pipeline: list channels failed", "account", l.cfg.AccountID, "campaign", c.ID, "err", err) + } + if l.prober != nil { + if logins := allowedPriority(c, l.cfg.StreamerPriority); len(logins) > 0 { + if ps, err := l.prober.ProbeChannels(ctx, l.cfg.Session, c, logins); err == nil { + streams = mergeStreams(ps, streams) + } + } + } + live[c.ID] = streams + } + l.live = live + l.forceLive = nil + if l.prober != nil && len(l.cfg.ForceWatch) > 0 { + if ps, err := l.prober.ProbeChannels(ctx, l.cfg.Session, platform.Campaign{}, l.cfg.ForceWatch); err == nil { + l.forceLive = ps + } + } +} + +func (l *Loop) isCurrent(channel string) bool { + return l.current != nil && l.current.Kind != planner.Idle && strings.EqualFold(l.current.Channel.Channel, channel) +} + +func (l *Loop) onSession(ctx context.Context, ev session.Event) { + now := l.cfg.Now() + switch ev.Kind { + case session.Progress: + l.applyProgress(ctx, ev.Progress) + case session.StreamDown: + if l.isCurrent(ev.Channel) { + l.cooldowns[strings.ToLower(ev.Channel)] = now.Add(l.cfg.DownCooldown) + } + case session.Stalled: + if l.isCurrent(ev.Channel) { + l.cooldowns[strings.ToLower(ev.Channel)] = now.Add(l.cfg.StallCooldown) + l.reconcile(ctx) + } + } +} + +func (l *Loop) applyProgress(ctx context.Context, progs []platform.Progress) { + now := l.cfg.Now() + for _, p := range progs { + prev, ok := l.rows[p.BenefitID] + if !ok { + continue // reconcile creates the row first + } + if p.InstanceID != "" { + dp := l.progress[p.BenefitID] + dp.DropID, dp.CampaignID, dp.InstanceID = p.BenefitID, prev.CampaignID, p.InstanceID + l.progress[p.BenefitID] = dp + } + next := dropstate.Apply(prev, dropstate.Observation{ + Known: true, Claimed: p.Claimed, Minutes: p.MinutesWatched, Required: prev.Required, + }, now) + l.maybeNotifyProgress(ctx, next) + l.commit(ctx, next) + } +} + +func (l *Loop) onPubSub(ctx context.Context, e pubsubEvent) { + switch e.kind { + case "progress": + if r, ok := l.rows[e.drop]; ok && r.Required == 0 && e.required > 0 { + r.Required = e.required + l.rows[e.drop] = r + } + l.applyProgress(ctx, []platform.Progress{{BenefitID: e.drop, MinutesWatched: e.minutes}}) + case "claimable": + r, ok := l.rows[e.drop] + if !ok || r.Status == dropstate.Claimed { + return + } + dp := l.progress[e.drop] + dp.DropID, dp.CampaignID, dp.InstanceID = e.drop, r.CampaignID, e.instance + l.progress[e.drop] = dp + minutes := r.Minutes + if minutes < r.Required { + minutes = r.Required + } + l.commit(ctx, dropstate.Apply(r, dropstate.Observation{Known: true, Minutes: minutes, Required: r.Required}, l.cfg.Now())) + case "down": + if l.current != nil && l.current.Channel.ChannelID != "" && l.current.Channel.ChannelID == e.channel { + l.cooldowns[strings.ToLower(l.current.Channel.Channel)] = l.cfg.Now().Add(l.cfg.DownCooldown) + } + } +} + +func (l *Loop) maybeNotifyProgress(ctx context.Context, r dropstate.Row) { + step := l.cfg.ProgressNotifyStepPct + if step <= 0 || r.Required <= 0 || l.cfg.Notifier == nil { + return + } + pct := r.Minutes * 100 / r.Required + if pct > 100 { + pct = 100 + } + m := pct / step * step + last, seen := l.milestones[r.DropID] + l.milestones[r.DropID] = m + if !seen || m <= last { + return // first sight records a baseline; no restart storms + } + b, c := l.benefit(r.DropID) + l.notify(ctx, "progress", c, b, map[string]any{"cur_min": r.Minutes, "req_min": r.Required}) +} + +func (l *Loop) claimReady(ctx context.Context) { + now := l.cfg.Now() + attempted := false + for id, r := range l.rows { + if !dropstate.ReadyToClaim(r, now) { + continue + } + dp, ok := l.progress[id] + if !ok || dp.DropID == "" { + dp = platform.DropProgress{DropID: id, CampaignID: r.CampaignID} + } + next, res := claimer.Attempt(ctx, l.claimer, l.cfg.Session, r, dp, now) + slog.Info("pipeline claim attempt", "kind", "claim", "account", l.cfg.AccountID, "drop", id, + "outcome", res.Outcome.String(), "detail", res.Detail, "link", res.LinkURL) + l.commit(ctx, next) + attempted = true + } + if attempted { + l.reconcile(ctx) // corrects lost responses and auto-granted rewards + } +} + +func (l *Loop) replan(ctx context.Context) { + d := planner.Plan(l.plannerInput()) + if l.current != nil && l.current.Same(d) { + l.current.Serves, l.current.Reason = d.Serves, d.Reason + l.updateSnapshot() + return + } + l.stopSession() + l.current, l.lastSwap = &d, l.cfg.Now() + slog.Info("pipeline decision", "kind", "state", "account", l.cfg.AccountID, + "decision", d.Kind.String(), "channel", d.Channel.Channel, "serves", len(d.Serves), "reason", d.Reason) + if l.cfg.Notifier != nil { + _ = l.cfg.Notifier.Notify(ctx, "state", map[string]any{"account": l.cfg.AccountID, "state": stateString(&d)}) + } + if d.Kind != planner.Idle { + l.startSession(ctx, d) + } + l.updateSnapshot() +} + +func (l *Loop) startSession(ctx context.Context, d planner.Decision) { + sctx, cancel := context.WithCancel(ctx) + l.sessCancel = cancel + if l.subs != nil && d.Channel.ChannelID != "" { + l.subs.SubscribeChannel(l.cfg.AccountID, d.Channel.ChannelID) + l.subscribed = d.Channel.ChannelID + } + ticker := time.NewTicker(l.cfg.BeatEvery) + cfg := session.Config{ + Backend: l.cfg.Backend, Session: l.cfg.Session, Stream: d.Channel, + Serves: d.Serves, Ticks: ticker.C, StallPolls: l.cfg.StallPolls, + } + if d.Kind == planner.ForceWatch { + cfg.StallPolls = 0 // channel-points farming has no drop progress to stall on + } + go func() { + defer ticker.Stop() + session.Run(sctx, cfg, l.sessEvents) + }() +} + +func (l *Loop) stopSession() { + if l.sessCancel != nil { + l.sessCancel() + l.sessCancel = nil + } + if l.subs != nil && l.subscribed != "" { + l.subs.UnsubscribeChannel(l.cfg.AccountID, l.subscribed) + l.subscribed = "" + } +} + +// stateString maps a decision onto the v1 dashboard state vocabulary. +func stateString(d *planner.Decision) string { + if d == nil { + return "pick_campaign" + } + switch d.Kind { + case planner.Mine: + return "watching" + case planner.ForceWatch: + return "force_watch" + default: + return "idle" + } +} + +func (l *Loop) updateSnapshot() { + s := watcher.Snapshot{AccountID: l.cfg.AccountID, State: stateString(l.current)} + if d := l.current; d != nil && d.Kind != planner.Idle { + s.Channel, s.ViewerCount, s.StartedAt = d.Channel.Channel, d.Channel.ViewerCount, l.lastSwap + if len(d.Serves) > 0 { + b, c := l.benefit(d.Serves[0]) + r := l.rows[d.Serves[0]] + s.CampaignID, s.CampaignName, s.CampaignGame = c.ID, c.Name, c.Game + s.BenefitID, s.BenefitName, s.BenefitImage = b.ID, b.Name, b.ImageURL + s.MinutesWatched, s.RequiredMinutes = r.Minutes, r.Required + } + } + l.mu.Lock() + l.snap = s + l.mu.Unlock() +} +``` + +- [ ] **Step 3: Run tests with race detector** + +Run: `go test ./internal/pipeline/loop/ -race -v` +Expected: PASS, no races. If `TestLoop_StallCoolsChannelAndSwaps` flakes, raise its timeout, not the stall logic. + +- [ ] **Step 4: Revert-proof checks** + +1. In `onSession`, remove the `l.isCurrent(ev.Channel)` guard for StreamDown: `TestLoop_StaleStreamDownIgnored` must FAIL. Restore. +2. In `claimReady`, remove `l.reconcile(ctx)`: run whole package; confirm tests still pass (reconcile is belt-and-braces there) and note it in the task report. +3. In `commit`, delete the `RecordClaimIfNew` call: `TestLoop_ClaimedOnWebsiteMidWatch` must FAIL. Restore. + +- [ ] **Step 5: Commit** + +```bash +gofmt -w internal/pipeline +git add internal/pipeline/loop +git commit -m "feat(pipeline-v2): per-account loop owning all drop state transitions + +Co-Authored-By: Claude Opus 5.5 (1M context) " +``` + +--- + +### Task 12: Scheduler interfaces + `GRUB_PIPELINE` wiring + +**Files:** +- Modify: `internal/scheduler/state.go:30-62` (use interfaces, not `*watcher.Watcher`) +- Modify: `internal/scheduler/discovery.go:25-45` (same) +- Test: `internal/scheduler/v2_runner_test.go` +- Modify: `internal/store/campaign_persister.go` (add `PipelineOverridePrefix`) +- Modify: `cmd/miner/main.go` (backfill at startup, `dropStore`, v2 branch in `build`, helpers) +- Test: `cmd/miner/main_test.go` (helpers) + +**Interfaces:** +- Consumes: `loop.New`, `loop.Config` (Task 11), `store.DropStateStore`, `store.BackfillDropState` (Tasks 3 and 4). +- Produces: `pipelineModeFor(ctx, q, accountID) string`, `matchAnyChannel(logins []string) func([]string) bool` in `cmd/miner`; `store.PipelineOverridePrefix = "pipeline_override:"`. + +- [ ] **Step 1: Failing scheduler test** + +First read `internal/scheduler/*_test.go` for how entries are added and snapshots read (existing tests use `NewEntry` + `AddEntry`). Then: + +```go +package scheduler + +import ( + "context" + "testing" + "time" + + "github.com/stretchr/testify/assert" + + "github.com/aalejandrofer/grubdrops/internal/platform" + "github.com/aalejandrofer/grubdrops/internal/watcher" +) + +type v2Runner struct{} + +func (v2Runner) Run(ctx context.Context) error { <-ctx.Done(); return ctx.Err() } +func (v2Runner) Snapshot() watcher.Snapshot { + return watcher.Snapshot{AccountID: "acc", State: "watching", Channel: "ch1"} +} +func (v2Runner) LastDiscovery() ([]platform.Campaign, time.Time) { + return []platform.Campaign{{ID: "c1"}}, time.Unix(1, 0) +} +func (v2Runner) AllowGame() func(string) bool { return func(string) bool { return true } } + +func TestScheduler_NonWatcherRunnerWithSnapshot(t *testing.T) { + s := New(Options{}) + s.AddEntry(NewEntry("acc", v2Runner{})) + snaps := s.WatcherSnapshots() + assert.Equal(t, "ch1", snaps[0].Channel) + assert.Equal(t, "watching", s.Snapshot()[0].State, "must not fall back to needs_auth") + disc := s.WatcherDiscoveries() + assert.Len(t, disc[0].Campaigns, 1) +} +``` + +Run: `go test ./internal/scheduler/ -run NonWatcherRunner -v` +Expected: FAIL (`needs_auth` / empty channel). + +- [ ] **Step 2: Implement scheduler interfaces** + +In `state.go` add: + +```go +// snapshotter is any runner that can describe its current work. Both the v1 +// *watcher.Watcher and the pipeline v2 loop implement it. +type snapshotter interface { + Snapshot() watcher.Snapshot +} +``` + +Replace the body of `Snapshot()`'s loop with: + +```go + for _, e := range s.entries { + if sn, ok := e.runner.(snapshotter); ok { + out = append(out, AccountState{AccountID: e.id, State: sn.Snapshot().State}) + continue + } + out = append(out, AccountState{AccountID: e.id, State: idleState(e.runner)}) + } +``` + +and `WatcherSnapshots()`'s loop with: + +```go + for _, e := range s.entries { + if sn, ok := e.runner.(snapshotter); ok { + out = append(out, sn.Snapshot()) + continue + } + out = append(out, watcher.Snapshot{AccountID: e.id, State: idleState(e.runner)}) + } +``` + +In `discovery.go` add (import `time` and `platform` as needed): + +```go +// discoveryReporter is any runner that caches its last campaign discovery. +type discoveryReporter interface { + LastDiscovery() ([]platform.Campaign, time.Time) + AllowGame() func(game string) bool +} +``` + +and in `WatcherDiscoveries` replace `w, ok := e.runner.(*watcher.Watcher)` with `w, ok := e.runner.(discoveryReporter)`. Remove the `watcher` import from `discovery.go` if now unused. + +Run: `go test ./internal/scheduler/ -v` +Expected: PASS (new + existing). + +- [ ] **Step 3: Failing main helper tests** (append to `cmd/miner/main_test.go`) + +```go +func TestPipelineModeFor(t *testing.T) { + ctx := context.Background() + db, err := store.Open(ctx, t.TempDir()+"/t.db") + require.NoError(t, err) + t.Cleanup(func() { _ = db.Close() }) + q := gen.New(db) + + t.Setenv("GRUB_PIPELINE", "") + assert.Equal(t, "v1", pipelineModeFor(ctx, q, "acc")) + t.Setenv("GRUB_PIPELINE", "v2") + assert.Equal(t, "v2", pipelineModeFor(ctx, q, "acc")) + require.NoError(t, q.UpsertSettingString(ctx, gen.UpsertSettingStringParams{Key: store.PipelineOverridePrefix + "acc", Value: []byte("v1")})) + assert.Equal(t, "v1", pipelineModeFor(ctx, q, "acc"), "per-account override beats env") + require.NoError(t, q.UpsertSettingString(ctx, gen.UpsertSettingStringParams{Key: store.PipelineOverridePrefix + "acc", Value: []byte("junk")})) + assert.Equal(t, "v2", pipelineModeFor(ctx, q, "acc"), "invalid override falls through") +} + +func TestMatchAnyChannel(t *testing.T) { + assert.Nil(t, matchAnyChannel(nil)) + m := matchAnyChannel([]string{"Fav"}) + assert.True(t, m([]string{"x", "fav"})) + assert.False(t, m([]string{"x"})) +} +``` + +(add `assert` import if missing.) + +Run: `go test ./cmd/miner/ -run 'PipelineModeFor|MatchAnyChannel' -v` +Expected: FAIL, undefined. + +- [ ] **Step 4: Implement helpers + wiring** + +In `internal/store/campaign_persister.go`, next to the other prefixes: + +```go +// PipelineOverridePrefix keys a per-account pipeline choice in kv. Full key: +// PipelineOverridePrefix + accountID, value "v1" or "v2". Beats GRUB_PIPELINE. +const PipelineOverridePrefix = "pipeline_override:" +``` + +In `cmd/miner/main.go`, add helpers near `loadAccountWhitelist`: + +```go +// pipelineModeFor picks v1 or v2 for an account: kv override first, then +// GRUB_PIPELINE, default v1. +func pipelineModeFor(ctx context.Context, q *gen.Queries, accountID string) string { + if v, err := q.GetSettingString(ctx, store.PipelineOverridePrefix+accountID); err == nil { + if s := string(v); s == "v1" || s == "v2" { + return s + } + } + if os.Getenv("GRUB_PIPELINE") == "v2" { + return "v2" + } + return "v1" +} + +// matchAnyChannel reports whether any campaign channel is one of logins. +// Nil when logins is empty, so the null-game gate stays off. +func matchAnyChannel(logins []string) func([]string) bool { + if len(logins) == 0 { + return nil + } + set := make(map[string]bool, len(logins)) + for _, l := range logins { + set[strings.ToLower(l)] = true + } + return func(chs []string) bool { + for _, c := range chs { + if set[strings.ToLower(c)] { + return true + } + } + return false + } +} +``` + +After `claimRecorder := store.NewClaimRecorder(q)` (around line 337) add: + +```go + dropStore := store.NewDropStateStore(q) + if n, err := store.BackfillDropState(ctx, q, time.Now()); err != nil { + logger.Warn("pipeline v2: drop_state backfill failed", "err", err) + } else if n > 0 { + logger.Info("pipeline v2: drop_state backfilled from v1 state", "rows", n) + } +``` + +In `build`, immediately before `acctLabel := a.DisplayName` / `w := watcher.New(...)`, insert: + +```go + if pipelineModeFor(ctx, q, a.ID) == "v2" { + prio, err := dropStore.StreamerPriority(ctx, a.ID) + if err != nil { + logger.Warn("pipeline v2: load streamer priority failed", "account", a.ID, "err", err) + } + var force []string + if rows, err := q.ListForceChannels(ctx, a.ID); err == nil { + for _, r := range rows { + force = append(force, r.Channel) + } + } + l, err := loop.New(loop.Config{ + AccountID: a.ID, AccountLabel: a.DisplayName, Platform: a.Platform, + Backend: b, Session: sess, + Store: dropStore, History: claimRecorder, Persister: campaignPersister, + Notifier: notifier, + AllowGame: allow, AllowChannel: matchAnyChannel(prio), GameRank: rank, + PriorityMode: priorityMode, ForceLinked: forceLinked, + StreamerPriority: prio, ForceWatch: force, + ProgressNotifyStepPct: progressStep, + }) + if err == nil { + logger.Info("pipeline v2 enabled for account", "account", a.ID, "platform", a.Platform) + return scheduler.NewEntry(a.ID, l), nil + } + logger.Warn("pipeline v2 unavailable, falling back to v1", "account", a.ID, "err", err) + } +``` + +Add import `"github.com/aalejandrofer/grubdrops/internal/pipeline/loop"`. `notifier` must satisfy `loop.Notifier` (same `Notify` signature as `watcher.Notifier`); if the variable's type is a concrete notify type, it already has the method. If `go build` reports `forceCollected` / `persistedSkips` unused only in some path, leave v1 code as is: they are still used by the v1 branch below. + +- [ ] **Step 5: Build + full tests** + +Run: `gofmt -w . && go build ./... && go test ./... 2>&1 | tail -30` +Expected: all packages `ok`. + +- [ ] **Step 6: Local smoke run** + +```bash +mkdir -p /tmp/grub-v2 && GRUB_MASTER_KEY=$(go run ./cmd/miner keygen) \ + GRUB_DB_PATH=/tmp/grub-v2/miner.db GRUB_SECURE_COOKIES=0 GRUB_PIPELINE=v2 \ + timeout 20 go run ./cmd/miner 2>&1 | grep -E "pipeline|backfill|error" | head -20 +``` + +Expected: starts without panic. With no accounts, there are no `pipeline v2 enabled` lines, and that's fine. This only proves boot + migration + backfill. + +- [ ] **Step 7: Commit** + +```bash +git add internal/scheduler internal/store/campaign_persister.go cmd/miner +git commit -m "feat(pipeline-v2): opt-in GRUB_PIPELINE=v2 wiring and scheduler interfaces + +Co-Authored-By: Claude Opus 5.5 (1M context) " +``` + +--- + +### Task 13: Changelog + spec refinements + whole-branch verification + +**Files:** +- Modify: `docs/CHANGELOG.md` (`## [Unreleased]`) +- Modify: `docs/superpowers/specs/2026-09-27-pipeline-v2-design.md` + +- [ ] **Step 1: Changelog entries** under `## [Unreleased]` + +```markdown +### Added +- Pipeline v2 (opt-in, `GRUB_PIPELINE=v2` or kv `pipeline_override:`): drop state now comes from each campaign's platform details, so drops claimed on the website or with a lost claim response show as claimed and mining moves on (#50). +- Streamer priority for pipeline v2: per-account priority streamers are watched first, then any live channel in the category (#49). Seeded from the existing null-game channel list. +- `drop_state` and `account_streamer_priority` tables (migration 0016), with a one-time backfill from claim history, manual marks and ghost-skips. + +### Changed +- Twitch claim code split into status fetch + classification (behaviour unchanged). +- Scheduler reads dashboard snapshots and discoveries through interfaces, so v1 and v2 runners both report state. + +### Fixed (pipeline v2 only) +- Claim failures back off 1m, 5m, 30m and stop after 5 attempts instead of looping forever. +- Kick link-required claims block with a reason instead of retrying every poll. +- Kick required minutes come from the platform, not an invented 120. +- Ghost-skipped drops are re-checked instead of being skipped forever. +``` + +- [ ] **Step 2: Record spec refinements** + +In the spec, add a short "## 10. Refinements during planning" section listing the four refinements from this plan's header (serial reconcile, unix INTEGER timestamps, `user_skip` no auto-expiry, interface names). + +- [ ] **Step 3: Whole-branch verification** + +Run: `gofmt -l . ; go vet ./... && go build ./... && go test -race ./... 2>&1 | tail -40` +Expected: `gofmt -l` prints nothing; all packages `ok`. + +- [ ] **Step 4: Commit** + +```bash +git add docs/CHANGELOG.md docs/superpowers/specs/2026-09-27-pipeline-v2-design.md +git commit -m "docs(pipeline-v2): changelog and spec refinements + +Co-Authored-By: Claude Opus 5.5 (1M context) " +``` + +--- + +### Task 14: Staging live verification (HUMAN-GATED, no tag) + +Accrual/claim change: CLAUDE.md requires a live drop before any tag. This task never tags. + +- [ ] **Step 1:** Push the branch and ask the user before deploying (staging runbook lives in the user's memory notes: local build, scp tar, `docker compose up` on the staging host; accounts are disabled by default there). +- [ ] **Step 2:** On staging, enable ONE Twitch account and ONE Kick account that have a live drop available, and set `GRUB_PIPELINE=v2` (or kv `pipeline_override:=v2` via `sqlite3`). +- [ ] **Step 3:** Watch logs for `pipeline decision`, `pipeline claim attempt`, `pipeline drop claimed`. Confirm on the platform's own site that minutes accrue and the claim lands. Confirm a drop claimed manually on the website flips to claimed within one reconcile (15 min) without a watch. +- [ ] **Step 4:** Record results (drop, channel, minutes, claim outcome) in the PR description. +- [ ] **Step 5:** Tear staging down (`docker compose down`) as soon as verification ends. +- [ ] **Step 6:** Hand back to the user for PR review and Plan 2. Do not merge or tag without their say-so. From bdf36609126210b39147c0bcaa6750470725d51e Mon Sep 17 00:00:00 2001 From: Alejandro Date: Sun, 27 Sep 2026 16:08:38 -0400 Subject: [PATCH 03/56] test(pipeline-v2): live probes for Twitch details self and Kick claimed retention Co-Authored-By: Claude Opus 5.5 (1M context) --- internal/platform/kick/live_probe_test.go | 34 ++++++++++++++++++ internal/platform/twitch/live_probe_test.go | 35 ++++++++++++++++++ internal/store/live_session_test.go | 39 +++++++++++++++++++++ 3 files changed, 108 insertions(+) create mode 100644 internal/platform/kick/live_probe_test.go create mode 100644 internal/platform/twitch/live_probe_test.go create mode 100644 internal/store/live_session_test.go diff --git a/internal/platform/kick/live_probe_test.go b/internal/platform/kick/live_probe_test.go new file mode 100644 index 0000000..4e6e3bc --- /dev/null +++ b/internal/platform/kick/live_probe_test.go @@ -0,0 +1,34 @@ +//go:build live + +package kick + +import ( + "context" + "encoding/json" + "net/http" + "os" + "testing" + "time" + + "github.com/stretchr/testify/require" + + "github.com/aalejandrofer/grubdrops/internal/platform" +) + +// TestLive_ProgressRaw dumps raw /drops/progress for a Kick account that has +// at least one CLAIMED reward, to learn whether claimed rewards stay listed. +// +// GRUB_LIVE_KICK_SESSION_JSON='{"cookies":{...}}' \ +// go test -tags live -run TestLive_ProgressRaw -v ./internal/platform/kick/ +func TestLive_ProgressRaw(t *testing.T) { + js := os.Getenv("GRUB_LIVE_KICK_SESSION_JSON") + if js == "" { + t.Skip("set GRUB_LIVE_KICK_SESSION_JSON") + } + var s platform.Session + require.NoError(t, json.Unmarshal([]byte(js), &s)) + b := New(nil, nil, "grubdrops-browser-{slug}", 9090, 10*time.Minute) + body, status, err := b.api.d.do(context.Background(), s, http.MethodGet, dropsBase+"/api/v1/drops/progress", nil) + require.NoError(t, err) + t.Logf("status=%d body=%s", status, body) +} diff --git a/internal/platform/twitch/live_probe_test.go b/internal/platform/twitch/live_probe_test.go new file mode 100644 index 0000000..38f2d88 --- /dev/null +++ b/internal/platform/twitch/live_probe_test.go @@ -0,0 +1,35 @@ +//go:build live + +package twitch + +import ( + "context" + "encoding/json" + "os" + "testing" + + "github.com/stretchr/testify/require" + + "github.com/aalejandrofer/grubdrops/internal/platform" +) + +// TestLive_DetailsSelf dumps raw DropCampaignDetails for a campaign the +// account is enrolled in (ideally one fully claimed), to confirm +// timeBasedDrops[].self carries isClaimed + currentMinutesWatched. +// +// GRUB_LIVE_TWITCH_TOKEN=... GRUB_LIVE_CAMPAIGN_ID=... \ +// go test -tags live -run TestLive_DetailsSelf -v ./internal/platform/twitch/ +func TestLive_DetailsSelf(t *testing.T) { + tok, camp := os.Getenv("GRUB_LIVE_TWITCH_TOKEN"), os.Getenv("GRUB_LIVE_CAMPAIGN_ID") + if tok == "" || camp == "" { + t.Skip("set GRUB_LIVE_TWITCH_TOKEN, GRUB_LIVE_CAMPAIGN_ID") + } + ctx := context.Background() + b := New() + login, err := b.disc.resolveCurrentLogin(ctx, platform.Session{AccessToken: tok}) + require.NoError(t, err) + var raw json.RawMessage + require.NoError(t, b.c.gql(ctx, tok, OpDropCampaignDetails, + map[string]any{"dropID": camp, "channelLogin": login}, &raw)) + t.Logf("%s", raw) +} diff --git a/internal/store/live_session_test.go b/internal/store/live_session_test.go new file mode 100644 index 0000000..eda84b9 --- /dev/null +++ b/internal/store/live_session_test.go @@ -0,0 +1,39 @@ +//go:build live + +package store + +import ( + "context" + "encoding/json" + "os" + "testing" + + "github.com/stretchr/testify/require" + + "github.com/aalejandrofer/grubdrops/internal/store/gen" +) + +// TestLive_DumpSession prints one account's decrypted session as JSON. +// Run against a COPY of the DB file: Open runs migrations. Output holds +// live tokens; do not paste it anywhere. +// +// GRUB_DB_PATH=/tmp/miner-copy.db GRUB_MASTER_KEY=... GRUB_LIVE_ACCOUNT=acc_x \ +// go test -tags live -run TestLive_DumpSession -v ./internal/store/ +func TestLive_DumpSession(t *testing.T) { + path, key, acct := os.Getenv("GRUB_DB_PATH"), os.Getenv("GRUB_MASTER_KEY"), os.Getenv("GRUB_LIVE_ACCOUNT") + if path == "" || key == "" || acct == "" { + t.Skip("set GRUB_DB_PATH, GRUB_MASTER_KEY, GRUB_LIVE_ACCOUNT") + } + ctx := context.Background() + db, err := Open(ctx, path) + require.NoError(t, err) + defer db.Close() + c, err := NewCryptor(key) + require.NoError(t, err) + s, ok, err := NewSessionStore(db, gen.New(db), c).Get(ctx, acct) + require.NoError(t, err) + require.True(t, ok, "no session for account") + b, err := json.Marshal(s) + require.NoError(t, err) + t.Logf("%s", b) +} From 76d00f83127d92c984f65432e33fc6342eb94d28 Mon Sep 17 00:00:00 2001 From: Alejandro Date: Sun, 27 Sep 2026 16:24:51 -0400 Subject: [PATCH 04/56] feat(pipeline-v2): per-drop state model with pure transitions Co-Authored-By: Claude Haiku 4.5 --- internal/pipeline/dropstate/dropstate.go | 248 ++++++++++++++++++ internal/pipeline/dropstate/dropstate_test.go | 141 ++++++++++ 2 files changed, 389 insertions(+) create mode 100644 internal/pipeline/dropstate/dropstate.go create mode 100644 internal/pipeline/dropstate/dropstate_test.go diff --git a/internal/pipeline/dropstate/dropstate.go b/internal/pipeline/dropstate/dropstate.go new file mode 100644 index 0000000..f529010 --- /dev/null +++ b/internal/pipeline/dropstate/dropstate.go @@ -0,0 +1,248 @@ +// Package dropstate is the per-(account, drop) state model for pipeline v2. +// Every transition is a pure function of (previous row, input, now) so the +// rules are table-testable and the account loop is the only writer. +package dropstate + +import "time" + +type Status string + +const ( + Eligible Status = "eligible" + Accruing Status = "accruing" + Claimable Status = "claimable" + Claimed Status = "claimed" + Blocked Status = "blocked" +) + +type Reason string + +const ( + NoReason Reason = "" + NeedsLink Reason = "needs_link" + ClaimFailed Reason = "claim_failed" + SubOnly Reason = "sub_only" + NoChannels Reason = "no_channels" + Expired Reason = "expired" + NotEnrolled Reason = "not_enrolled" + UserSkip Reason = "user_skip" +) + +type Source string + +const ( + FromPlatform Source = "platform" + FromUser Source = "user" +) + +// Retry windows for blocked rows. user_skip never auto-expires. +const ( + RetryNeedsLink = 24 * time.Hour + RetryClaimFailed = 6 * time.Hour + RetryNotEnrolled = time.Hour + RetrySubOnly = 24 * time.Hour + RetryExpired = 24 * time.Hour + MaxClaimFailures = 5 +) + +var claimBackoff = []time.Duration{time.Minute, 5 * time.Minute, 30 * time.Minute} + +type Row struct { + AccountID string + DropID string + CampaignID string + Platform string + Status Status + Reason Reason + Minutes int + Required int + Source Source + FailCount int + RetryAfter time.Time + SyncedAt time.Time + UpdatedAt time.Time +} + +// IsZero reports a row that has never been stored or derived. +func (r Row) IsZero() bool { return r.Status == "" } + +// Observation is what the platform said about one drop in one sync. +type Observation struct { + // Known is true when the platform returned a definite record for the + // drop. False means it said nothing, which proves nothing. + Known bool + // Unmineable marks drops the platform cannot track (scrape placeholders). + Unmineable bool + Claimed bool + Minutes int + Required int +} + +func derive(minutes, required int) (Status, Reason) { + switch { + case required <= 0: + return Blocked, SubOnly + case minutes >= required: + return Claimable, NoReason + case minutes > 0: + return Accruing, NoReason + default: + return Eligible, NoReason + } +} + +func retryFor(reason Reason, now time.Time) time.Time { + switch reason { + case NeedsLink: + return now.Add(RetryNeedsLink) + case ClaimFailed: + return now.Add(RetryClaimFailed) + case NotEnrolled, NoChannels: + return now.Add(RetryNotEnrolled) + case SubOnly: + return now.Add(RetrySubOnly) + case Expired: + return now.Add(RetryExpired) + } + return time.Time{} +} + +func (r Row) with(s Status, reason Reason, now time.Time) Row { + r.Status, r.Reason, r.UpdatedAt = s, reason, now + r.RetryAfter = retryFor(reason, now) + return r +} + +// Apply folds one platform observation into the previous row. The caller +// seeds identity fields (AccountID, DropID, CampaignID, Platform) on a new row. +func Apply(prev Row, obs Observation, now time.Time) Row { + r := prev + if obs.Known && !obs.Unmineable { + r.Minutes, r.Required, r.SyncedAt = obs.Minutes, obs.Required, now + } + // A platform-confirmed claim is final. + if prev.Status == Claimed && prev.Source == FromPlatform { + if r.Minutes < prev.Minutes { + r.Minutes = prev.Minutes + } + return r + } + if !obs.Known { + switch { + case prev.IsZero(): + r.Required = obs.Required + r.Source = FromPlatform + st, rs := derive(0, obs.Required) + return r.with(st, rs, now) + case (prev.Status == Accruing || prev.Status == Claimable) && prev.Source == FromPlatform: + return r.with(Blocked, NotEnrolled, now) + case prev.Status == Blocked && prev.Reason == NotEnrolled && !now.Before(prev.RetryAfter): + st, rs := derive(prev.Minutes, prev.Required) + return r.with(st, rs, now) + } + return r + } + if obs.Unmineable { + if prev.Status == Claimed { + return r + } + r.Source = FromPlatform + return r.with(Blocked, NotEnrolled, now) + } + if obs.Claimed { + r.Source, r.FailCount = FromPlatform, 0 + return r.with(Claimed, NoReason, now) + } + if prev.Status == Blocked && prev.Reason == UserSkip { + return r + } + if prev.Status == Blocked && (prev.Reason == NeedsLink || prev.Reason == ClaimFailed) && now.Before(prev.RetryAfter) { + return r + } + st, rs := derive(r.Minutes, r.Required) + if st == Claimable && prev.Status == Claimable && now.Before(prev.RetryAfter) { + return r // keep claim backoff + } + if prev.Status == Blocked { + r.FailCount = 0 + } + r.Source = FromPlatform + return r.with(st, rs, now) +} + +// ClaimOK records a successful or already-claimed claim. +func ClaimOK(prev Row, now time.Time) Row { + r := prev.with(Claimed, NoReason, now) + r.Source, r.FailCount = FromPlatform, 0 + return r +} + +// ClaimNeedsLink blocks a drop whose claim needs an external account link. +func ClaimNeedsLink(prev Row, now time.Time) Row { + r := prev.with(Blocked, NeedsLink, now) + r.Source = FromPlatform + return r +} + +// ClaimFailedAttempt counts a failed claim: backoff 1m, 5m, 30m, then +// blocked:claim_failed after MaxClaimFailures. +func ClaimFailedAttempt(prev Row, now time.Time) Row { + r := prev + r.FailCount++ + r.UpdatedAt = now + if r.FailCount >= MaxClaimFailures { + return r.with(Blocked, ClaimFailed, now) + } + i := r.FailCount - 1 + if i >= len(claimBackoff) { + i = len(claimBackoff) - 1 + } + r.RetryAfter = now.Add(claimBackoff[i]) + return r +} + +// MarkCollected is the user asserting the drop is claimed. +func MarkCollected(prev Row, now time.Time) Row { + r := prev.with(Claimed, NoReason, now) + r.Source = FromUser + return r +} + +// Skip is the user excluding a drop from mining until they retry it. +func Skip(prev Row, now time.Time) Row { + r := prev.with(Blocked, UserSkip, now) + r.Source = FromUser + return r +} + +// Retry clears any block and re-derives from the last known minutes. +func Retry(prev Row, now time.Time) Row { + st, rs := derive(prev.Minutes, prev.Required) + r := prev.with(st, rs, now) + r.FailCount, r.Source = 0, FromPlatform + return r +} + +// Expire blocks an unclaimed drop whose campaign has ended. +func Expire(prev Row, now time.Time) Row { + if prev.Status == Claimed { + return prev + } + return prev.with(Blocked, Expired, now) +} + +// BlockLink blocks a drop whose campaign needs an unlinked external account. +func BlockLink(prev Row, now time.Time) Row { + if prev.Status == Claimed || (prev.Status == Blocked && prev.Reason == UserSkip) { + return prev + } + return prev.with(Blocked, NeedsLink, now) +} + +// Mineable reports whether watching can still advance the drop. +func Mineable(r Row) bool { return r.Status == Eligible || r.Status == Accruing } + +// ReadyToClaim reports a claimable row whose backoff has elapsed. +func ReadyToClaim(r Row, now time.Time) bool { + return r.Status == Claimable && !now.Before(r.RetryAfter) +} diff --git a/internal/pipeline/dropstate/dropstate_test.go b/internal/pipeline/dropstate/dropstate_test.go new file mode 100644 index 0000000..620b1f6 --- /dev/null +++ b/internal/pipeline/dropstate/dropstate_test.go @@ -0,0 +1,141 @@ +package dropstate + +import ( + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +var t0 = time.Unix(1_000_000, 0) + +func base() Row { + return Row{AccountID: "a", DropID: "d", CampaignID: "c", Platform: "twitch", Source: FromPlatform} +} + +func row(s Status, r Reason, min, req int) Row { + x := base() + x.Status, x.Reason, x.Minutes, x.Required = s, r, min, req + return x +} + +func TestApply(t *testing.T) { + userClaimed := row(Claimed, NoReason, 0, 60) + userClaimed.Source = FromUser + notEnrolled := row(Blocked, NotEnrolled, 10, 60) + notEnrolled.RetryAfter = t0.Add(time.Hour) + skipped := row(Blocked, UserSkip, 0, 60) + skipped.Source = FromUser + needsLink := row(Blocked, NeedsLink, 60, 60) + needsLink.RetryAfter = t0.Add(RetryNeedsLink) + + cases := []struct { + name string + prev Row + obs Observation + at time.Time + want Status + reason Reason + src Source + }{ + {"new drop, platform silent -> eligible", base(), Observation{Required: 60}, t0, Eligible, NoReason, FromPlatform}, + {"new drop, platform silent, zero required -> sub_only", base(), Observation{}, t0, Blocked, SubOnly, FromPlatform}, + {"known progress -> accruing", row(Eligible, NoReason, 0, 60), Observation{Known: true, Minutes: 10, Required: 60}, t0, Accruing, NoReason, FromPlatform}, + {"known full -> claimable", row(Accruing, NoReason, 50, 60), Observation{Known: true, Minutes: 60, Required: 60}, t0, Claimable, NoReason, FromPlatform}, + {"known claimed -> claimed", row(Accruing, NoReason, 10, 60), Observation{Known: true, Claimed: true, Minutes: 60, Required: 60}, t0, Claimed, NoReason, FromPlatform}, + {"platform claimed never demoted", row(Claimed, NoReason, 60, 60), Observation{Known: true, Minutes: 0, Required: 60}, t0, Claimed, NoReason, FromPlatform}, + {"user mark kept while platform silent", userClaimed, Observation{Required: 60}, t0, Claimed, NoReason, FromUser}, + {"user mark overwritten by definite platform answer", userClaimed, Observation{Known: true, Minutes: 5, Required: 60}, t0, Accruing, NoReason, FromPlatform}, + {"accruing then platform silent -> not_enrolled", row(Accruing, NoReason, 10, 60), Observation{Required: 60}, t0, Blocked, NotEnrolled, FromPlatform}, + {"not_enrolled held inside window", notEnrolled, Observation{Required: 60}, t0, Blocked, NotEnrolled, FromPlatform}, + {"not_enrolled re-derived after window", notEnrolled, Observation{Required: 60}, t0.Add(2 * time.Hour), Accruing, NoReason, FromPlatform}, + {"unmineable placeholder -> not_enrolled", base(), Observation{Known: true, Unmineable: true}, t0, Blocked, NotEnrolled, FromPlatform}, + {"user skip survives progress", skipped, Observation{Known: true, Minutes: 30, Required: 60}, t0, Blocked, UserSkip, FromUser}, + {"claimed beats user skip", skipped, Observation{Known: true, Claimed: true, Minutes: 60, Required: 60}, t0, Claimed, NoReason, FromPlatform}, + {"needs_link held inside window", needsLink, Observation{Known: true, Minutes: 60, Required: 60}, t0, Blocked, NeedsLink, FromPlatform}, + {"needs_link re-derived after window", needsLink, Observation{Known: true, Minutes: 60, Required: 60}, t0.Add(25 * time.Hour), Claimable, NoReason, FromPlatform}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + got := Apply(tc.prev, tc.obs, tc.at) + assert.Equal(t, tc.want, got.Status) + assert.Equal(t, tc.reason, got.Reason) + assert.Equal(t, tc.src, got.Source) + assert.Equal(t, "d", got.DropID, "identity preserved") + }) + } +} + +func TestApply_PlatformClaimedKeepsHighestMinutes(t *testing.T) { + got := Apply(row(Claimed, NoReason, 60, 60), Observation{Known: true, Minutes: 0, Required: 60}, t0) + assert.Equal(t, 60, got.Minutes) +} + +func TestApply_KeepsClaimBackoff(t *testing.T) { + r := ClaimFailedAttempt(row(Claimable, NoReason, 60, 60), t0) + got := Apply(r, Observation{Known: true, Minutes: 60, Required: 60}, t0.Add(10*time.Second)) + assert.Equal(t, Claimable, got.Status) + assert.Equal(t, r.RetryAfter, got.RetryAfter, "sync must not reset claim backoff") + assert.Equal(t, 1, got.FailCount) +} + +func TestClaimFailedAttempt_Ladder(t *testing.T) { + r := row(Claimable, NoReason, 60, 60) + for _, wait := range []time.Duration{time.Minute, 5 * time.Minute, 30 * time.Minute, 30 * time.Minute} { + r = ClaimFailedAttempt(r, t0) + require.Equal(t, Claimable, r.Status) + require.Equal(t, t0.Add(wait), r.RetryAfter) + require.False(t, ReadyToClaim(r, t0)) + require.True(t, ReadyToClaim(r, t0.Add(wait))) + } + r = ClaimFailedAttempt(r, t0) + assert.Equal(t, Blocked, r.Status) + assert.Equal(t, ClaimFailed, r.Reason) + assert.Equal(t, t0.Add(RetryClaimFailed), r.RetryAfter) +} + +func TestClaimOutcomes(t *testing.T) { + ok := ClaimOK(row(Claimable, NoReason, 60, 60), t0) + assert.Equal(t, Claimed, ok.Status) + assert.Equal(t, FromPlatform, ok.Source) + link := ClaimNeedsLink(row(Claimable, NoReason, 60, 60), t0) + assert.Equal(t, Blocked, link.Status) + assert.Equal(t, NeedsLink, link.Reason) + assert.Equal(t, t0.Add(RetryNeedsLink), link.RetryAfter) +} + +func TestUserActions(t *testing.T) { + m := MarkCollected(row(Accruing, NoReason, 10, 60), t0) + assert.Equal(t, Claimed, m.Status) + assert.Equal(t, FromUser, m.Source) + + s := Skip(row(Accruing, NoReason, 10, 60), t0) + assert.Equal(t, Blocked, s.Status) + assert.Equal(t, UserSkip, s.Reason) + assert.True(t, s.RetryAfter.IsZero(), "user_skip never auto-expires") + + failed := row(Blocked, ClaimFailed, 60, 60) + failed.FailCount = 5 + r := Retry(failed, t0) + assert.Equal(t, Claimable, r.Status) + assert.Equal(t, 0, r.FailCount) +} + +func TestExpireAndBlockLink_LeaveClaimedAlone(t *testing.T) { + c := row(Claimed, NoReason, 60, 60) + assert.Equal(t, Claimed, Expire(c, t0).Status) + assert.Equal(t, Claimed, BlockLink(c, t0).Status) + assert.Equal(t, Expired, Expire(row(Accruing, NoReason, 10, 60), t0).Reason) + assert.Equal(t, NeedsLink, BlockLink(row(Eligible, NoReason, 0, 60), t0).Reason) + sk := row(Blocked, UserSkip, 0, 60) + assert.Equal(t, UserSkip, BlockLink(sk, t0).Reason) +} + +func TestMineable(t *testing.T) { + assert.True(t, Mineable(row(Eligible, NoReason, 0, 60))) + assert.True(t, Mineable(row(Accruing, NoReason, 1, 60))) + assert.False(t, Mineable(row(Claimable, NoReason, 60, 60))) + assert.False(t, Mineable(row(Blocked, NotEnrolled, 0, 60))) + assert.False(t, Mineable(Row{})) +} From 7f8edb3e5998e6e89b06c46debe3aee2624cc4a9 Mon Sep 17 00:00:00 2001 From: Alejandro Date: Sun, 27 Sep 2026 17:07:23 -0400 Subject: [PATCH 05/56] fix(pipeline-v2): guard platform claims in every mutator, re-derive sub_only Co-Authored-By: Claude Haiku 4.5 --- internal/pipeline/dropstate/dropstate.go | 17 +++++++- internal/pipeline/dropstate/dropstate_test.go | 43 +++++++++++++++++++ 2 files changed, 59 insertions(+), 1 deletion(-) diff --git a/internal/pipeline/dropstate/dropstate.go b/internal/pipeline/dropstate/dropstate.go index f529010..17eaed2 100644 --- a/internal/pipeline/dropstate/dropstate.go +++ b/internal/pipeline/dropstate/dropstate.go @@ -136,7 +136,7 @@ func Apply(prev Row, obs Observation, now time.Time) Row { return r.with(st, rs, now) case (prev.Status == Accruing || prev.Status == Claimable) && prev.Source == FromPlatform: return r.with(Blocked, NotEnrolled, now) - case prev.Status == Blocked && prev.Reason == NotEnrolled && !now.Before(prev.RetryAfter): + case prev.Status == Blocked && (prev.Reason == NotEnrolled || prev.Reason == SubOnly) && !now.Before(prev.RetryAfter): st, rs := derive(prev.Minutes, prev.Required) return r.with(st, rs, now) } @@ -179,6 +179,9 @@ func ClaimOK(prev Row, now time.Time) Row { // ClaimNeedsLink blocks a drop whose claim needs an external account link. func ClaimNeedsLink(prev Row, now time.Time) Row { + if prev.Status == Claimed && prev.Source == FromPlatform { + return prev + } r := prev.with(Blocked, NeedsLink, now) r.Source = FromPlatform return r @@ -187,6 +190,9 @@ func ClaimNeedsLink(prev Row, now time.Time) Row { // ClaimFailedAttempt counts a failed claim: backoff 1m, 5m, 30m, then // blocked:claim_failed after MaxClaimFailures. func ClaimFailedAttempt(prev Row, now time.Time) Row { + if prev.Status == Claimed && prev.Source == FromPlatform { + return prev + } r := prev r.FailCount++ r.UpdatedAt = now @@ -203,6 +209,9 @@ func ClaimFailedAttempt(prev Row, now time.Time) Row { // MarkCollected is the user asserting the drop is claimed. func MarkCollected(prev Row, now time.Time) Row { + if prev.Status == Claimed { + return prev + } r := prev.with(Claimed, NoReason, now) r.Source = FromUser return r @@ -210,6 +219,9 @@ func MarkCollected(prev Row, now time.Time) Row { // Skip is the user excluding a drop from mining until they retry it. func Skip(prev Row, now time.Time) Row { + if prev.Status == Claimed && prev.Source == FromPlatform { + return prev + } r := prev.with(Blocked, UserSkip, now) r.Source = FromUser return r @@ -217,6 +229,9 @@ func Skip(prev Row, now time.Time) Row { // Retry clears any block and re-derives from the last known minutes. func Retry(prev Row, now time.Time) Row { + if prev.Status == Claimed && prev.Source == FromPlatform { + return prev + } st, rs := derive(prev.Minutes, prev.Required) r := prev.with(st, rs, now) r.FailCount, r.Source = 0, FromPlatform diff --git a/internal/pipeline/dropstate/dropstate_test.go b/internal/pipeline/dropstate/dropstate_test.go index 620b1f6..2b31ca7 100644 --- a/internal/pipeline/dropstate/dropstate_test.go +++ b/internal/pipeline/dropstate/dropstate_test.go @@ -30,6 +30,9 @@ func TestApply(t *testing.T) { needsLink := row(Blocked, NeedsLink, 60, 60) needsLink.RetryAfter = t0.Add(RetryNeedsLink) + subOnly := row(Blocked, SubOnly, 0, 60) + subOnly.RetryAfter = t0.Add(RetrySubOnly) + cases := []struct { name string prev Row @@ -55,6 +58,8 @@ func TestApply(t *testing.T) { {"claimed beats user skip", skipped, Observation{Known: true, Claimed: true, Minutes: 60, Required: 60}, t0, Claimed, NoReason, FromPlatform}, {"needs_link held inside window", needsLink, Observation{Known: true, Minutes: 60, Required: 60}, t0, Blocked, NeedsLink, FromPlatform}, {"needs_link re-derived after window", needsLink, Observation{Known: true, Minutes: 60, Required: 60}, t0.Add(25 * time.Hour), Claimable, NoReason, FromPlatform}, + {"sub_only held inside window", subOnly, Observation{Required: 60}, t0, Blocked, SubOnly, FromPlatform}, + {"sub_only re-derived after window", subOnly, Observation{Required: 60}, t0.Add(25 * time.Hour), Eligible, NoReason, FromPlatform}, } for _, tc := range cases { t.Run(tc.name, func(t *testing.T) { @@ -139,3 +144,41 @@ func TestMineable(t *testing.T) { assert.False(t, Mineable(row(Blocked, NotEnrolled, 0, 60))) assert.False(t, Mineable(Row{})) } + +func TestMutators_NeverDemotePlatformClaim(t *testing.T) { + platformClaimed := row(Claimed, NoReason, 60, 60) + platformClaimed.Source = FromPlatform + + // ClaimNeedsLink must not demote a platform claim + got := ClaimNeedsLink(platformClaimed, t0) + assert.Equal(t, Claimed, got.Status) + assert.Equal(t, FromPlatform, got.Source) + + // ClaimFailedAttempt must not demote a platform claim + got = ClaimFailedAttempt(platformClaimed, t0) + assert.Equal(t, Claimed, got.Status) + assert.Equal(t, FromPlatform, got.Source) + + // Skip must not demote a platform claim + got = Skip(platformClaimed, t0) + assert.Equal(t, Claimed, got.Status) + assert.Equal(t, FromPlatform, got.Source) + + // Retry must not demote a platform claim + got = Retry(platformClaimed, t0) + assert.Equal(t, Claimed, got.Status) + assert.Equal(t, FromPlatform, got.Source) + + // MarkCollected must not downgrade a platform claim to user source + got = MarkCollected(platformClaimed, t0) + assert.Equal(t, Claimed, got.Status) + assert.Equal(t, FromPlatform, got.Source) + + // But Skip on a user-claimed row DOES produce Blocked/UserSkip + userClaimed := row(Claimed, NoReason, 60, 60) + userClaimed.Source = FromUser + got = Skip(userClaimed, t0) + assert.Equal(t, Blocked, got.Status) + assert.Equal(t, UserSkip, got.Reason) + assert.Equal(t, FromUser, got.Source) +} From 9fda8ef93a74587faa2d0ac92cadef6f7fdb8b9c Mon Sep 17 00:00:00 2001 From: Alejandro Date: Sun, 27 Sep 2026 17:11:09 -0400 Subject: [PATCH 06/56] feat(pipeline-v2): drop_state and streamer priority tables Co-Authored-By: Claude Opus 5.5 (1M context) --- internal/store/dropstate_store.go | 87 +++++++++++++++ internal/store/dropstate_store_test.go | 85 ++++++++++++++ internal/store/gen/drop_state.sql.go | 105 ++++++++++++++++++ internal/store/gen/models.go | 23 ++++ internal/store/gen/querier.go | 3 + internal/store/gen/streamer_priority.sql.go | 44 ++++++++ .../store/migrations/0016_pipeline_v2.sql | 46 ++++++++ internal/store/queries/drop_state.sql | 20 ++++ internal/store/queries/streamer_priority.sql | 4 + 9 files changed, 417 insertions(+) create mode 100644 internal/store/dropstate_store.go create mode 100644 internal/store/dropstate_store_test.go create mode 100644 internal/store/gen/drop_state.sql.go create mode 100644 internal/store/gen/streamer_priority.sql.go create mode 100644 internal/store/migrations/0016_pipeline_v2.sql create mode 100644 internal/store/queries/drop_state.sql create mode 100644 internal/store/queries/streamer_priority.sql diff --git a/internal/store/dropstate_store.go b/internal/store/dropstate_store.go new file mode 100644 index 0000000..d09b83b --- /dev/null +++ b/internal/store/dropstate_store.go @@ -0,0 +1,87 @@ +package store + +import ( + "context" + "time" + + "github.com/aalejandrofer/grubdrops/internal/pipeline/dropstate" + "github.com/aalejandrofer/grubdrops/internal/store/gen" +) + +// DropStateStore persists pipeline v2 drop_state rows and reads the +// per-account streamer priority list. +type DropStateStore struct { + Q *gen.Queries +} + +func NewDropStateStore(q *gen.Queries) *DropStateStore { return &DropStateStore{Q: q} } + +func dsUnix(t time.Time) int64 { + if t.IsZero() { + return 0 + } + return t.Unix() +} + +func dsTime(v int64) time.Time { + if v == 0 { + return time.Time{} + } + return time.Unix(v, 0) +} + +func (s *DropStateStore) List(ctx context.Context, accountID string) ([]dropstate.Row, error) { + rows, err := s.Q.ListDropStates(ctx, accountID) + if err != nil { + return nil, err + } + out := make([]dropstate.Row, 0, len(rows)) + for _, r := range rows { + out = append(out, dropstate.Row{ + AccountID: r.AccountID, + DropID: r.DropID, + CampaignID: r.CampaignID, + Platform: r.Platform, + Status: dropstate.Status(r.Status), + Reason: dropstate.Reason(r.BlockReason), + Minutes: int(r.Minutes), + Required: int(r.Required), + Source: dropstate.Source(r.Source), + FailCount: int(r.FailCount), + RetryAfter: dsTime(r.RetryAfter), + SyncedAt: dsTime(r.SyncedAt), + UpdatedAt: dsTime(r.UpdatedAt), + }) + } + return out, nil +} + +func (s *DropStateStore) Upsert(ctx context.Context, r dropstate.Row) error { + return s.Q.UpsertDropState(ctx, gen.UpsertDropStateParams{ + AccountID: r.AccountID, + DropID: r.DropID, + CampaignID: r.CampaignID, + Platform: r.Platform, + Status: string(r.Status), + BlockReason: string(r.Reason), + Minutes: int64(r.Minutes), + Required: int64(r.Required), + Source: string(r.Source), + FailCount: int64(r.FailCount), + RetryAfter: dsUnix(r.RetryAfter), + SyncedAt: dsUnix(r.SyncedAt), + UpdatedAt: dsUnix(r.UpdatedAt), + }) +} + +func (s *DropStateStore) StreamerPriority(ctx context.Context, accountID string) ([]string, error) { + rows, err := s.Q.ListStreamerPriority(ctx, accountID) + if err != nil { + return nil, err + } + out := make([]string, 0, len(rows)) + for _, r := range rows { + out = append(out, r.Login) + } + return out, nil +} diff --git a/internal/store/dropstate_store_test.go b/internal/store/dropstate_store_test.go new file mode 100644 index 0000000..6577308 --- /dev/null +++ b/internal/store/dropstate_store_test.go @@ -0,0 +1,85 @@ +package store + +import ( + "context" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/aalejandrofer/grubdrops/internal/pipeline/dropstate" + "github.com/aalejandrofer/grubdrops/internal/store/gen" +) + +func seedAccount(t *testing.T, q *gen.Queries, id, platform string) { + t.Helper() + now := time.Now().Unix() + _, err := q.CreateAccount(context.Background(), gen.CreateAccountParams{ + ID: id, Platform: platform, DisplayName: id, + Status: "idle", FingerprintJson: "{}", Enabled: 1, + CreatedAt: now, UpdatedAt: now, + }) + require.NoError(t, err) +} + +func TestDropStateStore_RoundTrip(t *testing.T) { + db := openTest(t) + q := gen.New(db) + ctx := context.Background() + seedAccount(t, q, "acc-1", "twitch") + s := NewDropStateStore(q) + + at := time.Unix(1_700_000_000, 0) + in := dropstate.Row{ + AccountID: "acc-1", DropID: "d1", CampaignID: "c1", Platform: "twitch", + Status: dropstate.Blocked, Reason: dropstate.ClaimFailed, + Minutes: 60, Required: 60, Source: dropstate.FromPlatform, FailCount: 5, + RetryAfter: at.Add(time.Hour), SyncedAt: at, UpdatedAt: at, + } + require.NoError(t, s.Upsert(ctx, in)) + got, err := s.List(ctx, "acc-1") + require.NoError(t, err) + require.Len(t, got, 1) + assert.Equal(t, in, got[0]) + + in.Status, in.Reason, in.RetryAfter = dropstate.Claimed, dropstate.NoReason, time.Time{} + require.NoError(t, s.Upsert(ctx, in)) + got, err = s.List(ctx, "acc-1") + require.NoError(t, err) + require.Len(t, got, 1, "upsert must not duplicate") + assert.Equal(t, dropstate.Claimed, got[0].Status) + assert.True(t, got[0].RetryAfter.IsZero(), "0 maps back to zero time") +} + +func TestDropStateStore_StreamerPrioritySeededFromAccountChannels(t *testing.T) { + db := openTest(t) + q := gen.New(db) + ctx := context.Background() + seedAccount(t, q, "acc-1", "kick") + s := NewDropStateStore(q) + + // Migration already ran; insert straight into the new table like the + // seed does, then read ordered by rank. + _, err := db.ExecContext(ctx, `INSERT INTO account_streamer_priority (account_id, platform, login, rank) VALUES + ('acc-1','kick','zeta',1), ('acc-1','kick','alpha',0)`) + require.NoError(t, err) + got, err := s.StreamerPriority(ctx, "acc-1") + require.NoError(t, err) + assert.Equal(t, []string{"alpha", "zeta"}, got) +} + +func TestMigration0016_SeedCopiesAccountChannels(t *testing.T) { + db := openTest(t) + q := gen.New(db) + ctx := context.Background() + seedAccount(t, q, "acc-1", "kick") + require.NoError(t, q.AddAccountChannel(ctx, gen.AddAccountChannelParams{AccountID: "acc-1", Channel: "oilrats", Rank: 0})) + // Re-run the seed statement from 0016 exactly. + _, err := db.ExecContext(ctx, `INSERT INTO account_streamer_priority (account_id, platform, login, rank) + SELECT ac.account_id, a.platform, ac.channel, ac.rank FROM account_channels ac JOIN accounts a ON a.id = ac.account_id`) + require.NoError(t, err) + got, err := NewDropStateStore(q).StreamerPriority(ctx, "acc-1") + require.NoError(t, err) + assert.Equal(t, []string{"oilrats"}, got) +} diff --git a/internal/store/gen/drop_state.sql.go b/internal/store/gen/drop_state.sql.go new file mode 100644 index 0000000..f45d4b7 --- /dev/null +++ b/internal/store/gen/drop_state.sql.go @@ -0,0 +1,105 @@ +// Code generated by sqlc. DO NOT EDIT. +// versions: +// sqlc v1.31.1 +// source: drop_state.sql + +package gen + +import ( + "context" +) + +const listDropStates = `-- name: ListDropStates :many +SELECT account_id, drop_id, campaign_id, platform, status, block_reason, minutes, required, source, fail_count, retry_after, synced_at, updated_at FROM drop_state WHERE account_id = ? ORDER BY campaign_id, drop_id +` + +func (q *Queries) ListDropStates(ctx context.Context, accountID string) ([]DropState, error) { + rows, err := q.db.QueryContext(ctx, listDropStates, accountID) + if err != nil { + return nil, err + } + defer rows.Close() + var items []DropState + for rows.Next() { + var i DropState + if err := rows.Scan( + &i.AccountID, + &i.DropID, + &i.CampaignID, + &i.Platform, + &i.Status, + &i.BlockReason, + &i.Minutes, + &i.Required, + &i.Source, + &i.FailCount, + &i.RetryAfter, + &i.SyncedAt, + &i.UpdatedAt, + ); err != nil { + return nil, err + } + items = append(items, i) + } + if err := rows.Close(); err != nil { + return nil, err + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + +const upsertDropState = `-- name: UpsertDropState :exec +INSERT INTO drop_state ( + account_id, drop_id, campaign_id, platform, status, block_reason, + minutes, required, source, fail_count, retry_after, synced_at, updated_at +) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) +ON CONFLICT(account_id, drop_id) DO UPDATE SET + campaign_id = excluded.campaign_id, + platform = excluded.platform, + status = excluded.status, + block_reason = excluded.block_reason, + minutes = excluded.minutes, + required = excluded.required, + source = excluded.source, + fail_count = excluded.fail_count, + retry_after = excluded.retry_after, + synced_at = excluded.synced_at, + updated_at = excluded.updated_at +` + +type UpsertDropStateParams struct { + AccountID string `json:"account_id"` + DropID string `json:"drop_id"` + CampaignID string `json:"campaign_id"` + Platform string `json:"platform"` + Status string `json:"status"` + BlockReason string `json:"block_reason"` + Minutes int64 `json:"minutes"` + Required int64 `json:"required"` + Source string `json:"source"` + FailCount int64 `json:"fail_count"` + RetryAfter int64 `json:"retry_after"` + SyncedAt int64 `json:"synced_at"` + UpdatedAt int64 `json:"updated_at"` +} + +func (q *Queries) UpsertDropState(ctx context.Context, arg UpsertDropStateParams) error { + _, err := q.db.ExecContext(ctx, upsertDropState, + arg.AccountID, + arg.DropID, + arg.CampaignID, + arg.Platform, + arg.Status, + arg.BlockReason, + arg.Minutes, + arg.Required, + arg.Source, + arg.FailCount, + arg.RetryAfter, + arg.SyncedAt, + arg.UpdatedAt, + ) + return err +} diff --git a/internal/store/gen/models.go b/internal/store/gen/models.go index a749370..dfce45d 100644 --- a/internal/store/gen/models.go +++ b/internal/store/gen/models.go @@ -49,6 +49,13 @@ type AccountGame struct { Rank int64 `json:"rank"` } +type AccountStreamerPriority struct { + AccountID string `json:"account_id"` + Platform string `json:"platform"` + Login string `json:"login"` + Rank int64 `json:"rank"` +} + type Admin struct { ID int64 `json:"id"` PasswordHash string `json:"password_hash"` @@ -92,6 +99,22 @@ type Claim struct { ValueMetaJson string `json:"value_meta_json"` } +type DropState struct { + AccountID string `json:"account_id"` + DropID string `json:"drop_id"` + CampaignID string `json:"campaign_id"` + Platform string `json:"platform"` + Status string `json:"status"` + BlockReason string `json:"block_reason"` + Minutes int64 `json:"minutes"` + Required int64 `json:"required"` + Source string `json:"source"` + FailCount int64 `json:"fail_count"` + RetryAfter int64 `json:"retry_after"` + SyncedAt int64 `json:"synced_at"` + UpdatedAt int64 `json:"updated_at"` +} + type Game struct { ID string `json:"id"` Name string `json:"name"` diff --git a/internal/store/gen/querier.go b/internal/store/gen/querier.go index 4000085..3f51d21 100644 --- a/internal/store/gen/querier.go +++ b/internal/store/gen/querier.go @@ -74,6 +74,7 @@ type Querier interface { // Campaigns currently in flight (starts_at <= now < ends_at). // Whitelist filtering is applied in Go. ListCurrentCampaigns(ctx context.Context, arg ListCurrentCampaignsParams) ([]Campaign, error) + ListDropStates(ctx context.Context, accountID string) ([]DropState, error) ListEnabledAccounts(ctx context.Context) ([]Account, error) ListForceChannels(ctx context.Context, accountID string) ([]ListForceChannelsRow, error) ListGlobalGames(ctx context.Context) ([]ListGlobalGamesRow, error) @@ -81,6 +82,7 @@ type Querier interface { // Campaigns that have ended. Whitelist filtering is applied in Go. ListPastCampaigns(ctx context.Context, arg ListPastCampaignsParams) ([]Campaign, error) ListRecentClaims(ctx context.Context, limit int64) ([]ListRecentClaimsRow, error) + ListStreamerPriority(ctx context.Context, accountID string) ([]ListStreamerPriorityRow, error) ListUnclaimedProgressForAccount(ctx context.Context, arg ListUnclaimedProgressForAccountParams) ([]Progress, error) // Campaigns announced but not yet started. Whitelist filtering is // applied in Go. @@ -100,6 +102,7 @@ type Querier interface { UpsertAdmin(ctx context.Context, arg UpsertAdminParams) error UpsertBenefit(ctx context.Context, arg UpsertBenefitParams) error UpsertCampaign(ctx context.Context, arg UpsertCampaignParams) error + UpsertDropState(ctx context.Context, arg UpsertDropStateParams) error UpsertGame(ctx context.Context, arg UpsertGameParams) error UpsertProgress(ctx context.Context, arg UpsertProgressParams) error UpsertSession(ctx context.Context, arg UpsertSessionParams) error diff --git a/internal/store/gen/streamer_priority.sql.go b/internal/store/gen/streamer_priority.sql.go new file mode 100644 index 0000000..1e9abbd --- /dev/null +++ b/internal/store/gen/streamer_priority.sql.go @@ -0,0 +1,44 @@ +// Code generated by sqlc. DO NOT EDIT. +// versions: +// sqlc v1.31.1 +// source: streamer_priority.sql + +package gen + +import ( + "context" +) + +const listStreamerPriority = `-- name: ListStreamerPriority :many +SELECT login, rank FROM account_streamer_priority +WHERE account_id = ? +ORDER BY rank ASC, login ASC +` + +type ListStreamerPriorityRow struct { + Login string `json:"login"` + Rank int64 `json:"rank"` +} + +func (q *Queries) ListStreamerPriority(ctx context.Context, accountID string) ([]ListStreamerPriorityRow, error) { + rows, err := q.db.QueryContext(ctx, listStreamerPriority, accountID) + if err != nil { + return nil, err + } + defer rows.Close() + var items []ListStreamerPriorityRow + for rows.Next() { + var i ListStreamerPriorityRow + if err := rows.Scan(&i.Login, &i.Rank); err != nil { + return nil, err + } + items = append(items, i) + } + if err := rows.Close(); err != nil { + return nil, err + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} diff --git a/internal/store/migrations/0016_pipeline_v2.sql b/internal/store/migrations/0016_pipeline_v2.sql new file mode 100644 index 0000000..b5bcdef --- /dev/null +++ b/internal/store/migrations/0016_pipeline_v2.sql @@ -0,0 +1,46 @@ +-- +goose Up +-- +goose StatementBegin +-- Pipeline v2: one authoritative mining state row per account and drop. +-- Timestamps are unix seconds, 0 means unset. +CREATE TABLE drop_state ( + account_id TEXT NOT NULL REFERENCES accounts(id) ON DELETE CASCADE, + drop_id TEXT NOT NULL, + campaign_id TEXT NOT NULL, + platform TEXT NOT NULL, + status TEXT NOT NULL, + block_reason TEXT NOT NULL DEFAULT '', + minutes INTEGER NOT NULL DEFAULT 0, + required INTEGER NOT NULL DEFAULT 0, + source TEXT NOT NULL DEFAULT 'platform', + fail_count INTEGER NOT NULL DEFAULT 0, + retry_after INTEGER NOT NULL DEFAULT 0, + synced_at INTEGER NOT NULL DEFAULT 0, + updated_at INTEGER NOT NULL, + PRIMARY KEY (account_id, drop_id) +); +CREATE INDEX idx_drop_state_status ON drop_state(account_id, status); + +-- Ordered per-account priority streamers. Seeded from account_channels, +-- the null-game channel whitelist, which is the same concept. +CREATE TABLE account_streamer_priority ( + account_id TEXT NOT NULL REFERENCES accounts(id) ON DELETE CASCADE, + platform TEXT NOT NULL, + login TEXT NOT NULL, + rank INTEGER NOT NULL, + PRIMARY KEY (account_id, login) +); +CREATE INDEX idx_account_streamer_priority_acct ON account_streamer_priority(account_id, rank); + +INSERT INTO account_streamer_priority (account_id, platform, login, rank) +SELECT ac.account_id, a.platform, ac.channel, ac.rank +FROM account_channels ac +JOIN accounts a ON a.id = ac.account_id; +-- +goose StatementEnd + +-- +goose Down +-- +goose StatementBegin +DROP INDEX IF EXISTS idx_account_streamer_priority_acct; +DROP TABLE IF EXISTS account_streamer_priority; +DROP INDEX IF EXISTS idx_drop_state_status; +DROP TABLE IF EXISTS drop_state; +-- +goose StatementEnd diff --git a/internal/store/queries/drop_state.sql b/internal/store/queries/drop_state.sql new file mode 100644 index 0000000..c1132cb --- /dev/null +++ b/internal/store/queries/drop_state.sql @@ -0,0 +1,20 @@ +-- name: ListDropStates :many +SELECT * FROM drop_state WHERE account_id = ? ORDER BY campaign_id, drop_id; + +-- name: UpsertDropState :exec +INSERT INTO drop_state ( + account_id, drop_id, campaign_id, platform, status, block_reason, + minutes, required, source, fail_count, retry_after, synced_at, updated_at +) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) +ON CONFLICT(account_id, drop_id) DO UPDATE SET + campaign_id = excluded.campaign_id, + platform = excluded.platform, + status = excluded.status, + block_reason = excluded.block_reason, + minutes = excluded.minutes, + required = excluded.required, + source = excluded.source, + fail_count = excluded.fail_count, + retry_after = excluded.retry_after, + synced_at = excluded.synced_at, + updated_at = excluded.updated_at; diff --git a/internal/store/queries/streamer_priority.sql b/internal/store/queries/streamer_priority.sql new file mode 100644 index 0000000..d34e63b --- /dev/null +++ b/internal/store/queries/streamer_priority.sql @@ -0,0 +1,4 @@ +-- name: ListStreamerPriority :many +SELECT login, rank FROM account_streamer_priority +WHERE account_id = ? +ORDER BY rank ASC, login ASC; From d3346dd076aef72497f73a4ff0164089c27631e4 Mon Sep 17 00:00:00 2001 From: Alejandro Date: Sun, 27 Sep 2026 17:26:20 -0400 Subject: [PATCH 07/56] feat(pipeline-v2): backfill drop_state from claims and v1 overrides Co-Authored-By: Claude Sonnet 5 --- internal/store/dropstate_backfill.go | 88 +++++++++++++++++++++++ internal/store/dropstate_backfill_test.go | 67 +++++++++++++++++ internal/store/gen/drop_state.sql.go | 61 ++++++++++++++++ internal/store/gen/querier.go | 2 + internal/store/queries/drop_state.sql | 12 ++++ 5 files changed, 230 insertions(+) create mode 100644 internal/store/dropstate_backfill.go create mode 100644 internal/store/dropstate_backfill_test.go diff --git a/internal/store/dropstate_backfill.go b/internal/store/dropstate_backfill.go new file mode 100644 index 0000000..63bf382 --- /dev/null +++ b/internal/store/dropstate_backfill.go @@ -0,0 +1,88 @@ +package store + +import ( + "context" + "database/sql" + "fmt" + "strings" + "time" + + "github.com/aalejandrofer/grubdrops/internal/pipeline/dropstate" + "github.com/aalejandrofer/grubdrops/internal/store/gen" +) + +// DropStateBackfilledKey marks the one-shot v1 to v2 state backfill as done. +const DropStateBackfilledKey = "drop_state_backfilled" + +// BackfillDropState seeds drop_state from v1 state: ghost-skips become +// not_enrolled with an immediate retry, manual marks become user claims, and +// claim history becomes platform claims. Later sources win. Runs once; the v1 +// kv keys are left in place so rolling back to v1 is safe. +func BackfillDropState(ctx context.Context, q *gen.Queries, now time.Time) (int, error) { + if v, err := q.GetSettingString(ctx, DropStateBackfilledKey); err == nil && string(v) == "1" { + return 0, nil + } + rows := map[string]dropstate.Row{} + put := func(r dropstate.Row) { rows[r.AccountID+"|"+r.DropID] = r } + + fromKV := func(prefix string, mk func(acct, drop, campaign, plat string) dropstate.Row) error { + kvs, err := q.ListKVByPrefix(ctx, sql.NullString{String: prefix, Valid: true}) + if err != nil { + return fmt.Errorf("list %s: %w", prefix, err) + } + for _, kv := range kvs { + drop, acct, ok := splitOverrideKey(kv.Key, prefix) + if !ok { + continue + } + bc, err := q.GetBenefitCampaign(ctx, drop) + if err != nil { + continue // benefit row gone, nothing to seed + } + put(mk(acct, drop, bc.CampaignID, bc.Platform)) + } + return nil + } + if err := fromKV(SkipOverridePrefix, func(acct, drop, campaign, plat string) dropstate.Row { + return dropstate.Row{AccountID: acct, DropID: drop, CampaignID: campaign, Platform: plat, + Status: dropstate.Blocked, Reason: dropstate.NotEnrolled, Source: dropstate.FromPlatform, + RetryAfter: now, UpdatedAt: now} + }); err != nil { + return 0, err + } + if err := fromKV(CollectOverridePrefix, func(acct, drop, campaign, plat string) dropstate.Row { + return dropstate.Row{AccountID: acct, DropID: drop, CampaignID: campaign, Platform: plat, + Status: dropstate.Claimed, Source: dropstate.FromUser, UpdatedAt: now} + }); err != nil { + return 0, err + } + claims, err := q.ListClaimsForBackfill(ctx) + if err != nil { + return 0, fmt.Errorf("list claims: %w", err) + } + for _, c := range claims { + put(dropstate.Row{AccountID: c.AccountID, DropID: c.BenefitID, CampaignID: c.CampaignID, Platform: c.Platform, + Status: dropstate.Claimed, Source: dropstate.FromPlatform, UpdatedAt: now}) + } + + st := NewDropStateStore(q) + for _, r := range rows { + if err := st.Upsert(ctx, r); err != nil { + return 0, fmt.Errorf("upsert %s/%s: %w", r.AccountID, r.DropID, err) + } + } + if err := q.UpsertSettingString(ctx, gen.UpsertSettingStringParams{Key: DropStateBackfilledKey, Value: []byte("1")}); err != nil { + return 0, fmt.Errorf("mark backfilled: %w", err) + } + return len(rows), nil +} + +// splitOverrideKey parses prefix + dropID + ":" + accountID. +func splitOverrideKey(key, prefix string) (drop, acct string, ok bool) { + rest := strings.TrimPrefix(key, prefix) + i := strings.LastIndex(rest, ":") + if i <= 0 || i == len(rest)-1 { + return "", "", false + } + return rest[:i], rest[i+1:], true +} diff --git a/internal/store/dropstate_backfill_test.go b/internal/store/dropstate_backfill_test.go new file mode 100644 index 0000000..b35d33e --- /dev/null +++ b/internal/store/dropstate_backfill_test.go @@ -0,0 +1,67 @@ +package store + +import ( + "context" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/aalejandrofer/grubdrops/internal/pipeline/dropstate" + "github.com/aalejandrofer/grubdrops/internal/platform" + "github.com/aalejandrofer/grubdrops/internal/store/gen" +) + +func TestBackfillDropState(t *testing.T) { + db := openTest(t) + q := gen.New(db) + ctx := context.Background() + seedAccount(t, q, "acc-1", "twitch") + require.NoError(t, NewCampaignPersister(q).PersistCampaigns(ctx, []platform.Campaign{{ + ID: "c1", Platform: "twitch", Game: "G", Name: "C", Status: "active", + Benefits: []platform.DropBenefit{ + {ID: "claimed1", CampaignID: "c1", Name: "A", RequiredMinutes: 60}, + {ID: "marked1", CampaignID: "c1", Name: "B", RequiredMinutes: 60}, + {ID: "skipped1", CampaignID: "c1", Name: "C", RequiredMinutes: 60}, + {ID: "both1", CampaignID: "c1", Name: "D", RequiredMinutes: 60}, + }, + }})) + rec := NewClaimRecorder(q) + require.NoError(t, rec.RecordClaim(ctx, "acc-1", platform.DropBenefit{ID: "claimed1"})) + require.NoError(t, rec.RecordClaim(ctx, "acc-1", platform.DropBenefit{ID: "both1"})) + for _, k := range []string{ + CollectOverridePrefix + "marked1:acc-1", + SkipOverridePrefix + "skipped1:acc-1", + SkipOverridePrefix + "both1:acc-1", // claim history must win over a skip + SkipOverridePrefix + "gone:acc-1", // benefit row missing: ignored + } { + require.NoError(t, q.UpsertSettingString(ctx, gen.UpsertSettingStringParams{Key: k, Value: []byte("1")})) + } + + now := time.Unix(1_700_000_000, 0) + n, err := BackfillDropState(ctx, q, now) + require.NoError(t, err) + assert.Equal(t, 4, n) + + rows, err := NewDropStateStore(q).List(ctx, "acc-1") + require.NoError(t, err) + by := map[string]dropstate.Row{} + for _, r := range rows { + by[r.DropID] = r + } + assert.Equal(t, dropstate.Claimed, by["claimed1"].Status) + assert.Equal(t, dropstate.FromPlatform, by["claimed1"].Source) + assert.Equal(t, dropstate.Claimed, by["marked1"].Status) + assert.Equal(t, dropstate.FromUser, by["marked1"].Source) + assert.Equal(t, dropstate.Blocked, by["skipped1"].Status) + assert.Equal(t, dropstate.NotEnrolled, by["skipped1"].Reason) + assert.Equal(t, now, by["skipped1"].RetryAfter, "skips get one fresh evaluation") + assert.Equal(t, dropstate.Claimed, by["both1"].Status) + assert.Equal(t, "c1", by["claimed1"].CampaignID) + + // Idempotent: second run is a no-op. + n, err = BackfillDropState(ctx, q, now) + require.NoError(t, err) + assert.Equal(t, 0, n) +} diff --git a/internal/store/gen/drop_state.sql.go b/internal/store/gen/drop_state.sql.go index f45d4b7..bf3485b 100644 --- a/internal/store/gen/drop_state.sql.go +++ b/internal/store/gen/drop_state.sql.go @@ -9,6 +9,67 @@ import ( "context" ) +const getBenefitCampaign = `-- name: GetBenefitCampaign :one +SELECT b.campaign_id, cp.platform +FROM benefits b +JOIN campaigns cp ON cp.id = b.campaign_id +WHERE b.id = ? +` + +type GetBenefitCampaignRow struct { + CampaignID string `json:"campaign_id"` + Platform string `json:"platform"` +} + +func (q *Queries) GetBenefitCampaign(ctx context.Context, id string) (GetBenefitCampaignRow, error) { + row := q.db.QueryRowContext(ctx, getBenefitCampaign, id) + var i GetBenefitCampaignRow + err := row.Scan(&i.CampaignID, &i.Platform) + return i, err +} + +const listClaimsForBackfill = `-- name: ListClaimsForBackfill :many +SELECT c.account_id, c.benefit_id, b.campaign_id, a.platform +FROM claims c +JOIN benefits b ON b.id = c.benefit_id +JOIN accounts a ON a.id = c.account_id +` + +type ListClaimsForBackfillRow struct { + AccountID string `json:"account_id"` + BenefitID string `json:"benefit_id"` + CampaignID string `json:"campaign_id"` + Platform string `json:"platform"` +} + +func (q *Queries) ListClaimsForBackfill(ctx context.Context) ([]ListClaimsForBackfillRow, error) { + rows, err := q.db.QueryContext(ctx, listClaimsForBackfill) + if err != nil { + return nil, err + } + defer rows.Close() + var items []ListClaimsForBackfillRow + for rows.Next() { + var i ListClaimsForBackfillRow + if err := rows.Scan( + &i.AccountID, + &i.BenefitID, + &i.CampaignID, + &i.Platform, + ); err != nil { + return nil, err + } + items = append(items, i) + } + if err := rows.Close(); err != nil { + return nil, err + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + const listDropStates = `-- name: ListDropStates :many SELECT account_id, drop_id, campaign_id, platform, status, block_reason, minutes, required, source, fail_count, retry_after, synced_at, updated_at FROM drop_state WHERE account_id = ? ORDER BY campaign_id, drop_id ` diff --git a/internal/store/gen/querier.go b/internal/store/gen/querier.go index 3f51d21..567380d 100644 --- a/internal/store/gen/querier.go +++ b/internal/store/gen/querier.go @@ -49,6 +49,7 @@ type Querier interface { DeleteKV(ctx context.Context, key string) error GetAccount(ctx context.Context, id string) (Account, error) GetAdmin(ctx context.Context) (Admin, error) + GetBenefitCampaign(ctx context.Context, id string) (GetBenefitCampaignRow, error) GetCampaign(ctx context.Context, id string) (Campaign, error) GetProgress(ctx context.Context, arg GetProgressParams) (Progress, error) GetSession(ctx context.Context, accountID string) (Session, error) @@ -68,6 +69,7 @@ type Querier interface { // skips re-mining these: a claim is keyed by benefit id which is unique per // drop instance, so owning a claim means THIS exact drop is done. ListClaimedBenefitIDsForAccount(ctx context.Context, accountID string) ([]string, error) + ListClaimsForBackfill(ctx context.Context) ([]ListClaimsForBackfillRow, error) // Which accounts have claimed each benefit in a campaign. Powers the // per-account COLLECTED marks on the /drops expanded item list. ListClaimsForCampaign(ctx context.Context, campaignID string) ([]ListClaimsForCampaignRow, error) diff --git a/internal/store/queries/drop_state.sql b/internal/store/queries/drop_state.sql index c1132cb..c545b5a 100644 --- a/internal/store/queries/drop_state.sql +++ b/internal/store/queries/drop_state.sql @@ -18,3 +18,15 @@ ON CONFLICT(account_id, drop_id) DO UPDATE SET retry_after = excluded.retry_after, synced_at = excluded.synced_at, updated_at = excluded.updated_at; + +-- name: ListClaimsForBackfill :many +SELECT c.account_id, c.benefit_id, b.campaign_id, a.platform +FROM claims c +JOIN benefits b ON b.id = c.benefit_id +JOIN accounts a ON a.id = c.account_id; + +-- name: GetBenefitCampaign :one +SELECT b.campaign_id, cp.platform +FROM benefits b +JOIN campaigns cp ON cp.id = b.campaign_id +WHERE b.id = ?; From aeb04e1b88b37829ccad1615331fe747d23cdb38 Mon Sep 17 00:00:00 2001 From: Alejandro Date: Sun, 27 Sep 2026 17:32:08 -0400 Subject: [PATCH 08/56] fix(pipeline-v2): backfill skips overrides for deleted accounts Co-Authored-By: Claude Sonnet 5 --- internal/store/dropstate_backfill.go | 12 +++++++- internal/store/dropstate_backfill_test.go | 36 +++++++++++++++++++++++ 2 files changed, 47 insertions(+), 1 deletion(-) diff --git a/internal/store/dropstate_backfill.go b/internal/store/dropstate_backfill.go index 63bf382..b871fcc 100644 --- a/internal/store/dropstate_backfill.go +++ b/internal/store/dropstate_backfill.go @@ -3,6 +3,7 @@ package store import ( "context" "database/sql" + "errors" "fmt" "strings" "time" @@ -35,9 +36,18 @@ func BackfillDropState(ctx context.Context, q *gen.Queries, now time.Time) (int, if !ok { continue } + if _, err := q.GetAccount(ctx, acct); err != nil { + if errors.Is(err, sql.ErrNoRows) { + continue // account deleted, kv leftover: nothing to seed + } + return fmt.Errorf("get account %s: %w", acct, err) + } bc, err := q.GetBenefitCampaign(ctx, drop) if err != nil { - continue // benefit row gone, nothing to seed + if errors.Is(err, sql.ErrNoRows) { + continue // benefit row gone, nothing to seed + } + return fmt.Errorf("get benefit campaign %s: %w", drop, err) } put(mk(acct, drop, bc.CampaignID, bc.Platform)) } diff --git a/internal/store/dropstate_backfill_test.go b/internal/store/dropstate_backfill_test.go index b35d33e..7627989 100644 --- a/internal/store/dropstate_backfill_test.go +++ b/internal/store/dropstate_backfill_test.go @@ -65,3 +65,39 @@ func TestBackfillDropState(t *testing.T) { require.NoError(t, err) assert.Equal(t, 0, n) } + +func TestBackfillDropState_SkipsDeletedAccount(t *testing.T) { + db := openTest(t) + q := gen.New(db) + ctx := context.Background() + seedAccount(t, q, "acc-1", "twitch") + require.NoError(t, NewCampaignPersister(q).PersistCampaigns(ctx, []platform.Campaign{{ + ID: "c1", Platform: "twitch", Game: "G", Name: "C", Status: "active", + Benefits: []platform.DropBenefit{ + {ID: "valid1", CampaignID: "c1", Name: "A", RequiredMinutes: 60}, + }, + }})) + for _, k := range []string{ + SkipOverridePrefix + "valid1:acc-1", + SkipOverridePrefix + "valid1:acc-deleted", + CollectOverridePrefix + "valid1:acc-deleted", + } { + require.NoError(t, q.UpsertSettingString(ctx, gen.UpsertSettingStringParams{Key: k, Value: []byte("1")})) + } + + now := time.Unix(1_700_000_000, 0) + n, err := BackfillDropState(ctx, q, now) + require.NoError(t, err) + assert.Equal(t, 1, n) + + rows, err := NewDropStateStore(q).List(ctx, "acc-1") + require.NoError(t, err) + require.Len(t, rows, 1) + assert.Equal(t, "valid1", rows[0].DropID) + assert.Equal(t, dropstate.Blocked, rows[0].Status) + + // Idempotent: the flag was still set despite the orphaned overrides. + n, err = BackfillDropState(ctx, q, now) + require.NoError(t, err) + assert.Equal(t, 0, n) +} From 8b3486b607c745d0e5a4d97df6edea3d6e1909f3 Mon Sep 17 00:00:00 2001 From: Alejandro Date: Sun, 27 Sep 2026 17:34:14 -0400 Subject: [PATCH 09/56] feat(pipeline-v2): pure planner with streamer priority and swap hysteresis Refs #49 Co-Authored-By: Claude Haiku 4.5 --- internal/pipeline/planner/planner.go | 251 ++++++++++++++++++++++ internal/pipeline/planner/planner_test.go | 163 ++++++++++++++ 2 files changed, 414 insertions(+) create mode 100644 internal/pipeline/planner/planner.go create mode 100644 internal/pipeline/planner/planner_test.go diff --git a/internal/pipeline/planner/planner.go b/internal/pipeline/planner/planner.go new file mode 100644 index 0000000..504859d --- /dev/null +++ b/internal/pipeline/planner/planner.go @@ -0,0 +1,251 @@ +// Package planner turns drop state, campaigns and live channels into one +// decision: which channel to watch and which drops that serves. Pure: no +// I/O, no clock reads, no goroutines. +package planner + +import ( + "sort" + "strings" + "time" + + "github.com/aalejandrofer/grubdrops/internal/pipeline/dropstate" + "github.com/aalejandrofer/grubdrops/internal/platform" +) + +type Kind int + +const ( + Idle Kind = iota + Mine + ForceWatch +) + +func (k Kind) String() string { + switch k { + case Mine: + return "mine" + case ForceWatch: + return "force_watch" + default: + return "idle" + } +} + +// SwapHold is the minimum time on a channel before the planner switches to +// a better one, unless the current channel is dead or cooling down. +const SwapHold = 10 * time.Minute + +const ( + ReasonNothingToMine = "nothing_to_mine" + ReasonNoLiveChannels = "no_live_channels" +) + +type Settings struct { + GameRank func(game string) int + PriorityMode string + StreamerPriority []string + ForceWatch []platform.Stream +} + +type Input struct { + Now time.Time + Rows map[string]dropstate.Row + Campaigns []platform.Campaign + Settings Settings + Live map[string][]platform.Stream + Cooldowns map[string]time.Time + Current *Decision + LastSwap time.Time +} + +type Decision struct { + Kind Kind + CampaignID string + Channel platform.Stream + Serves []string + Reason string +} + +// Same reports whether two decisions watch the same thing. +func (d Decision) Same(o Decision) bool { + return d.Kind == o.Kind && strings.EqualFold(d.Channel.Channel, o.Channel.Channel) +} + +func key(ch string) string { return strings.ToLower(ch) } + +func cooling(in Input, ch string) bool { + until, ok := in.Cooldowns[key(ch)] + return ok && in.Now.Before(until) +} + +func inWindow(c platform.Campaign, now time.Time) bool { + if c.Status != "" && c.Status != "active" { + return false + } + if !c.StartsAt.IsZero() && now.Before(c.StartsAt) { + return false + } + return c.EndsAt.IsZero() || now.Before(c.EndsAt) +} + +// mineable returns the campaign's mineable drop ids, lowest tier first. +func mineable(in Input, c platform.Campaign) []string { + type d struct { + id string + req int + } + seen := map[string]bool{} + var ds []d + for _, b := range c.Benefits { + if seen[b.ID] || !dropstate.Mineable(in.Rows[b.ID]) { + continue + } + ok := true + for _, p := range b.Preconditions { + if in.Rows[p].Status != dropstate.Claimed { + ok = false + break + } + } + if !ok { + continue + } + seen[b.ID] = true + ds = append(ds, d{b.ID, b.RequiredMinutes}) + } + sort.SliceStable(ds, func(i, j int) bool { + if ds[i].req != ds[j].req { + return ds[i].req < ds[j].req + } + return ds[i].id < ds[j].id + }) + out := make([]string, len(ds)) + for i, x := range ds { + out[i] = x.id + } + return out +} + +func ends(c platform.Campaign) time.Time { + if c.EndsAt.IsZero() { + return time.Unix(1<<62, 0) + } + return c.EndsAt +} + +// Candidates returns in-window campaigns with at least one mineable drop, +// best first: game rank then ending soonest (ending soonest first when +// PriorityMode is "ending_soonest"). +func Candidates(in Input) []platform.Campaign { + var out []platform.Campaign + for _, c := range in.Campaigns { + if inWindow(c, in.Now) && len(mineable(in, c)) > 0 { + out = append(out, c) + } + } + rank := func(g string) int { + if in.Settings.GameRank == nil { + return 0 + } + return in.Settings.GameRank(g) + } + sort.SliceStable(out, func(i, j int) bool { + a, b := out[i], out[j] + if in.Settings.PriorityMode == "ending_soonest" && !ends(a).Equal(ends(b)) { + return ends(a).Before(ends(b)) + } + if ra, rb := rank(a.Game), rank(b.Game); ra != rb { + return ra < rb + } + if !ends(a).Equal(ends(b)) { + return ends(a).Before(ends(b)) + } + return a.ID < b.ID + }) + return out +} + +func hasChannel(streams []platform.Stream, ch string) bool { + for _, s := range streams { + if strings.EqualFold(s.Channel, ch) { + return true + } + } + return false +} + +func servedBy(in Input, cands []platform.Campaign, ch string) []string { + var out []string + for _, c := range cands { + if hasChannel(in.Live[c.ID], ch) { + out = append(out, mineable(in, c)...) + } + } + return out +} + +func pickChannel(in Input, c platform.Campaign) (platform.Stream, bool) { + prio := map[string]int{} + for i, l := range in.Settings.StreamerPriority { + if _, dup := prio[key(l)]; !dup { + prio[key(l)] = i + } + } + live := append([]platform.Stream(nil), in.Live[c.ID]...) + sort.SliceStable(live, func(i, j int) bool { + pi, iok := prio[key(live[i].Channel)] + pj, jok := prio[key(live[j].Channel)] + if iok != jok { + return iok + } + if iok && pi != pj { + return pi < pj + } + return live[i].ViewerCount > live[j].ViewerCount + }) + for _, s := range live { + if !cooling(in, s.Channel) { + return s, true + } + } + return platform.Stream{}, false +} + +func forceOrIdle(in Input, reason string) Decision { + for _, s := range in.Settings.ForceWatch { + if !cooling(in, s.Channel) { + return Decision{Kind: ForceWatch, Channel: s, Reason: "force-watch " + s.Channel} + } + } + return Decision{Kind: Idle, Reason: reason} +} + +// Plan decides what the account should watch now. +func Plan(in Input) Decision { + cands := Candidates(in) + if len(cands) == 0 { + return forceOrIdle(in, ReasonNothingToMine) + } + if cur := in.Current; cur != nil && cur.Kind == Mine && in.Now.Sub(in.LastSwap) < SwapHold && !cooling(in, cur.Channel.Channel) { + if serves := servedBy(in, cands, cur.Channel.Channel); len(serves) > 0 { + d := *cur + d.Serves, d.Reason = serves, "holding current channel" + return d + } + } + for _, c := range cands { + ch, ok := pickChannel(in, c) + if !ok { + continue + } + reason := "top viewers" + for _, l := range in.Settings.StreamerPriority { + if strings.EqualFold(l, ch.Channel) { + reason = "priority streamer" + break + } + } + return Decision{Kind: Mine, CampaignID: c.ID, Channel: ch, Serves: servedBy(in, cands, ch.Channel), Reason: reason} + } + return forceOrIdle(in, ReasonNoLiveChannels) +} diff --git a/internal/pipeline/planner/planner_test.go b/internal/pipeline/planner/planner_test.go new file mode 100644 index 0000000..89e0d48 --- /dev/null +++ b/internal/pipeline/planner/planner_test.go @@ -0,0 +1,163 @@ +package planner + +import ( + "testing" + "time" + + "github.com/stretchr/testify/assert" + + "github.com/aalejandrofer/grubdrops/internal/pipeline/dropstate" + "github.com/aalejandrofer/grubdrops/internal/platform" +) + +var now = time.Unix(1_700_000_000, 0) + +func camp(id, game string, ends time.Time, drops ...platform.DropBenefit) platform.Campaign { + return platform.Campaign{ID: id, Game: game, Status: "active", EndsAt: ends, Benefits: drops} +} + +func drop(id, campaign string, req int, pre ...string) platform.DropBenefit { + return platform.DropBenefit{ID: id, CampaignID: campaign, RequiredMinutes: req, Preconditions: pre} +} + +func rows(rs ...dropstate.Row) map[string]dropstate.Row { + m := map[string]dropstate.Row{} + for _, r := range rs { + m[r.DropID] = r + } + return m +} + +func st(id string, s dropstate.Status) dropstate.Row { return dropstate.Row{DropID: id, Status: s} } + +func stream(ch string, viewers int) platform.Stream { + return platform.Stream{Channel: ch, ViewerCount: viewers} +} + +func TestPlan_PicksHighestViewerByDefault(t *testing.T) { + in := Input{Now: now, + Campaigns: []platform.Campaign{camp("c1", "G", time.Time{}, drop("d1", "c1", 60))}, + Rows: rows(st("d1", dropstate.Eligible)), + Live: map[string][]platform.Stream{"c1": {stream("small", 5), stream("big", 500)}}, + } + d := Plan(in) + assert.Equal(t, Mine, d.Kind) + assert.Equal(t, "big", d.Channel.Channel) + assert.Equal(t, []string{"d1"}, d.Serves) +} + +func TestPlan_PriorityStreamerFirst(t *testing.T) { + in := Input{Now: now, + Campaigns: []platform.Campaign{camp("c1", "G", time.Time{}, drop("d1", "c1", 60))}, + Rows: rows(st("d1", dropstate.Eligible)), + Live: map[string][]platform.Stream{"c1": {stream("big", 500), stream("Fav2", 1), stream("fav1", 2)}}, + Settings: Settings{StreamerPriority: []string{"fav1", "fav2"}}, + } + assert.Equal(t, "fav1", Plan(in).Channel.Channel) + in.Live["c1"] = []platform.Stream{stream("big", 500), stream("Fav2", 1)} + assert.Equal(t, "Fav2", Plan(in).Channel.Channel, "case-insensitive, next priority") + in.Live["c1"] = []platform.Stream{stream("big", 500)} + assert.Equal(t, "big", Plan(in).Channel.Channel, "fallback to category") +} + +func TestPlan_GameRankThenEndingSoonest(t *testing.T) { + soon, late := now.Add(time.Hour), now.Add(48*time.Hour) + in := Input{Now: now, + Campaigns: []platform.Campaign{camp("late", "A", late, drop("d1", "late", 60)), camp("soon", "B", soon, drop("d2", "soon", 60))}, + Rows: rows(st("d1", dropstate.Eligible), st("d2", dropstate.Eligible)), + Live: map[string][]platform.Stream{"late": {stream("x", 1)}, "soon": {stream("y", 1)}}, + Settings: Settings{GameRank: func(g string) int { return map[string]int{"A": 0, "B": 1}[g] }}, + } + assert.Equal(t, "late", Plan(in).CampaignID, "ordered: game rank wins") + in.Settings.PriorityMode = "ending_soonest" + assert.Equal(t, "soon", Plan(in).CampaignID) +} + +func TestPlan_SkipsNonMineableAndLockedPreconditions(t *testing.T) { + in := Input{Now: now, + Campaigns: []platform.Campaign{camp("c1", "G", time.Time{}, + drop("claimed", "c1", 30), drop("locked", "c1", 60, "gate"), drop("gate", "c1", 10))}, + Rows: rows(st("claimed", dropstate.Claimed), st("locked", dropstate.Eligible), st("gate", dropstate.Blocked)), + Live: map[string][]platform.Stream{"c1": {stream("x", 1)}}, + } + assert.Equal(t, Idle, Plan(in).Kind, "gate is blocked, locked waits on it") + + in.Rows["gate"] = dropstate.Row{DropID: "gate", Status: dropstate.Claimed, Source: dropstate.FromUser} + d := Plan(in) + assert.Equal(t, Mine, d.Kind, "user-marked precondition unlocks (P9)") + assert.Equal(t, []string{"locked"}, d.Serves) +} + +func TestPlan_ServesDropsAcrossCampaignsOnSameChannel(t *testing.T) { + in := Input{Now: now, + Campaigns: []platform.Campaign{ + camp("c1", "Rust", time.Time{}, drop("r1", "c1", 60), drop("r2", "c1", 30)), + camp("c2", "Rust", time.Time{}, drop("r3", "c2", 120)), + }, + Rows: rows(st("r1", dropstate.Eligible), st("r2", dropstate.Accruing), st("r3", dropstate.Eligible)), + Live: map[string][]platform.Stream{"c1": {stream("oilrats", 9)}, "c2": {stream("OILRATS", 9)}}, + } + d := Plan(in) + assert.Equal(t, []string{"r2", "r1", "r3"}, d.Serves, "lowest tier first, then other campaigns") +} + +func TestPlan_CooldownSkipsChannel(t *testing.T) { + in := Input{Now: now, + Campaigns: []platform.Campaign{camp("c1", "G", time.Time{}, drop("d1", "c1", 60))}, + Rows: rows(st("d1", dropstate.Eligible)), + Live: map[string][]platform.Stream{"c1": {stream("big", 500), stream("small", 5)}}, + Cooldowns: map[string]time.Time{"big": now.Add(time.Minute)}, + } + assert.Equal(t, "small", Plan(in).Channel.Channel) +} + +func TestPlan_HysteresisHoldsThenBypassesDeadChannel(t *testing.T) { + cur := Decision{Kind: Mine, CampaignID: "c1", Channel: stream("big", 500), Serves: []string{"d1"}} + in := Input{Now: now, + Campaigns: []platform.Campaign{camp("c1", "G", time.Time{}, drop("d1", "c1", 60))}, + Rows: rows(st("d1", dropstate.Eligible)), + Live: map[string][]platform.Stream{"c1": {stream("big", 500), stream("fav", 1)}}, + Settings: Settings{StreamerPriority: []string{"fav"}}, + Current: &cur, LastSwap: now.Add(-time.Minute), + } + assert.Equal(t, "big", Plan(in).Channel.Channel, "inside hold: keep current") + in.LastSwap = now.Add(-SwapHold - time.Second) + assert.Equal(t, "fav", Plan(in).Channel.Channel, "hold over: priority streamer wins") + in.LastSwap = now.Add(-time.Minute) + in.Live["c1"] = []platform.Stream{stream("fav", 1)} + assert.Equal(t, "fav", Plan(in).Channel.Channel, "current went offline: bypass hold") +} + +func TestPlan_ForceWatchOnlyWhenNothingToMine(t *testing.T) { + in := Input{Now: now, Settings: Settings{ForceWatch: []platform.Stream{stream("pts", 1)}}} + d := Plan(in) + assert.Equal(t, ForceWatch, d.Kind) + assert.Equal(t, "pts", d.Channel.Channel) + + in.Settings.ForceWatch = nil + d = Plan(in) + assert.Equal(t, Idle, d.Kind) + assert.Equal(t, ReasonNothingToMine, d.Reason) +} + +func TestPlan_NoLiveChannelsIdleReason(t *testing.T) { + in := Input{Now: now, + Campaigns: []platform.Campaign{camp("c1", "G", time.Time{}, drop("d1", "c1", 60))}, + Rows: rows(st("d1", dropstate.Eligible)), + } + assert.Equal(t, ReasonNoLiveChannels, Plan(in).Reason) +} + +func TestPlan_OutsideWindowIgnored(t *testing.T) { + c := camp("c1", "G", now.Add(-time.Minute), drop("d1", "c1", 60)) + in := Input{Now: now, Campaigns: []platform.Campaign{c}, Rows: rows(st("d1", dropstate.Eligible)), + Live: map[string][]platform.Stream{"c1": {stream("x", 1)}}} + assert.Equal(t, Idle, Plan(in).Kind) +} + +func TestDecision_Same(t *testing.T) { + a := Decision{Kind: Mine, Channel: stream("Big", 1)} + assert.True(t, a.Same(Decision{Kind: Mine, Channel: stream("big", 9)})) + assert.False(t, a.Same(Decision{Kind: ForceWatch, Channel: stream("big", 9)})) + assert.True(t, Decision{Kind: Idle}.Same(Decision{Kind: Idle, Reason: "x"})) +} From 01f62d5cba53faf6d83b52758ba47f0f9eda4351 Mon Sep 17 00:00:00 2001 From: Alejandro Date: Sun, 27 Sep 2026 18:00:10 -0400 Subject: [PATCH 10/56] feat(pipeline-v2): accrual session reports progress, stream-down, stall Co-Authored-By: Claude Haiku 4.5 --- internal/pipeline/session/session.go | 107 +++++++++++++++++++ internal/pipeline/session/session_test.go | 122 ++++++++++++++++++++++ 2 files changed, 229 insertions(+) create mode 100644 internal/pipeline/session/session.go create mode 100644 internal/pipeline/session/session_test.go diff --git a/internal/pipeline/session/session.go b/internal/pipeline/session/session.go new file mode 100644 index 0000000..e240500 --- /dev/null +++ b/internal/pipeline/session/session.go @@ -0,0 +1,107 @@ +// Package session runs one accrual session on one channel and reports what +// it sees. It makes no decisions. +package session + +import ( + "context" + "time" + + "github.com/aalejandrofer/grubdrops/internal/platform" +) + +type Kind int + +const ( + Progress Kind = iota + StreamDown + Stalled +) + +type Event struct { + Kind Kind + Channel string + Progress []platform.Progress + Err error +} + +type Config struct { + Backend platform.Backend + Session platform.Session + Stream platform.Stream + Serves []string + // Ticks drives beats. Production passes a 60s ticker: Twitch credits one + // minute per beacon, so a slower cadence under-credits. + Ticks <-chan time.Time + StallPolls int +} + +// Run starts the watch, beats immediately and on every tick, and stops the +// watch when ctx ends or the stream goes down. +func Run(ctx context.Context, cfg Config, out chan<- Event) { + ch := cfg.Stream.Channel + send := func(e Event) { + e.Channel = ch + select { + case out <- e: + case <-ctx.Done(): + } + } + h, err := cfg.Backend.StartWatch(ctx, cfg.Session, cfg.Stream) + if err != nil { + send(Event{Kind: StreamDown, Err: err}) + return + } + defer func() { + stopCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + _ = cfg.Backend.StopWatch(stopCtx, h) + }() + + serves := make(map[string]bool, len(cfg.Serves)) + for _, id := range cfg.Serves { + serves[id] = true + } + best, still, stalled := -1, 0, false + beat := func() bool { + if err := cfg.Backend.Heartbeat(ctx, h); err != nil { + if ctx.Err() == nil { + send(Event{Kind: StreamDown, Err: err}) + } + return false + } + prog, err := cfg.Backend.InventoryProgress(ctx, cfg.Session) + if err != nil { + return true // transient; the next tick retries + } + send(Event{Kind: Progress, Progress: prog}) + total := 0 + for _, p := range prog { + if serves[p.BenefitID] { + total += p.MinutesWatched + } + } + if total > best { + best, still = total, 0 + } else { + still++ + } + if !stalled && cfg.StallPolls > 0 && still >= cfg.StallPolls { + stalled = true + send(Event{Kind: Stalled}) + } + return true + } + if !beat() { + return + } + for { + select { + case <-ctx.Done(): + return + case <-cfg.Ticks: + if !beat() { + return + } + } + } +} diff --git a/internal/pipeline/session/session_test.go b/internal/pipeline/session/session_test.go new file mode 100644 index 0000000..67bf3a7 --- /dev/null +++ b/internal/pipeline/session/session_test.go @@ -0,0 +1,122 @@ +package session + +import ( + "context" + "errors" + "sync" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/aalejandrofer/grubdrops/internal/platform" +) + +type fakeBackend struct { + platform.Backend + mu sync.Mutex + startErr error + beatErr error + minutes []int // minutes for d1 per poll; last value repeats + polls int + beats int + stopped bool +} + +func (f *fakeBackend) StartWatch(_ context.Context, _ platform.Session, s platform.Stream) (platform.WatchHandle, error) { + return platform.WatchHandle{Channel: s.Channel}, f.startErr +} +func (f *fakeBackend) Heartbeat(context.Context, platform.WatchHandle) error { + f.mu.Lock() + defer f.mu.Unlock() + f.beats++ + return f.beatErr +} +func (f *fakeBackend) StopWatch(context.Context, platform.WatchHandle) error { + f.mu.Lock() + defer f.mu.Unlock() + f.stopped = true + return nil +} +func (f *fakeBackend) InventoryProgress(context.Context, platform.Session) ([]platform.Progress, error) { + f.mu.Lock() + defer f.mu.Unlock() + i := f.polls + if i >= len(f.minutes) { + i = len(f.minutes) - 1 + } + f.polls++ + return []platform.Progress{{BenefitID: "d1", MinutesWatched: f.minutes[i]}}, nil +} + +func recv(t *testing.T, ch <-chan Event) Event { + t.Helper() + select { + case e := <-ch: + return e + case <-time.After(2 * time.Second): + t.Fatal("no event") + return Event{} + } +} + +func TestRun_ProgressEachTickAndStopOnCancel(t *testing.T) { + f := &fakeBackend{minutes: []int{1, 2}} + ticks := make(chan time.Time) + out := make(chan Event, 8) + ctx, cancel := context.WithCancel(context.Background()) + done := make(chan struct{}) + go func() { + Run(ctx, Config{Backend: f, Stream: platform.Stream{Channel: "ch"}, Serves: []string{"d1"}, Ticks: ticks, StallPolls: 5}, out) + close(done) + }() + + e := recv(t, out) + assert.Equal(t, Progress, e.Kind, "first beat is immediate") + assert.Equal(t, 1, e.Progress[0].MinutesWatched) + ticks <- time.Now() + assert.Equal(t, 2, recv(t, out).Progress[0].MinutesWatched) + cancel() + <-done + assert.True(t, f.stopped) +} + +func TestRun_HeartbeatErrorIsStreamDown(t *testing.T) { + f := &fakeBackend{minutes: []int{1}, beatErr: errors.New("offline")} + out := make(chan Event, 8) + Run(context.Background(), Config{Backend: f, Stream: platform.Stream{Channel: "ch"}, Ticks: make(chan time.Time)}, out) + e := recv(t, out) + assert.Equal(t, StreamDown, e.Kind) + assert.Equal(t, "ch", e.Channel) + assert.True(t, f.stopped) +} + +func TestRun_StartErrorIsStreamDown(t *testing.T) { + f := &fakeBackend{startErr: errors.New("sidecar")} + out := make(chan Event, 8) + Run(context.Background(), Config{Backend: f, Stream: platform.Stream{Channel: "ch"}, Ticks: make(chan time.Time)}, out) + assert.Equal(t, StreamDown, recv(t, out).Kind) +} + +func TestRun_StalledOnceAfterNoGain(t *testing.T) { + f := &fakeBackend{minutes: []int{5}} + ticks := make(chan time.Time) + out := make(chan Event, 32) + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + go Run(ctx, Config{Backend: f, Stream: platform.Stream{Channel: "ch"}, Serves: []string{"d1"}, Ticks: ticks, StallPolls: 2}, out) + require.Equal(t, Progress, recv(t, out).Kind) // poll 1 sets baseline + ticks <- time.Now() + require.Equal(t, Progress, recv(t, out).Kind) // no gain 1 + ticks <- time.Now() + require.Equal(t, Progress, recv(t, out).Kind) // no gain 2 + assert.Equal(t, Stalled, recv(t, out).Kind) + ticks <- time.Now() + assert.Equal(t, Progress, recv(t, out).Kind) + select { + case e := <-out: + t.Fatalf("unexpected second event %v", e.Kind) + case <-time.After(50 * time.Millisecond): + } +} From 908144079ca3fef45f5c4b6f945214fb502b0566 Mon Sep 17 00:00:00 2001 From: Alejandro Date: Sun, 27 Sep 2026 18:22:28 -0400 Subject: [PATCH 11/56] fix(pipeline-v2): per-drop stall watermark so claimed drops don't fake a stall Co-Authored-By: Claude Opus 5.5 (1M context) --- internal/pipeline/session/session.go | 15 ++++--- internal/pipeline/session/session_test.go | 51 ++++++++++++++++++++++- 2 files changed, 60 insertions(+), 6 deletions(-) mode change 100644 => 100755 internal/pipeline/session/session_test.go diff --git a/internal/pipeline/session/session.go b/internal/pipeline/session/session.go index e240500..385f414 100644 --- a/internal/pipeline/session/session.go +++ b/internal/pipeline/session/session.go @@ -61,7 +61,8 @@ func Run(ctx context.Context, cfg Config, out chan<- Event) { for _, id := range cfg.Serves { serves[id] = true } - best, still, stalled := -1, 0, false + best := make(map[string]int) + still, stalled := 0, false beat := func() bool { if err := cfg.Backend.Heartbeat(ctx, h); err != nil { if ctx.Err() == nil { @@ -74,14 +75,18 @@ func Run(ctx context.Context, cfg Config, out chan<- Event) { return true // transient; the next tick retries } send(Event{Kind: Progress, Progress: prog}) - total := 0 + gained := false for _, p := range prog { if serves[p.BenefitID] { - total += p.MinutesWatched + // First time seeing this id counts as gain (baseline), or if it improved + if _, seen := best[p.BenefitID]; !seen || p.MinutesWatched > best[p.BenefitID] { + best[p.BenefitID] = p.MinutesWatched + gained = true + } } } - if total > best { - best, still = total, 0 + if gained { + still = 0 } else { still++ } diff --git a/internal/pipeline/session/session_test.go b/internal/pipeline/session/session_test.go old mode 100644 new mode 100755 index 67bf3a7..38ca65a --- a/internal/pipeline/session/session_test.go +++ b/internal/pipeline/session/session_test.go @@ -18,7 +18,8 @@ type fakeBackend struct { mu sync.Mutex startErr error beatErr error - minutes []int // minutes for d1 per poll; last value repeats + minutes []int // minutes for d1 per poll; last value repeats + script [][]platform.Progress // scripted responses per poll; overrides minutes polls int beats int stopped bool @@ -42,6 +43,14 @@ func (f *fakeBackend) StopWatch(context.Context, platform.WatchHandle) error { func (f *fakeBackend) InventoryProgress(context.Context, platform.Session) ([]platform.Progress, error) { f.mu.Lock() defer f.mu.Unlock() + if f.script != nil { + i := f.polls + if i >= len(f.script) { + i = len(f.script) - 1 + } + f.polls++ + return f.script[i], nil + } i := f.polls if i >= len(f.minutes) { i = len(f.minutes) - 1 @@ -120,3 +129,43 @@ func TestRun_StalledOnceAfterNoGain(t *testing.T) { case <-time.After(50 * time.Millisecond): } } + +func TestRun_ClaimedDropLeavingInventoryIsNotAStall(t *testing.T) { + // Simulate d1 being claimed and disappearing from inventory while d2 keeps accruing. + // With summed watermark (buggy), the claim would cause false stall. + // With per-drop watermark (fixed), d2's gain prevents stall. + f := &fakeBackend{script: [][]platform.Progress{ + {{BenefitID: "d1", MinutesWatched: 30}, {BenefitID: "d2", MinutesWatched: 1}}, + {{BenefitID: "d1", MinutesWatched: 30}, {BenefitID: "d2", MinutesWatched: 2}}, + {{BenefitID: "d2", MinutesWatched: 3}}, // d1 claimed, absent + {{BenefitID: "d2", MinutesWatched: 4}}, + {{BenefitID: "d2", MinutesWatched: 5}}, + }} + ticks := make(chan time.Time) + out := make(chan Event, 32) + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + go Run(ctx, Config{Backend: f, Stream: platform.Stream{Channel: "ch"}, Serves: []string{"d1", "d2"}, Ticks: ticks, StallPolls: 2}, out) + + // Poll 1: d1=30, d2=1 (total=31), baseline set + require.Equal(t, Progress, recv(t, out).Kind) + // Poll 2: d1=30, d2=2 (total=32), gain + ticks <- time.Now() + require.Equal(t, Progress, recv(t, out).Kind) + // Poll 3: d1 absent, d2=3, gain on d2 despite d1 gone + ticks <- time.Now() + require.Equal(t, Progress, recv(t, out).Kind) + // Poll 4: d2=4, continued gain + ticks <- time.Now() + require.Equal(t, Progress, recv(t, out).Kind) + // Poll 5: d2=5, still gaining + ticks <- time.Now() + require.Equal(t, Progress, recv(t, out).Kind) + + // No Stalled event should arrive + select { + case e := <-out: + t.Fatalf("unexpected stalled event %v", e.Kind) + case <-time.After(50 * time.Millisecond): + } +} From e6f24ee62c5c3ef7ab94bf1d48e63bd84446879b Mon Sep 17 00:00:00 2001 From: Alejandro Date: Sun, 27 Sep 2026 18:26:20 -0400 Subject: [PATCH 12/56] feat(pipeline-v2): platform progress/claim/probe capabilities, Twitch adapter Co-Authored-By: Claude Sonnet 5 --- internal/platform/pipeline.go | 69 ++++++++++++ internal/platform/twitch/campaigns.go | 7 ++ internal/platform/twitch/claim.go | 28 +++-- internal/platform/twitch/pipeline.go | 122 +++++++++++++++++++++ internal/platform/twitch/pipeline_test.go | 123 ++++++++++++++++++++++ 5 files changed, 339 insertions(+), 10 deletions(-) create mode 100644 internal/platform/pipeline.go create mode 100644 internal/platform/twitch/pipeline.go create mode 100644 internal/platform/twitch/pipeline_test.go diff --git a/internal/platform/pipeline.go b/internal/platform/pipeline.go new file mode 100644 index 0000000..6490fcc --- /dev/null +++ b/internal/platform/pipeline.go @@ -0,0 +1,69 @@ +package platform + +import "context" + +// DropProgress is one drop's per-account state as the platform reports it. +// Pipeline v2's reconciler folds these into drop_state. +type DropProgress struct { + DropID string + CampaignID string + Minutes int + Required int + Claimed bool + InstanceID string + // Known is true when the platform returned a definite record for this + // drop. False means the platform said nothing about it (Kick lists a + // reward only after it accrues), which is not evidence of anything. + Known bool + // Unmineable marks drops whose progress the platform cannot track, + // such as Twitch scrape placeholders. The reconciler blocks them. + Unmineable bool +} + +type ClaimOutcome int + +const ( + ClaimOK ClaimOutcome = iota + ClaimAlready + ClaimNeedsLink + ClaimFailed +) + +func (o ClaimOutcome) String() string { + switch o { + case ClaimOK: + return "ok" + case ClaimAlready: + return "already_claimed" + case ClaimNeedsLink: + return "needs_link" + default: + return "failed" + } +} + +// ClaimResult is a typed claim outcome, replacing string-matched errors. +type ClaimResult struct { + Outcome ClaimOutcome + LinkURL string + Detail string +} + +// DropProgressSource reports authoritative per-drop progress for the given +// campaigns. It returns an error rather than a partial result, so a failed +// sync never looks like "the platform forgot these drops". +type DropProgressSource interface { + DropProgress(ctx context.Context, s Session, camps []Campaign) ([]DropProgress, error) +} + +// DropClaimer claims one drop and classifies the outcome. +type DropClaimer interface { + ClaimDrop(ctx context.Context, s Session, d DropProgress) ClaimResult +} + +// ChannelProber checks specific channels for liveness and, when c.Game is +// set, that they stream the campaign's game. Used for priority streamers +// that platform directories may not list. +type ChannelProber interface { + ProbeChannels(ctx context.Context, s Session, c Campaign, logins []string) ([]Stream, error) +} diff --git a/internal/platform/twitch/campaigns.go b/internal/platform/twitch/campaigns.go index 58c7057..d83d241 100644 --- a/internal/platform/twitch/campaigns.go +++ b/internal/platform/twitch/campaigns.go @@ -131,6 +131,13 @@ type campaignDetailsData struct { PreconditionDrops []struct { ID string `json:"id"` } `json:"preconditionDrops"` + // self is the viewer's own progress on this drop. Null when + // the account is not enrolled in the campaign yet. + Self *struct { + CurrentMinutesWatched int `json:"currentMinutesWatched"` + IsClaimed bool `json:"isClaimed"` + DropInstanceID string `json:"dropInstanceID"` + } `json:"self"` } `json:"timeBasedDrops"` } `json:"dropCampaign"` } `json:"user"` diff --git a/internal/platform/twitch/claim.go b/internal/platform/twitch/claim.go index da92e92..1d2bbcf 100644 --- a/internal/platform/twitch/claim.go +++ b/internal/platform/twitch/claim.go @@ -18,6 +18,20 @@ type claimResult struct { } func (cl *claimer) claim(ctx context.Context, sess platform.Session, b platform.DropBenefit, userID int64) error { + status, err := cl.claimStatus(ctx, sess, b, userID) + if err != nil { + return err + } + switch status { + case "ELIGIBLE_FOR_ALL", "DROP_INSTANCE_ALREADY_CLAIMED", "": + return nil + default: + return fmt.Errorf("claim status: %s", status) + } +} + +// claimStatus sends the claim mutation and returns Twitch's raw status. +func (cl *claimer) claimStatus(ctx context.Context, sess platform.Session, b platform.DropBenefit, userID int64) (string, error) { // Prefer the per-account instance id captured at progress time. // When it's missing, construct DevilXD's synthetic instance id // `userID#campaignID#dropID` (inventory.py generate_claim) — Twitch @@ -32,15 +46,9 @@ func (cl *claimer) claim(ctx context.Context, sess platform.Session, b platform. id = b.ID } var out claimResult - err := cl.c.gql(ctx, sess.AccessToken, OpClaimDrop, - map[string]any{"input": map[string]any{"dropInstanceID": id}}, &out) - if err != nil { - return fmt.Errorf("claim %s: %w", id, err) - } - switch out.ClaimDropRewards.Status { - case "ELIGIBLE_FOR_ALL", "DROP_INSTANCE_ALREADY_CLAIMED", "": - return nil - default: - return fmt.Errorf("claim status: %s", out.ClaimDropRewards.Status) + if err := cl.c.gql(ctx, sess.AccessToken, OpClaimDrop, + map[string]any{"input": map[string]any{"dropInstanceID": id}}, &out); err != nil { + return "", fmt.Errorf("claim %s: %w", id, err) } + return out.ClaimDropRewards.Status, nil } diff --git a/internal/platform/twitch/pipeline.go b/internal/platform/twitch/pipeline.go new file mode 100644 index 0000000..a945a6f --- /dev/null +++ b/internal/platform/twitch/pipeline.go @@ -0,0 +1,122 @@ +package twitch + +import ( + "context" + "fmt" + "strings" + + "github.com/aalejandrofer/grubdrops/internal/platform" +) + +// isSyntheticID reports scrape-fallback placeholder ids ("Game|Name"). +// Twitch can't report progress for them. +func isSyntheticID(id string) bool { return strings.ContainsAny(id, "| ") } + +// dropProgress reads DropCampaignDetails uncached, because self changes as +// the viewer watches and claims. One entry per drop id. +func (d *discovery) dropProgress(ctx context.Context, sess platform.Session, campaignID string) ([]platform.DropProgress, error) { + channelLogin := d.userLogin + if channelLogin == "" { + channelLogin = "twitch" + } + var det campaignDetailsData + if err := d.c.gql(ctx, sess.AccessToken, OpDropCampaignDetails, + map[string]any{"dropID": campaignID, "channelLogin": channelLogin}, &det); err != nil { + return nil, fmt.Errorf("drop progress %s: %w", campaignID, err) + } + seen := map[string]bool{} + var out []platform.DropProgress + for _, td := range det.User.DropCampaign.TimeBasedDrops { + if td.ID == "" || seen[td.ID] { + continue + } + seen[td.ID] = true + dp := platform.DropProgress{DropID: td.ID, CampaignID: campaignID, Required: td.RequiredMinutesWatched, Known: true} + if td.Self != nil { + dp.Minutes = td.Self.CurrentMinutesWatched + dp.Claimed = td.Self.IsClaimed + dp.InstanceID = td.Self.DropInstanceID + } + out = append(out, dp) + } + return out, nil +} + +// DropProgress satisfies platform.DropProgressSource. Per-campaign details +// are the truth (they survive the campaign leaving the in-progress +// inventory); the in-progress inventory only fills fresher minutes and the +// instance id. +func (b *Backend) DropProgress(ctx context.Context, s platform.Session, camps []platform.Campaign) ([]platform.DropProgress, error) { + inv, err := b.disc.inventory(ctx, s) + if err != nil { + return nil, err + } + byID := make(map[string]platform.Progress, len(inv)) + for _, p := range inv { + byID[p.BenefitID] = p + } + var out []platform.DropProgress + for _, c := range camps { + if isSyntheticID(c.ID) { + for _, bf := range c.Benefits { + out = append(out, platform.DropProgress{DropID: bf.ID, CampaignID: c.ID, Known: true, Unmineable: true}) + } + continue + } + dps, err := b.disc.dropProgress(ctx, s, c.ID) + if err != nil { + return nil, err + } + for i := range dps { + p, ok := byID[dps[i].DropID] + if !ok { + continue + } + if p.MinutesWatched > dps[i].Minutes { + dps[i].Minutes = p.MinutesWatched + } + dps[i].Claimed = dps[i].Claimed || p.Claimed + if dps[i].InstanceID == "" { + dps[i].InstanceID = p.InstanceID + } + } + out = append(out, dps...) + } + return out, nil +} + +// classifyClaimStatus maps Twitch's claimDropRewards.status to an outcome. +func classifyClaimStatus(status string) platform.ClaimResult { + switch status { + case "ELIGIBLE_FOR_ALL", "": + return platform.ClaimResult{Outcome: platform.ClaimOK} + case "DROP_INSTANCE_ALREADY_CLAIMED": + return platform.ClaimResult{Outcome: platform.ClaimAlready} + default: + return platform.ClaimResult{Outcome: platform.ClaimFailed, Detail: "claim status: " + status} + } +} + +// ClaimDrop satisfies platform.DropClaimer. Unlinked campaigns are blocked +// before claiming (reconciler reads self.isAccountConnected), so Twitch has +// no link outcome here. +func (b *Backend) ClaimDrop(ctx context.Context, s platform.Session, d platform.DropProgress) platform.ClaimResult { + userID, _ := b.watch.resolveUserID(ctx, s) + status, err := b.claim.claimStatus(ctx, s, platform.DropBenefit{ID: d.DropID, CampaignID: d.CampaignID, InstanceID: d.InstanceID}, userID) + if err != nil { + return platform.ClaimResult{Outcome: platform.ClaimFailed, Detail: err.Error()} + } + return classifyClaimStatus(status) +} + +// ProbeChannels satisfies platform.ChannelProber: parallel live check of the +// given logins, filtered to c.Game when set. +func (b *Backend) ProbeChannels(ctx context.Context, s platform.Session, c platform.Campaign, logins []string) ([]platform.Stream, error) { + return b.chans.listEligible(ctx, s, c, logins) +} + +var ( + _ platform.DropProgressSource = (*Backend)(nil) + _ platform.DropClaimer = (*Backend)(nil) + _ platform.ChannelProber = (*Backend)(nil) +) diff --git a/internal/platform/twitch/pipeline_test.go b/internal/platform/twitch/pipeline_test.go new file mode 100644 index 0000000..68051a3 --- /dev/null +++ b/internal/platform/twitch/pipeline_test.go @@ -0,0 +1,123 @@ +package twitch + +import ( + "context" + "encoding/json" + "net/http" + "net/http/httptest" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/aalejandrofer/grubdrops/internal/platform" +) + +// pipelineTestBackend serves DropCampaignDetails + Inventory from fixtures. +func pipelineTestBackend(t *testing.T, details map[string]string, inventory string, detailsStatus int) *Backend { + t.Helper() + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + var req gqlRequest + require.NoError(t, json.NewDecoder(r.Body).Decode(&req)) + switch req.OperationName { + case OpDropCampaignDetails.Name: + if detailsStatus != 0 { + w.WriteHeader(detailsStatus) + return + } + id, _ := req.Variables["dropID"].(string) + _, _ = w.Write([]byte(details[id])) + case OpInventory.Name: + _, _ = w.Write([]byte(inventory)) + default: + t.Fatalf("unexpected op %q", req.OperationName) + } + })) + t.Cleanup(srv.Close) + return &Backend{disc: &discovery{c: newTestClient(srv.URL), userLogin: "testuser"}} +} + +const emptyInventory = `{"data":{"currentUser":{"inventory":{"dropCampaignsInProgress":[]}}}}` + +func TestDropProgress_ReadsSelfForCompletedCampaign(t *testing.T) { + // Campaign left dropCampaignsInProgress (fully claimed), but details self + // still reports isClaimed. This is the P1 fix. + b := pipelineTestBackend(t, map[string]string{"c1": `{"data":{"user":{"dropCampaign":{"timeBasedDrops":[ + {"id":"d1","requiredMinutesWatched":60,"benefitEdges":[],"self":{"currentMinutesWatched":60,"isClaimed":true,"dropInstanceID":"i1"}} + ]}}}}`}, emptyInventory, 0) + got, err := b.DropProgress(context.Background(), platform.Session{AccessToken: "t"}, + []platform.Campaign{{ID: "c1", Platform: "twitch"}}) + require.NoError(t, err) + require.Len(t, got, 1) + assert.Equal(t, platform.DropProgress{DropID: "d1", CampaignID: "c1", Minutes: 60, Required: 60, Claimed: true, InstanceID: "i1", Known: true}, got[0]) +} + +func TestDropProgress_NullSelfIsUnclaimed(t *testing.T) { + b := pipelineTestBackend(t, map[string]string{"c1": `{"data":{"user":{"dropCampaign":{"timeBasedDrops":[ + {"id":"d1","requiredMinutesWatched":60,"benefitEdges":[],"self":null} + ]}}}}`}, emptyInventory, 0) + got, err := b.DropProgress(context.Background(), platform.Session{AccessToken: "t"}, + []platform.Campaign{{ID: "c1", Platform: "twitch"}}) + require.NoError(t, err) + require.Len(t, got, 1) + assert.False(t, got[0].Claimed) + assert.Equal(t, 0, got[0].Minutes) + assert.True(t, got[0].Known) +} + +func TestDropProgress_DedupesMultiItemDrop(t *testing.T) { + b := pipelineTestBackend(t, map[string]string{"c1": `{"data":{"user":{"dropCampaign":{"timeBasedDrops":[ + {"id":"d1","requiredMinutesWatched":60,"benefitEdges":[{"benefit":{"id":"b1","name":"A"}},{"benefit":{"id":"b2","name":"B"}}],"self":null}, + {"id":"d1","requiredMinutesWatched":60,"benefitEdges":[],"self":null} + ]}}}}`}, emptyInventory, 0) + got, err := b.DropProgress(context.Background(), platform.Session{AccessToken: "t"}, + []platform.Campaign{{ID: "c1", Platform: "twitch"}}) + require.NoError(t, err) + assert.Len(t, got, 1) +} + +func TestDropProgress_InventoryFillsInstanceAndMinutes(t *testing.T) { + b := pipelineTestBackend(t, map[string]string{"c1": `{"data":{"user":{"dropCampaign":{"timeBasedDrops":[ + {"id":"d1","requiredMinutesWatched":60,"benefitEdges":[],"self":{"currentMinutesWatched":10,"isClaimed":false,"dropInstanceID":""}} + ]}}}}`}, `{"data":{"currentUser":{"inventory":{"dropCampaignsInProgress":[ + {"id":"c1","timeBasedDrops":[{"id":"d1","self":{"currentMinutesWatched":25,"isClaimed":false,"dropInstanceID":"inst"}}]} + ]}}}}`, 0) + got, err := b.DropProgress(context.Background(), platform.Session{AccessToken: "t"}, + []platform.Campaign{{ID: "c1", Platform: "twitch"}}) + require.NoError(t, err) + require.Len(t, got, 1) + assert.Equal(t, 25, got[0].Minutes) + assert.Equal(t, "inst", got[0].InstanceID) +} + +func TestDropProgress_SyntheticCampaignIsUnmineable(t *testing.T) { + b := pipelineTestBackend(t, nil, emptyInventory, 0) + got, err := b.DropProgress(context.Background(), platform.Session{AccessToken: "t"}, + []platform.Campaign{{ID: "Game|Name", Platform: "twitch", Benefits: []platform.DropBenefit{{ID: "Game|Name_default"}}}}) + require.NoError(t, err) + require.Len(t, got, 1) + assert.True(t, got[0].Unmineable) + assert.True(t, got[0].Known) +} + +func TestDropProgress_DetailsErrorFailsWhole(t *testing.T) { + b := pipelineTestBackend(t, nil, emptyInventory, http.StatusInternalServerError) + _, err := b.DropProgress(context.Background(), platform.Session{AccessToken: "t"}, + []platform.Campaign{{ID: "c1", Platform: "twitch"}}) + require.Error(t, err) +} + +func TestClassifyClaimStatus(t *testing.T) { + assert.Equal(t, platform.ClaimOK, classifyClaimStatus("ELIGIBLE_FOR_ALL").Outcome) + assert.Equal(t, platform.ClaimOK, classifyClaimStatus("").Outcome) + assert.Equal(t, platform.ClaimAlready, classifyClaimStatus("DROP_INSTANCE_ALREADY_CLAIMED").Outcome) + r := classifyClaimStatus("DROP_INSTANCE_ALREADY_EXPIRED") + assert.Equal(t, platform.ClaimFailed, r.Outcome) + assert.Contains(t, r.Detail, "DROP_INSTANCE_ALREADY_EXPIRED") +} + +func TestBackend_SatisfiesPipelineInterfaces(t *testing.T) { + var _ platform.DropProgressSource = (*Backend)(nil) + var _ platform.DropClaimer = (*Backend)(nil) + var _ platform.ChannelProber = (*Backend)(nil) +} From bc3aafc102ad8e9a8644fdaaf750230f60931548 Mon Sep 17 00:00:00 2001 From: Alejandro Date: Sun, 27 Sep 2026 19:12:02 -0400 Subject: [PATCH 13/56] test(pipeline-v2): pin inventory claimed flag winning over details Co-Authored-By: Claude Sonnet 5 --- internal/platform/twitch/pipeline.go | 6 ++++-- internal/platform/twitch/pipeline_test.go | 16 ++++++++++++++++ 2 files changed, 20 insertions(+), 2 deletions(-) diff --git a/internal/platform/twitch/pipeline.go b/internal/platform/twitch/pipeline.go index a945a6f..078a5e1 100644 --- a/internal/platform/twitch/pipeline.go +++ b/internal/platform/twitch/pipeline.go @@ -44,8 +44,10 @@ func (d *discovery) dropProgress(ctx context.Context, sess platform.Session, cam // DropProgress satisfies platform.DropProgressSource. Per-campaign details // are the truth (they survive the campaign leaving the in-progress -// inventory); the in-progress inventory only fills fresher minutes and the -// instance id. +// inventory); the in-progress inventory fills fresher minutes and the +// instance id, and a claimed=true from either source wins — isClaimed is +// a positive statement from the same self object and never reverts on +// Twitch. func (b *Backend) DropProgress(ctx context.Context, s platform.Session, camps []platform.Campaign) ([]platform.DropProgress, error) { inv, err := b.disc.inventory(ctx, s) if err != nil { diff --git a/internal/platform/twitch/pipeline_test.go b/internal/platform/twitch/pipeline_test.go index 68051a3..c97e6c1 100644 --- a/internal/platform/twitch/pipeline_test.go +++ b/internal/platform/twitch/pipeline_test.go @@ -90,6 +90,22 @@ func TestDropProgress_InventoryFillsInstanceAndMinutes(t *testing.T) { assert.Equal(t, "inst", got[0].InstanceID) } +func TestDropProgress_InventoryClaimedWins(t *testing.T) { + // Details self says isClaimed:false but inventory reports isClaimed:true + // for the same drop — claimed=true from either source wins because + // isClaimed never reverts on Twitch. + b := pipelineTestBackend(t, map[string]string{"c1": `{"data":{"user":{"dropCampaign":{"timeBasedDrops":[ + {"id":"d1","requiredMinutesWatched":60,"benefitEdges":[],"self":{"currentMinutesWatched":60,"isClaimed":false,"dropInstanceID":"i1"}} + ]}}}}`}, `{"data":{"currentUser":{"inventory":{"dropCampaignsInProgress":[ + {"id":"c1","timeBasedDrops":[{"id":"d1","self":{"currentMinutesWatched":60,"isClaimed":true,"dropInstanceID":"i1"}}]} + ]}}}}`, 0) + got, err := b.DropProgress(context.Background(), platform.Session{AccessToken: "t"}, + []platform.Campaign{{ID: "c1", Platform: "twitch"}}) + require.NoError(t, err) + require.Len(t, got, 1) + assert.True(t, got[0].Claimed) +} + func TestDropProgress_SyntheticCampaignIsUnmineable(t *testing.T) { b := pipelineTestBackend(t, nil, emptyInventory, 0) got, err := b.DropProgress(context.Background(), platform.Session{AccessToken: "t"}, From a0a21bc532410f7ca39466091916ad29af4c2995 Mon Sep 17 00:00:00 2001 From: Alejandro Date: Sun, 27 Sep 2026 19:15:11 -0400 Subject: [PATCH 14/56] feat(pipeline-v2): Kick adapter with platform-sourced required minutes Co-Authored-By: Claude Opus 5.5 (1M context) --- internal/platform/kick/api.go | 2 + internal/platform/kick/pipeline.go | 81 +++++++++++++++++++++++++ internal/platform/kick/pipeline_test.go | 77 +++++++++++++++++++++++ 3 files changed, 160 insertions(+) create mode 100644 internal/platform/kick/pipeline.go create mode 100644 internal/platform/kick/pipeline_test.go diff --git a/internal/platform/kick/api.go b/internal/platform/kick/api.go index 042fd17..2eb3c81 100644 --- a/internal/platform/kick/api.go +++ b/internal/platform/kick/api.go @@ -374,6 +374,7 @@ type progressReward struct { Name string Fraction float64 // 0..1 watched toward this reward Claimed bool + Required int // required_units } // progressDetail returns every in-progress reward with its parent campaign id, @@ -401,6 +402,7 @@ func (a *api) progressDetail(ctx context.Context, sess platform.Session) ([]prog Name: mstr(rm, "name", "title"), Fraction: mfloat(rm, "progress", "progress_fraction"), Claimed: mbool(rm, "claimed", "is_claimed", "isClaimed"), + Required: mnum(rm, "required_units", "required_minutes", "requiredMinutes", "minutes"), }) } } diff --git a/internal/platform/kick/pipeline.go b/internal/platform/kick/pipeline.go new file mode 100644 index 0000000..d1dee22 --- /dev/null +++ b/internal/platform/kick/pipeline.go @@ -0,0 +1,81 @@ +package kick + +import ( + "context" + "errors" + "fmt" + "math" + "strings" + + "github.com/aalejandrofer/grubdrops/internal/platform" +) + +// DropProgress satisfies platform.DropProgressSource. Required minutes come +// from /drops/progress required_units whenever Kick lists the reward. Kick +// lists a reward only after it accrues, so an unlisted reward is reported +// with Known=false instead of guessing. +func (b *Backend) DropProgress(ctx context.Context, s platform.Session, camps []platform.Campaign) ([]platform.DropProgress, error) { + rewards, err := b.api.progressDetail(ctx, s) + if err != nil { + return nil, fmt.Errorf("kick drop progress: %w", err) + } + byID := make(map[string]progressReward, len(rewards)) + for _, r := range rewards { + byID[r.RewardID] = r + } + var out []platform.DropProgress + for _, c := range camps { + for _, bf := range c.Benefits { + r, ok := byID[bf.ID] + if !ok { + out = append(out, platform.DropProgress{DropID: bf.ID, CampaignID: c.ID, Required: bf.RequiredMinutes}) + continue + } + req := r.Required + if req <= 0 { + req = bf.RequiredMinutes + } + minutes := 0 + if req > 0 && r.Fraction > 0 { + minutes = int(math.Round(r.Fraction * float64(req))) + } + out = append(out, platform.DropProgress{ + DropID: bf.ID, CampaignID: c.ID, Minutes: minutes, Required: req, + Claimed: r.Claimed, Known: true, + }) + } + } + return out, nil +} + +// ClaimDrop satisfies platform.DropClaimer. A claim error on an already +// auto-granted reward shows up as ClaimFailed; the post-claim reconcile +// then sees claimed=true and corrects the state. +func (b *Backend) ClaimDrop(ctx context.Context, s platform.Session, d platform.DropProgress) platform.ClaimResult { + err := b.api.Claim(ctx, s, d.DropID, d.CampaignID) + if err == nil { + return platform.ClaimResult{Outcome: platform.ClaimOK} + } + var linkErr *ClaimNeedsLinkError + if errors.As(err, &linkErr) { + return platform.ClaimResult{Outcome: platform.ClaimNeedsLink, LinkURL: linkErr.ConnectURL, Detail: err.Error()} + } + return platform.ClaimResult{Outcome: platform.ClaimFailed, Detail: err.Error()} +} + +// ProbeChannels satisfies platform.ChannelProber on top of probeLive. +func (b *Backend) ProbeChannels(ctx context.Context, s platform.Session, c platform.Campaign, logins []string) ([]platform.Stream, error) { + pool := make([]kickChannel, 0, len(logins)) + for _, l := range logins { + if l = strings.ToLower(strings.TrimSpace(l)); l != "" { + pool = append(pool, kickChannel{Slug: l}) + } + } + return b.probeLive(ctx, s, c, pool), nil +} + +var ( + _ platform.DropProgressSource = (*Backend)(nil) + _ platform.DropClaimer = (*Backend)(nil) + _ platform.ChannelProber = (*Backend)(nil) +) diff --git a/internal/platform/kick/pipeline_test.go b/internal/platform/kick/pipeline_test.go new file mode 100644 index 0000000..ee855e3 --- /dev/null +++ b/internal/platform/kick/pipeline_test.go @@ -0,0 +1,77 @@ +package kick + +import ( + "context" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/aalejandrofer/grubdrops/internal/platform" +) + +const progressURL = "https://web.kick.com/api/v1/drops/progress" + +func TestKickDropProgress_RequiredFromPlatform(t *testing.T) { + f := &fakeDoer{resp: map[string]fakeResp{progressURL: {200, `{"data":[ + {"id":"c1","rewards":[{"id":"r1","claimed":false,"progress":0.5,"required_units":30}]} + ]}`}}} + b := withFake(f) + got, err := b.DropProgress(context.Background(), sess("acc1"), []platform.Campaign{{ + ID: "c1", Benefits: []platform.DropBenefit{{ID: "r1", CampaignID: "c1", RequiredMinutes: 120}}, + }}) + require.NoError(t, err) + require.Len(t, got, 1) + // Platform says 30, adapter's invented 120 must not win (P6). + assert.Equal(t, platform.DropProgress{DropID: "r1", CampaignID: "c1", Minutes: 15, Required: 30, Known: true}, got[0]) +} + +func TestKickDropProgress_AbsentRewardIsUnknown(t *testing.T) { + f := &fakeDoer{resp: map[string]fakeResp{progressURL: {200, `{"data":[]}`}}} + b := withFake(f) + got, err := b.DropProgress(context.Background(), sess("acc1"), []platform.Campaign{{ + ID: "c1", Benefits: []platform.DropBenefit{{ID: "r1", CampaignID: "c1", RequiredMinutes: 120}}, + }}) + require.NoError(t, err) + require.Len(t, got, 1) + assert.False(t, got[0].Known, "absence proves nothing") + assert.Equal(t, 120, got[0].Required) +} + +func TestKickDropProgress_ErrorFailsWhole(t *testing.T) { + f := &fakeDoer{resp: map[string]fakeResp{progressURL: {403, `{}`}}} + _, err := withFake(f).DropProgress(context.Background(), sess("acc1"), []platform.Campaign{{ID: "c1"}}) + require.Error(t, err) +} + +func TestKickClaimDrop_Outcomes(t *testing.T) { + claimURL := "https://web.kick.com/api/v1/drops/claim" + ok := withFake(&fakeDoer{resp: map[string]fakeResp{claimURL: {200, `{}`}}}) + assert.Equal(t, platform.ClaimOK, ok.ClaimDrop(context.Background(), sess("a"), platform.DropProgress{DropID: "r1", CampaignID: "c1"}).Outcome) + + link := withFake(&fakeDoer{resp: map[string]fakeResp{claimURL: {400, `{"connect_url":"https://kick.com/connect/riot"}`}}}) + r := link.ClaimDrop(context.Background(), sess("a"), platform.DropProgress{DropID: "r1", CampaignID: "c1"}) + assert.Equal(t, platform.ClaimNeedsLink, r.Outcome) + assert.Equal(t, "https://kick.com/connect/riot", r.LinkURL) + + fail := withFake(&fakeDoer{resp: map[string]fakeResp{claimURL: {500, `{}`}}}) + assert.Equal(t, platform.ClaimFailed, fail.ClaimDrop(context.Background(), sess("a"), platform.DropProgress{DropID: "r1", CampaignID: "c1"}).Outcome) +} + +func TestKickProbeChannels(t *testing.T) { + f := &fakeDoer{resp: map[string]fakeResp{ + "https://kick.com/api/v2/channels/fav/livestream": {200, `{"data":{"id":7,"viewer_count":3,"categories":[{"name":"Rust","slug":"rust"}]}}`}, + "https://kick.com/api/v2/channels/off/livestream": {200, `{"data":null}`}, + "https://kick.com/api/v2/channels/other/livestream": {200, `{"data":{"id":8,"viewer_count":9,"categories":[{"name":"Slots","slug":"slots"}]}}`}, + }} + got, err := withFake(f).ProbeChannels(context.Background(), sess("a"), platform.Campaign{Game: "Rust"}, []string{"Fav", "off", "other"}) + require.NoError(t, err) + require.Len(t, got, 1) + assert.Equal(t, "fav", got[0].Channel) +} + +func TestKickBackend_SatisfiesPipelineInterfaces(t *testing.T) { + var _ platform.DropProgressSource = (*Backend)(nil) + var _ platform.DropClaimer = (*Backend)(nil) + var _ platform.ChannelProber = (*Backend)(nil) +} From d839fb7b0dbc89785c3d9536d1ce42d6f1764cbe Mon Sep 17 00:00:00 2001 From: Alejandro Date: Sun, 27 Sep 2026 20:16:33 -0400 Subject: [PATCH 15/56] test(pipeline-v2): pin Kick required fallback for rows without required_units Co-Authored-By: Claude Opus 5.5 (1M context) --- internal/platform/kick/pipeline.go | 3 +++ internal/platform/kick/pipeline_test.go | 13 +++++++++++++ 2 files changed, 16 insertions(+) diff --git a/internal/platform/kick/pipeline.go b/internal/platform/kick/pipeline.go index d1dee22..a72b833 100644 --- a/internal/platform/kick/pipeline.go +++ b/internal/platform/kick/pipeline.go @@ -32,6 +32,9 @@ func (b *Backend) DropProgress(ctx context.Context, s platform.Session, camps [] continue } req := r.Required + // Kick's live payload always carries required_units; this fallback + // only covers a malformed row. Reporting 0 would wrongly block the + // drop as sub_only, so we use the campaign's RequiredMinutes instead. if req <= 0 { req = bf.RequiredMinutes } diff --git a/internal/platform/kick/pipeline_test.go b/internal/platform/kick/pipeline_test.go index ee855e3..778388e 100644 --- a/internal/platform/kick/pipeline_test.go +++ b/internal/platform/kick/pipeline_test.go @@ -38,6 +38,19 @@ func TestKickDropProgress_AbsentRewardIsUnknown(t *testing.T) { assert.Equal(t, 120, got[0].Required) } +func TestKickDropProgress_ListedWithoutRequiredUsesCampaign(t *testing.T) { + f := &fakeDoer{resp: map[string]fakeResp{progressURL: {200, `{"data":[ + {"id":"c1","rewards":[{"id":"r1","claimed":false,"progress":0.5}]} + ]}`}}} + b := withFake(f) + got, err := b.DropProgress(context.Background(), sess("acc1"), []platform.Campaign{{ + ID: "c1", Benefits: []platform.DropBenefit{{ID: "r1", CampaignID: "c1", RequiredMinutes: 60}}, + }}) + require.NoError(t, err) + require.Len(t, got, 1) + assert.Equal(t, platform.DropProgress{DropID: "r1", CampaignID: "c1", Minutes: 30, Required: 60, Known: true}, got[0]) +} + func TestKickDropProgress_ErrorFailsWhole(t *testing.T) { f := &fakeDoer{resp: map[string]fakeResp{progressURL: {403, `{}`}}} _, err := withFake(f).DropProgress(context.Background(), sess("acc1"), []platform.Campaign{{ID: "c1"}}) From c53b6d04c2827f236c59bfda2efe1e7a1d2eaf5c Mon Sep 17 00:00:00 2001 From: Alejandro Date: Sun, 27 Sep 2026 20:18:21 -0400 Subject: [PATCH 16/56] feat(pipeline-v2): reconciler folds platform progress into drop state Co-Authored-By: Claude Opus 5.5 (1M context) --- internal/pipeline/reconcile/reconcile.go | 135 ++++++++++++++++ internal/pipeline/reconcile/reconcile_test.go | 144 ++++++++++++++++++ 2 files changed, 279 insertions(+) create mode 100644 internal/pipeline/reconcile/reconcile.go create mode 100644 internal/pipeline/reconcile/reconcile_test.go diff --git a/internal/pipeline/reconcile/reconcile.go b/internal/pipeline/reconcile/reconcile.go new file mode 100644 index 0000000..e2021bf --- /dev/null +++ b/internal/pipeline/reconcile/reconcile.go @@ -0,0 +1,135 @@ +// Package reconcile folds platform per-campaign progress into drop_state. +// It never decides what to watch. +package reconcile + +import ( + "context" + "fmt" + "time" + + "github.com/aalejandrofer/grubdrops/internal/pipeline/dropstate" + "github.com/aalejandrofer/grubdrops/internal/platform" +) + +type CampaignPersister interface { + PersistCampaigns(ctx context.Context, camps []platform.Campaign) error +} + +type Config struct { + AccountID string + Platform string + Backend platform.Backend + Progress platform.DropProgressSource + Session platform.Session + AllowGame func(game string) bool + AllowChannel func(channels []string) bool + ForceLinked func(campaignID string) bool + Persister CampaignPersister +} + +type Result struct { + Campaigns []platform.Campaign + Rows []dropstate.Row + Progress map[string]platform.DropProgress +} + +// InScope keeps campaigns the account mines: whitelisted game, or a +// campaign whose channels match the account's priority streamers +// (null-game drops). One-click reward campaigns have no watch time. +func InScope(camps []platform.Campaign, allowGame func(string) bool, allowChannel func([]string) bool) []platform.Campaign { + var out []platform.Campaign + for _, c := range camps { + if c.Kind == "reward" { + continue + } + ok := allowGame == nil || allowGame(c.Game) + if !ok && allowChannel != nil && allowChannel(c.AllowedChannels) { + ok = true + } + if ok { + out = append(out, c) + } + } + return out +} + +func isActive(c platform.Campaign, now time.Time) bool { + if c.Status != "" && c.Status != "active" { + return false + } + return c.EndsAt.IsZero() || now.Before(c.EndsAt) +} + +// Run lists campaigns, persists the in-scope ones, reads their progress and +// returns every drop row it touched. On any platform error it returns the +// error and no rows, so a failed sync never changes state. +func Run(ctx context.Context, cfg Config, prev map[string]dropstate.Row, now time.Time) (Result, error) { + camps, err := cfg.Backend.ListActiveCampaigns(ctx, cfg.Session) + if err != nil { + return Result{}, fmt.Errorf("list campaigns: %w", err) + } + scope := InScope(camps, cfg.AllowGame, cfg.AllowChannel) + if cfg.Persister != nil && len(scope) > 0 { + if err := cfg.Persister.PersistCampaigns(ctx, scope); err != nil { + return Result{}, fmt.Errorf("persist campaigns: %w", err) + } + } + var active []platform.Campaign + for _, c := range scope { + if isActive(c, now) { + active = append(active, c) + } + } + byDrop := map[string]platform.DropProgress{} + if len(active) > 0 { + obs, err := cfg.Progress.DropProgress(ctx, cfg.Session, active) + if err != nil { + return Result{}, fmt.Errorf("drop progress: %w", err) + } + for _, o := range obs { + byDrop[o.DropID] = o + } + } + + res := Result{Campaigns: active, Progress: byDrop} + seen := map[string]bool{} + for _, c := range scope { + if c.Status == "upcoming" { + continue + } + expired := !isActive(c, now) + linked := c.AccountLinked || !c.AccountLinkChecked || (cfg.ForceLinked != nil && cfg.ForceLinked(c.ID)) + for _, b := range c.Benefits { + if b.ID == "" || seen[b.ID] { + continue + } + seen[b.ID] = true + p, ok := prev[b.ID] + if !ok || p.IsZero() { + p = dropstate.Row{AccountID: cfg.AccountID, DropID: b.ID, CampaignID: c.ID, Platform: cfg.Platform, Source: dropstate.FromPlatform} + } + var next dropstate.Row + if expired { + if p.IsZero() { + continue // never tracked, nothing to expire + } + next = dropstate.Expire(p, now) + } else { + o, known := byDrop[b.ID] + req := b.RequiredMinutes + if known { + req = o.Required + } + next = dropstate.Apply(p, dropstate.Observation{ + Known: known && o.Known, Unmineable: o.Unmineable, Claimed: o.Claimed, + Minutes: o.Minutes, Required: req, + }, now) + if !linked { + next = dropstate.BlockLink(next, now) + } + } + res.Rows = append(res.Rows, next) + } + } + return res, nil +} diff --git a/internal/pipeline/reconcile/reconcile_test.go b/internal/pipeline/reconcile/reconcile_test.go new file mode 100644 index 0000000..f1e9333 --- /dev/null +++ b/internal/pipeline/reconcile/reconcile_test.go @@ -0,0 +1,144 @@ +package reconcile + +import ( + "context" + "errors" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/aalejandrofer/grubdrops/internal/pipeline/dropstate" + "github.com/aalejandrofer/grubdrops/internal/platform" +) + +type fakeBackend struct { + platform.Backend // nil: unimplemented methods panic + camps []platform.Campaign + progress []platform.DropProgress + progErr error +} + +func (f *fakeBackend) ListActiveCampaigns(context.Context, platform.Session) ([]platform.Campaign, error) { + return f.camps, nil +} + +func (f *fakeBackend) DropProgress(context.Context, platform.Session, []platform.Campaign) ([]platform.DropProgress, error) { + return f.progress, f.progErr +} + +type fakePersister struct{ got []platform.Campaign } + +func (p *fakePersister) PersistCampaigns(_ context.Context, c []platform.Campaign) error { + p.got = append(p.got, c...) + return nil +} + +var now = time.Unix(1_700_000_000, 0) + +func camp(id, game string, drops ...string) platform.Campaign { + c := platform.Campaign{ID: id, Platform: "twitch", Game: game, Status: "active", AccountLinked: true, AccountLinkChecked: true} + for _, d := range drops { + c.Benefits = append(c.Benefits, platform.DropBenefit{ID: d, CampaignID: id, RequiredMinutes: 60}) + } + return c +} + +func cfg(f *fakeBackend, p *fakePersister) Config { + return Config{AccountID: "a", Platform: "twitch", Backend: f, Progress: f, Persister: p, + AllowGame: func(g string) bool { return g == "G" }} +} + +func byDrop(rows []dropstate.Row) map[string]dropstate.Row { + m := map[string]dropstate.Row{} + for _, r := range rows { + m[r.DropID] = r + } + return m +} + +func TestRun_ClaimedAfterLeavingInventory(t *testing.T) { + f := &fakeBackend{camps: []platform.Campaign{camp("c1", "G", "d1")}, + progress: []platform.DropProgress{{DropID: "d1", CampaignID: "c1", Minutes: 60, Required: 60, Claimed: true, Known: true}}} + prev := map[string]dropstate.Row{"d1": {AccountID: "a", DropID: "d1", CampaignID: "c1", Platform: "twitch", Status: dropstate.Accruing, Minutes: 30, Required: 60, Source: dropstate.FromPlatform}} + res, err := Run(context.Background(), cfg(f, &fakePersister{}), prev, now) + require.NoError(t, err) + assert.Equal(t, dropstate.Claimed, byDrop(res.Rows)["d1"].Status) +} + +func TestRun_NewDropSeedsIdentity(t *testing.T) { + f := &fakeBackend{camps: []platform.Campaign{camp("c1", "G", "d1")}, + progress: []platform.DropProgress{{DropID: "d1", CampaignID: "c1", Minutes: 5, Required: 60, Known: true}}} + res, err := Run(context.Background(), cfg(f, &fakePersister{}), nil, now) + require.NoError(t, err) + r := byDrop(res.Rows)["d1"] + assert.Equal(t, "a", r.AccountID) + assert.Equal(t, "c1", r.CampaignID) + assert.Equal(t, "twitch", r.Platform) + assert.Equal(t, dropstate.Accruing, r.Status) +} + +func TestRun_ProgressErrorChangesNothing(t *testing.T) { + f := &fakeBackend{camps: []platform.Campaign{camp("c1", "G", "d1")}, progErr: errors.New("details 500")} + prev := map[string]dropstate.Row{"d1": {DropID: "d1", Status: dropstate.Accruing, Minutes: 30, Required: 60}} + res, err := Run(context.Background(), cfg(f, &fakePersister{}), prev, now) + require.Error(t, err) + assert.Empty(t, res.Rows) +} + +func TestRun_MultiItemDropOneRow(t *testing.T) { + c := camp("c1", "G", "d1", "d1") + f := &fakeBackend{camps: []platform.Campaign{c}, + progress: []platform.DropProgress{{DropID: "d1", CampaignID: "c1", Required: 60, Known: true}}} + res, err := Run(context.Background(), cfg(f, &fakePersister{}), nil, now) + require.NoError(t, err) + assert.Len(t, res.Rows, 1) +} + +func TestRun_UnlinkedCampaignBlocked_UnlessForced(t *testing.T) { + c := camp("c1", "G", "d1") + c.AccountLinked = false + f := &fakeBackend{camps: []platform.Campaign{c}, + progress: []platform.DropProgress{{DropID: "d1", CampaignID: "c1", Required: 60, Known: true}}} + res, err := Run(context.Background(), cfg(f, &fakePersister{}), nil, now) + require.NoError(t, err) + assert.Equal(t, dropstate.NeedsLink, byDrop(res.Rows)["d1"].Reason) + + cf := cfg(f, &fakePersister{}) + cf.ForceLinked = func(id string) bool { return id == "c1" } + res, err = Run(context.Background(), cf, nil, now) + require.NoError(t, err) + assert.Equal(t, dropstate.Eligible, byDrop(res.Rows)["d1"].Status) +} + +func TestRun_ExpiredCampaign(t *testing.T) { + c := camp("c1", "G", "d1") + c.Status = "expired" + f := &fakeBackend{camps: []platform.Campaign{c}} + prev := map[string]dropstate.Row{"d1": {DropID: "d1", Status: dropstate.Accruing, Minutes: 30, Required: 60}} + res, err := Run(context.Background(), cfg(f, &fakePersister{}), prev, now) + require.NoError(t, err) + assert.Equal(t, dropstate.Expired, byDrop(res.Rows)["d1"].Reason) + assert.Empty(t, res.Campaigns, "expired campaigns are not mining candidates") +} + +func TestRun_ScopeAndPersist(t *testing.T) { + nullGame := camp("c2", "", "d2") + nullGame.AllowedChannels = []string{"fav"} + reward := camp("c3", "G", "d3") + reward.Kind = "reward" + f := &fakeBackend{camps: []platform.Campaign{camp("c1", "Other", "d1"), nullGame, reward}, + progress: []platform.DropProgress{{DropID: "d2", CampaignID: "c2", Required: 60, Known: true}}} + p := &fakePersister{} + cf := cfg(f, p) + cf.AllowChannel = func(chs []string) bool { return len(chs) > 0 && chs[0] == "fav" } + res, err := Run(context.Background(), cf, nil, now) + require.NoError(t, err) + got := byDrop(res.Rows) + assert.NotContains(t, got, "d1", "non-whitelisted game") + assert.Contains(t, got, "d2", "null-game campaign allowed by channel") + assert.NotContains(t, got, "d3", "reward kind has no watch time") + require.Len(t, p.got, 1) + assert.Equal(t, "c2", p.got[0].ID) +} From 8d85f8335518768c9fb9deea8c8ca072d0f60523 Mon Sep 17 00:00:00 2001 From: Alejandro Date: Sun, 27 Sep 2026 20:22:04 -0400 Subject: [PATCH 17/56] feat(pipeline-v2): claimer maps typed claim outcomes to drop state Co-Authored-By: Claude Haiku 4.5 --- internal/pipeline/claimer/claimer.go | 25 +++++++++++ internal/pipeline/claimer/claimer_test.go | 51 +++++++++++++++++++++++ 2 files changed, 76 insertions(+) create mode 100644 internal/pipeline/claimer/claimer.go create mode 100644 internal/pipeline/claimer/claimer_test.go diff --git a/internal/pipeline/claimer/claimer.go b/internal/pipeline/claimer/claimer.go new file mode 100644 index 0000000..277ca78 --- /dev/null +++ b/internal/pipeline/claimer/claimer.go @@ -0,0 +1,25 @@ +// Package claimer turns a claimable drop into claimed or blocked. +package claimer + +import ( + "context" + "time" + + "github.com/aalejandrofer/grubdrops/internal/pipeline/dropstate" + "github.com/aalejandrofer/grubdrops/internal/platform" +) + +// Attempt claims one drop and returns its next state. The caller must +// reconcile afterwards: a lost response shows up as ClaimFailed here and is +// corrected when the platform reports isClaimed. +func Attempt(ctx context.Context, c platform.DropClaimer, s platform.Session, row dropstate.Row, dp platform.DropProgress, now time.Time) (dropstate.Row, platform.ClaimResult) { + res := c.ClaimDrop(ctx, s, dp) + switch res.Outcome { + case platform.ClaimOK, platform.ClaimAlready: + return dropstate.ClaimOK(row, now), res + case platform.ClaimNeedsLink: + return dropstate.ClaimNeedsLink(row, now), res + default: + return dropstate.ClaimFailedAttempt(row, now), res + } +} diff --git a/internal/pipeline/claimer/claimer_test.go b/internal/pipeline/claimer/claimer_test.go new file mode 100644 index 0000000..4e122db --- /dev/null +++ b/internal/pipeline/claimer/claimer_test.go @@ -0,0 +1,51 @@ +package claimer + +import ( + "context" + "testing" + "time" + + "github.com/stretchr/testify/assert" + + "github.com/aalejandrofer/grubdrops/internal/pipeline/dropstate" + "github.com/aalejandrofer/grubdrops/internal/platform" +) + +type fakeClaimer struct { + res platform.ClaimResult + got platform.DropProgress +} + +func (f *fakeClaimer) ClaimDrop(_ context.Context, _ platform.Session, d platform.DropProgress) platform.ClaimResult { + f.got = d + return f.res +} + +func TestAttempt(t *testing.T) { + now := time.Unix(1_700_000_000, 0) + row := dropstate.Row{DropID: "d1", CampaignID: "c1", Status: dropstate.Claimable, Minutes: 60, Required: 60} + cases := []struct { + out platform.ClaimOutcome + status dropstate.Status + reason dropstate.Reason + }{ + {platform.ClaimOK, dropstate.Claimed, dropstate.NoReason}, + {platform.ClaimAlready, dropstate.Claimed, dropstate.NoReason}, + {platform.ClaimNeedsLink, dropstate.Blocked, dropstate.NeedsLink}, + {platform.ClaimFailed, dropstate.Claimable, dropstate.NoReason}, + } + for _, tc := range cases { + t.Run(tc.out.String(), func(t *testing.T) { + fc := &fakeClaimer{res: platform.ClaimResult{Outcome: tc.out}} + got, res := Attempt(context.Background(), fc, platform.Session{}, row, platform.DropProgress{DropID: "d1", InstanceID: "i"}, now) + assert.Equal(t, tc.status, got.Status) + assert.Equal(t, tc.reason, got.Reason) + assert.Equal(t, tc.out, res.Outcome) + assert.Equal(t, "i", fc.got.InstanceID, "instance id forwarded") + }) + } + fc := &fakeClaimer{res: platform.ClaimResult{Outcome: platform.ClaimFailed}} + got, _ := Attempt(context.Background(), fc, platform.Session{}, row, platform.DropProgress{DropID: "d1"}, now) + assert.Equal(t, 1, got.FailCount) + assert.Equal(t, now.Add(time.Minute), got.RetryAfter) +} From b5b325b051ee85bb448b5ccadec04a033816b59b Mon Sep 17 00:00:00 2001 From: Alejandro Date: Sun, 27 Sep 2026 20:30:19 -0400 Subject: [PATCH 18/56] feat(pipeline-v2): per-account loop owning all drop state transitions Co-Authored-By: Claude Opus 5.5 (1M context) --- internal/pipeline/loop/loop.go | 615 ++++++++++++++++++++++++++++ internal/pipeline/loop/loop_test.go | 243 +++++++++++ 2 files changed, 858 insertions(+) create mode 100644 internal/pipeline/loop/loop.go create mode 100644 internal/pipeline/loop/loop_test.go diff --git a/internal/pipeline/loop/loop.go b/internal/pipeline/loop/loop.go new file mode 100644 index 0000000..f7315dd --- /dev/null +++ b/internal/pipeline/loop/loop.go @@ -0,0 +1,615 @@ +// Package loop is the per-account pipeline v2 runner. One goroutine owns +// every drop state transition: it reconciles, plans, runs one session at a +// time, and claims. +package loop + +import ( + "context" + "fmt" + "log/slog" + "strings" + "sync" + "time" + + "github.com/aalejandrofer/grubdrops/internal/pipeline/claimer" + "github.com/aalejandrofer/grubdrops/internal/pipeline/dropstate" + "github.com/aalejandrofer/grubdrops/internal/pipeline/planner" + "github.com/aalejandrofer/grubdrops/internal/pipeline/reconcile" + "github.com/aalejandrofer/grubdrops/internal/pipeline/session" + "github.com/aalejandrofer/grubdrops/internal/platform" + "github.com/aalejandrofer/grubdrops/internal/watcher" +) + +type Notifier interface { + Notify(ctx context.Context, event string, fields map[string]any) error +} + +type Store interface { + List(ctx context.Context, accountID string) ([]dropstate.Row, error) + Upsert(ctx context.Context, r dropstate.Row) error +} + +type ClaimHistory interface { + RecordClaimIfNew(ctx context.Context, accountID string, b platform.DropBenefit) (bool, error) + ClaimedBenefitIDs(ctx context.Context, accountID string) (map[string]bool, error) +} + +type Config struct { + AccountID string + AccountLabel string + Platform string + Backend platform.Backend + Session platform.Session + Store Store + History ClaimHistory + Persister reconcile.CampaignPersister + Notifier Notifier + + AllowGame func(game string) bool + AllowChannel func(channels []string) bool + GameRank func(game string) int + PriorityMode string + ForceLinked func(campaignID string) bool + StreamerPriority []string + ForceWatch []string + ProgressNotifyStepPct int + + Now func() time.Time + ReconcileEvery time.Duration // default 15m + LiveEvery time.Duration // default 5m + BeatEvery time.Duration // default 60s; production never overrides (Twitch credit gate) + StallPolls int // default 5 + StallCooldown time.Duration // default 30m + DownCooldown time.Duration // default 2m +} + +// maxLiveCampaigns bounds channel lookups per refresh. +const maxLiveCampaigns = 8 + +type pubsubEvent struct { + kind string // "progress" | "claimable" | "down" + drop string + instance string + minutes int + required int + channel string +} + +type Loop struct { + cfg Config + prog platform.DropProgressSource + claimer platform.DropClaimer + prober platform.ChannelProber + subs platform.ChannelSubscriber + + nudge chan struct{} + pubsub chan pubsubEvent + sessEvents chan session.Event + + mu sync.Mutex + snap watcher.Snapshot + discovery []platform.Campaign + discoverAt time.Time + + // Owned by the Run goroutine. + rows map[string]dropstate.Row + camps []platform.Campaign + progress map[string]platform.DropProgress + live map[string][]platform.Stream + forceLive []platform.Stream + cooldowns map[string]time.Time + current *planner.Decision + lastSwap time.Time + sessCancel context.CancelFunc + subscribed string + milestones map[string]int +} + +// New assembles the loop, checking the backend's pipeline capabilities once. +func New(cfg Config) (*Loop, error) { + prog, ok := cfg.Backend.(platform.DropProgressSource) + if !ok { + return nil, fmt.Errorf("backend %T has no DropProgress: pipeline v2 unsupported", cfg.Backend) + } + cl, ok := cfg.Backend.(platform.DropClaimer) + if !ok { + return nil, fmt.Errorf("backend %T has no ClaimDrop: pipeline v2 unsupported", cfg.Backend) + } + if cfg.Now == nil { + cfg.Now = time.Now + } + if cfg.ReconcileEvery <= 0 { + cfg.ReconcileEvery = 15 * time.Minute + } + if cfg.LiveEvery <= 0 { + cfg.LiveEvery = 5 * time.Minute + } + if cfg.BeatEvery <= 0 { + cfg.BeatEvery = 60 * time.Second + } + if cfg.StallPolls <= 0 { + cfg.StallPolls = 5 + } + if cfg.StallCooldown <= 0 { + cfg.StallCooldown = 30 * time.Minute + } + if cfg.DownCooldown <= 0 { + cfg.DownCooldown = 2 * time.Minute + } + cfg.Session.AccountID = cfg.AccountID + if cfg.AllowGame != nil { + cfg.Session.GameFilter = cfg.AllowGame + } + l := &Loop{ + cfg: cfg, prog: prog, claimer: cl, + nudge: make(chan struct{}, 1), + pubsub: make(chan pubsubEvent, 64), + sessEvents: make(chan session.Event, 64), + rows: map[string]dropstate.Row{}, + progress: map[string]platform.DropProgress{}, + live: map[string][]platform.Stream{}, + cooldowns: map[string]time.Time{}, + milestones: map[string]int{}, + } + l.prober, _ = cfg.Backend.(platform.ChannelProber) + l.subs, _ = cfg.Backend.(platform.ChannelSubscriber) + l.snap = watcher.Snapshot{AccountID: cfg.AccountID, State: "pick_campaign"} + if ps, ok := cfg.Backend.(platform.PubSubAware); ok { + ps.SetAccountPubSubHooks(cfg.AccountID, l.hooks()) + } + return l, nil +} + +func (l *Loop) hooks() platform.PubSubHooks { + push := func(e pubsubEvent) { + select { + case l.pubsub <- e: + default: // never block the PubSub reader + } + } + return platform.PubSubHooks{ + OnDropProgress: func(dropID string, cur, req int64) { + push(pubsubEvent{kind: "progress", drop: dropID, minutes: int(cur), required: int(req)}) + }, + OnDropClaim: func(dropID, instanceID string) { + push(pubsubEvent{kind: "claimable", drop: dropID, instance: instanceID}) + }, + OnStreamDown: func(channelID string) { + push(pubsubEvent{kind: "down", channel: channelID}) + }, + } +} + +// Nudge asks the loop to reconcile and refresh channels now. +func (l *Loop) Nudge() { + select { + case l.nudge <- struct{}{}: + default: + } +} + +func (l *Loop) Snapshot() watcher.Snapshot { + l.mu.Lock() + defer l.mu.Unlock() + return l.snap +} + +func (l *Loop) LastDiscovery() ([]platform.Campaign, time.Time) { + l.mu.Lock() + defer l.mu.Unlock() + return append([]platform.Campaign(nil), l.discovery...), l.discoverAt +} + +func (l *Loop) AllowGame() func(game string) bool { return l.cfg.AllowGame } + +func (l *Loop) Run(ctx context.Context) error { + if err := l.load(ctx); err != nil { + return err + } + l.reconcile(ctx) + l.refreshLive(ctx) + l.claimReady(ctx) + l.replan(ctx) + recT := time.NewTicker(l.cfg.ReconcileEvery) + liveT := time.NewTicker(l.cfg.LiveEvery) + defer recT.Stop() + defer liveT.Stop() + defer l.stopSession() + for { + select { + case <-ctx.Done(): + return ctx.Err() + case <-recT.C: + l.reconcile(ctx) + case <-liveT.C: + l.refreshLive(ctx) + case <-l.nudge: + l.reconcile(ctx) + l.refreshLive(ctx) + case ev := <-l.sessEvents: + l.onSession(ctx, ev) + case pe := <-l.pubsub: + l.onPubSub(ctx, pe) + } + l.claimReady(ctx) + l.replan(ctx) + } +} + +// load reads stored state, then bridges claim rows written by the v1-era +// mark-collected UI (that handler reloads the scheduler, so this runs). +func (l *Loop) load(ctx context.Context) error { + rows, err := l.cfg.Store.List(ctx, l.cfg.AccountID) + if err != nil { + return fmt.Errorf("load drop state: %w", err) + } + for _, r := range rows { + l.rows[r.DropID] = r + } + if l.cfg.History == nil { + return nil + } + ids, err := l.cfg.History.ClaimedBenefitIDs(ctx, l.cfg.AccountID) + if err != nil { + slog.Warn("pipeline: read claim history failed", "kind", "error", "account", l.cfg.AccountID, "err", err) + return nil + } + now := l.cfg.Now() + for id := range ids { + if r, ok := l.rows[id]; ok && r.Status != dropstate.Claimed { + next := dropstate.MarkCollected(r, now) + l.rows[id] = next + if err := l.cfg.Store.Upsert(ctx, next); err != nil { + slog.Warn("pipeline: persist bridged mark failed", "kind", "error", "account", l.cfg.AccountID, "drop", id, "err", err) + } + } + } + return nil +} + +func (l *Loop) reconcile(ctx context.Context) { + res, err := reconcile.Run(ctx, reconcile.Config{ + AccountID: l.cfg.AccountID, Platform: l.cfg.Platform, + Backend: l.cfg.Backend, Progress: l.prog, Session: l.cfg.Session, + AllowGame: l.cfg.AllowGame, AllowChannel: l.cfg.AllowChannel, + ForceLinked: l.cfg.ForceLinked, Persister: l.cfg.Persister, + }, l.rows, l.cfg.Now()) + if err != nil { + slog.Warn("pipeline reconcile failed; keeping last state", "kind", "error", "account", l.cfg.AccountID, "err", err) + return + } + l.camps = res.Campaigns + for id, dp := range res.Progress { + l.progress[id] = dp + } + l.mu.Lock() + l.discovery, l.discoverAt = res.Campaigns, l.cfg.Now() + l.mu.Unlock() + for _, r := range res.Rows { + l.commit(ctx, r) + } +} + +func (l *Loop) benefit(dropID string) (platform.DropBenefit, platform.Campaign) { + for _, c := range l.camps { + for _, b := range c.Benefits { + if b.ID == dropID { + return b, c + } + } + } + return platform.DropBenefit{ID: dropID}, platform.Campaign{} +} + +// commit stores a row and handles the transition into claimed. +func (l *Loop) commit(ctx context.Context, next dropstate.Row) { + prev := l.rows[next.DropID] + l.rows[next.DropID] = next + if err := l.cfg.Store.Upsert(ctx, next); err != nil { + slog.Warn("pipeline: persist drop state failed", "kind", "error", "account", l.cfg.AccountID, "drop", next.DropID, "err", err) + } + if next.Status != dropstate.Claimed || prev.Status == dropstate.Claimed { + return + } + b, c := l.benefit(next.DropID) + if l.cfg.History != nil { + if _, err := l.cfg.History.RecordClaimIfNew(ctx, l.cfg.AccountID, b); err != nil { + slog.Warn("pipeline: record claim history failed", "kind", "error", "account", l.cfg.AccountID, "drop", next.DropID, "err", err) + } + } + slog.Info("pipeline drop claimed", "kind", "claim", "account", l.cfg.AccountID, "drop", next.DropID, "source", string(next.Source)) + // Only notify live transitions, not historical claims seen on first sync. + if !prev.IsZero() && next.Source == dropstate.FromPlatform { + l.notify(ctx, "claim", c, b, nil) + } +} + +func (l *Loop) notify(ctx context.Context, event string, c platform.Campaign, b platform.DropBenefit, extra map[string]any) { + if l.cfg.Notifier == nil { + return + } + f := map[string]any{"account": l.cfg.AccountID} + if l.cfg.AccountLabel != "" { + f["account_label"] = l.cfg.AccountLabel + } + if l.cfg.Platform != "" { + f["platform"] = l.cfg.Platform + } + if c.Game != "" { + f["game"] = c.Game + } + if c.Name != "" { + f["campaign"] = c.Name + } + if b.Name != "" { + f["drop"] = b.Name + } + for k, v := range extra { + f[k] = v + } + _ = l.cfg.Notifier.Notify(ctx, event, f) +} + +func (l *Loop) plannerInput() planner.Input { + return planner.Input{ + Now: l.cfg.Now(), Rows: l.rows, Campaigns: l.camps, + Settings: planner.Settings{ + GameRank: l.cfg.GameRank, PriorityMode: l.cfg.PriorityMode, + StreamerPriority: l.cfg.StreamerPriority, ForceWatch: l.forceLive, + }, + Live: l.live, Cooldowns: l.cooldowns, Current: l.current, LastSwap: l.lastSwap, + } +} + +// allowedPriority returns the priority logins a campaign accepts. +func allowedPriority(c platform.Campaign, prio []string) []string { + if len(c.AllowedChannels) == 0 { + return prio + } + allowed := map[string]bool{} + for _, a := range c.AllowedChannels { + allowed[strings.ToLower(a)] = true + } + var out []string + for _, p := range prio { + if allowed[strings.ToLower(p)] { + out = append(out, p) + } + } + return out +} + +func mergeStreams(first, rest []platform.Stream) []platform.Stream { + seen := map[string]bool{} + var out []platform.Stream + for _, s := range append(append([]platform.Stream(nil), first...), rest...) { + k := strings.ToLower(s.Channel) + if !seen[k] { + seen[k] = true + out = append(out, s) + } + } + return out +} + +func (l *Loop) refreshLive(ctx context.Context) { + live := map[string][]platform.Stream{} + for i, c := range planner.Candidates(l.plannerInput()) { + if i >= maxLiveCampaigns { + break + } + streams, err := l.cfg.Backend.ListEligibleChannels(ctx, l.cfg.Session, c) + if err != nil { + slog.Debug("pipeline: list channels failed", "account", l.cfg.AccountID, "campaign", c.ID, "err", err) + } + if l.prober != nil { + if logins := allowedPriority(c, l.cfg.StreamerPriority); len(logins) > 0 { + if ps, err := l.prober.ProbeChannels(ctx, l.cfg.Session, c, logins); err == nil { + streams = mergeStreams(ps, streams) + } + } + } + live[c.ID] = streams + } + l.live = live + l.forceLive = nil + if l.prober != nil && len(l.cfg.ForceWatch) > 0 { + if ps, err := l.prober.ProbeChannels(ctx, l.cfg.Session, platform.Campaign{}, l.cfg.ForceWatch); err == nil { + l.forceLive = ps + } + } +} + +func (l *Loop) isCurrent(channel string) bool { + return l.current != nil && l.current.Kind != planner.Idle && strings.EqualFold(l.current.Channel.Channel, channel) +} + +func (l *Loop) onSession(ctx context.Context, ev session.Event) { + now := l.cfg.Now() + switch ev.Kind { + case session.Progress: + l.applyProgress(ctx, ev.Progress) + case session.StreamDown: + if l.isCurrent(ev.Channel) { + l.cooldowns[strings.ToLower(ev.Channel)] = now.Add(l.cfg.DownCooldown) + } + case session.Stalled: + if l.isCurrent(ev.Channel) { + l.cooldowns[strings.ToLower(ev.Channel)] = now.Add(l.cfg.StallCooldown) + l.reconcile(ctx) + } + } +} + +func (l *Loop) applyProgress(ctx context.Context, progs []platform.Progress) { + now := l.cfg.Now() + for _, p := range progs { + prev, ok := l.rows[p.BenefitID] + if !ok { + continue // reconcile creates the row first + } + if p.InstanceID != "" { + dp := l.progress[p.BenefitID] + dp.DropID, dp.CampaignID, dp.InstanceID = p.BenefitID, prev.CampaignID, p.InstanceID + l.progress[p.BenefitID] = dp + } + next := dropstate.Apply(prev, dropstate.Observation{ + Known: true, Claimed: p.Claimed, Minutes: p.MinutesWatched, Required: prev.Required, + }, now) + l.maybeNotifyProgress(ctx, next) + l.commit(ctx, next) + } +} + +func (l *Loop) onPubSub(ctx context.Context, e pubsubEvent) { + switch e.kind { + case "progress": + if r, ok := l.rows[e.drop]; ok && r.Required == 0 && e.required > 0 { + r.Required = e.required + l.rows[e.drop] = r + } + l.applyProgress(ctx, []platform.Progress{{BenefitID: e.drop, MinutesWatched: e.minutes}}) + case "claimable": + r, ok := l.rows[e.drop] + if !ok || r.Status == dropstate.Claimed { + return + } + dp := l.progress[e.drop] + dp.DropID, dp.CampaignID, dp.InstanceID = e.drop, r.CampaignID, e.instance + l.progress[e.drop] = dp + minutes := r.Minutes + if minutes < r.Required { + minutes = r.Required + } + l.commit(ctx, dropstate.Apply(r, dropstate.Observation{Known: true, Minutes: minutes, Required: r.Required}, l.cfg.Now())) + case "down": + if l.current != nil && l.current.Channel.ChannelID != "" && l.current.Channel.ChannelID == e.channel { + l.cooldowns[strings.ToLower(l.current.Channel.Channel)] = l.cfg.Now().Add(l.cfg.DownCooldown) + } + } +} + +func (l *Loop) maybeNotifyProgress(ctx context.Context, r dropstate.Row) { + step := l.cfg.ProgressNotifyStepPct + if step <= 0 || r.Required <= 0 || l.cfg.Notifier == nil { + return + } + pct := r.Minutes * 100 / r.Required + if pct > 100 { + pct = 100 + } + m := pct / step * step + last, seen := l.milestones[r.DropID] + l.milestones[r.DropID] = m + if !seen || m <= last { + return // first sight records a baseline; no restart storms + } + b, c := l.benefit(r.DropID) + l.notify(ctx, "progress", c, b, map[string]any{"cur_min": r.Minutes, "req_min": r.Required}) +} + +func (l *Loop) claimReady(ctx context.Context) { + now := l.cfg.Now() + attempted := false + for id, r := range l.rows { + if !dropstate.ReadyToClaim(r, now) { + continue + } + dp, ok := l.progress[id] + if !ok || dp.DropID == "" { + dp = platform.DropProgress{DropID: id, CampaignID: r.CampaignID} + } + next, res := claimer.Attempt(ctx, l.claimer, l.cfg.Session, r, dp, now) + slog.Info("pipeline claim attempt", "kind", "claim", "account", l.cfg.AccountID, "drop", id, + "outcome", res.Outcome.String(), "detail", res.Detail, "link", res.LinkURL) + l.commit(ctx, next) + attempted = true + } + if attempted { + l.reconcile(ctx) // corrects lost responses and auto-granted rewards + } +} + +func (l *Loop) replan(ctx context.Context) { + d := planner.Plan(l.plannerInput()) + if l.current != nil && l.current.Same(d) { + l.current.Serves, l.current.Reason = d.Serves, d.Reason + l.updateSnapshot() + return + } + l.stopSession() + l.current, l.lastSwap = &d, l.cfg.Now() + slog.Info("pipeline decision", "kind", "state", "account", l.cfg.AccountID, + "decision", d.Kind.String(), "channel", d.Channel.Channel, "serves", len(d.Serves), "reason", d.Reason) + if l.cfg.Notifier != nil { + _ = l.cfg.Notifier.Notify(ctx, "state", map[string]any{"account": l.cfg.AccountID, "state": stateString(&d)}) + } + if d.Kind != planner.Idle { + l.startSession(ctx, d) + } + l.updateSnapshot() +} + +func (l *Loop) startSession(ctx context.Context, d planner.Decision) { + sctx, cancel := context.WithCancel(ctx) + l.sessCancel = cancel + if l.subs != nil && d.Channel.ChannelID != "" { + l.subs.SubscribeChannel(l.cfg.AccountID, d.Channel.ChannelID) + l.subscribed = d.Channel.ChannelID + } + ticker := time.NewTicker(l.cfg.BeatEvery) + cfg := session.Config{ + Backend: l.cfg.Backend, Session: l.cfg.Session, Stream: d.Channel, + Serves: d.Serves, Ticks: ticker.C, StallPolls: l.cfg.StallPolls, + } + if d.Kind == planner.ForceWatch { + cfg.StallPolls = 0 // channel-points farming has no drop progress to stall on + } + go func() { + defer ticker.Stop() + session.Run(sctx, cfg, l.sessEvents) + }() +} + +func (l *Loop) stopSession() { + if l.sessCancel != nil { + l.sessCancel() + l.sessCancel = nil + } + if l.subs != nil && l.subscribed != "" { + l.subs.UnsubscribeChannel(l.cfg.AccountID, l.subscribed) + l.subscribed = "" + } +} + +// stateString maps a decision onto the v1 dashboard state vocabulary. +func stateString(d *planner.Decision) string { + if d == nil { + return "pick_campaign" + } + switch d.Kind { + case planner.Mine: + return "watching" + case planner.ForceWatch: + return "force_watch" + default: + return "idle" + } +} + +func (l *Loop) updateSnapshot() { + s := watcher.Snapshot{AccountID: l.cfg.AccountID, State: stateString(l.current)} + if d := l.current; d != nil && d.Kind != planner.Idle { + s.Channel, s.ViewerCount, s.StartedAt = d.Channel.Channel, d.Channel.ViewerCount, l.lastSwap + if len(d.Serves) > 0 { + b, c := l.benefit(d.Serves[0]) + r := l.rows[d.Serves[0]] + s.CampaignID, s.CampaignName, s.CampaignGame = c.ID, c.Name, c.Game + s.BenefitID, s.BenefitName, s.BenefitImage = b.ID, b.Name, b.ImageURL + s.MinutesWatched, s.RequiredMinutes = r.Minutes, r.Required + } + } + l.mu.Lock() + l.snap = s + l.mu.Unlock() +} diff --git a/internal/pipeline/loop/loop_test.go b/internal/pipeline/loop/loop_test.go new file mode 100644 index 0000000..b837dd4 --- /dev/null +++ b/internal/pipeline/loop/loop_test.go @@ -0,0 +1,243 @@ +package loop + +import ( + "context" + "strings" + "sync" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/aalejandrofer/grubdrops/internal/pipeline/dropstate" + "github.com/aalejandrofer/grubdrops/internal/pipeline/session" + "github.com/aalejandrofer/grubdrops/internal/platform" +) + +// fakeBackend: one game "G", campaign c1 with drop d1 (60 min). Tests +// mutate the fields under mu to script the platform. +type fakeBackend struct { + platform.Backend + mu sync.Mutex + camps []platform.Campaign + progress []platform.DropProgress + inventory []platform.Progress + live []platform.Stream + claimRes platform.ClaimResult + claims int + watching []string + stops int + beatFailOn string +} + +func (f *fakeBackend) ListActiveCampaigns(context.Context, platform.Session) ([]platform.Campaign, error) { + f.mu.Lock() + defer f.mu.Unlock() + return f.camps, nil +} +func (f *fakeBackend) DropProgress(context.Context, platform.Session, []platform.Campaign) ([]platform.DropProgress, error) { + f.mu.Lock() + defer f.mu.Unlock() + return append([]platform.DropProgress(nil), f.progress...), nil +} +func (f *fakeBackend) ListEligibleChannels(context.Context, platform.Session, platform.Campaign) ([]platform.Stream, error) { + f.mu.Lock() + defer f.mu.Unlock() + return append([]platform.Stream(nil), f.live...), nil +} +func (f *fakeBackend) InventoryProgress(context.Context, platform.Session) ([]platform.Progress, error) { + f.mu.Lock() + defer f.mu.Unlock() + return append([]platform.Progress(nil), f.inventory...), nil +} +func (f *fakeBackend) StartWatch(_ context.Context, _ platform.Session, s platform.Stream) (platform.WatchHandle, error) { + f.mu.Lock() + defer f.mu.Unlock() + f.watching = append(f.watching, s.Channel) + return platform.WatchHandle{Channel: s.Channel}, nil +} +func (f *fakeBackend) Heartbeat(_ context.Context, h platform.WatchHandle) error { + f.mu.Lock() + defer f.mu.Unlock() + if f.beatFailOn != "" && strings.EqualFold(h.Channel, f.beatFailOn) { + return context.DeadlineExceeded + } + return nil +} +func (f *fakeBackend) StopWatch(context.Context, platform.WatchHandle) error { + f.mu.Lock() + defer f.mu.Unlock() + f.stops++ + return nil +} +func (f *fakeBackend) ClaimDrop(context.Context, platform.Session, platform.DropProgress) platform.ClaimResult { + f.mu.Lock() + defer f.mu.Unlock() + f.claims++ + return f.claimRes +} + +type memStore struct { + mu sync.Mutex + rows map[string]dropstate.Row +} + +func (m *memStore) List(context.Context, string) ([]dropstate.Row, error) { + m.mu.Lock() + defer m.mu.Unlock() + var out []dropstate.Row + for _, r := range m.rows { + out = append(out, r) + } + return out, nil +} +func (m *memStore) Upsert(_ context.Context, r dropstate.Row) error { + m.mu.Lock() + defer m.mu.Unlock() + m.rows[r.DropID] = r + return nil +} +func (m *memStore) get(id string) dropstate.Row { + m.mu.Lock() + defer m.mu.Unlock() + return m.rows[id] +} + +type memHistory struct { + mu sync.Mutex + recorded []string + claimed map[string]bool +} + +func (h *memHistory) RecordClaimIfNew(_ context.Context, _ string, b platform.DropBenefit) (bool, error) { + h.mu.Lock() + defer h.mu.Unlock() + h.recorded = append(h.recorded, b.ID) + return true, nil +} +func (h *memHistory) ClaimedBenefitIDs(context.Context, string) (map[string]bool, error) { + return h.claimed, nil +} + +func setup(t *testing.T) (*fakeBackend, *memStore, *memHistory, Config) { + f := &fakeBackend{ + camps: []platform.Campaign{{ID: "c1", Platform: "twitch", Game: "G", Name: "Camp", Status: "active", + AccountLinked: true, AccountLinkChecked: true, + Benefits: []platform.DropBenefit{{ID: "d1", CampaignID: "c1", Name: "Drop", RequiredMinutes: 60}}}}, + progress: []platform.DropProgress{{DropID: "d1", CampaignID: "c1", Minutes: 10, Required: 60, Known: true}}, + inventory: []platform.Progress{{BenefitID: "d1", MinutesWatched: 10}}, + live: []platform.Stream{{Channel: "ch1", ViewerCount: 10}, {Channel: "ch2", ViewerCount: 5}}, + claimRes: platform.ClaimResult{Outcome: platform.ClaimOK}, + } + st := &memStore{rows: map[string]dropstate.Row{}} + h := &memHistory{} + cfg := Config{ + AccountID: "acc", Platform: "twitch", Backend: f, Store: st, History: h, + AllowGame: func(g string) bool { return g == "G" }, + ReconcileEvery: time.Hour, LiveEvery: time.Hour, + BeatEvery: 10 * time.Millisecond, StallPolls: 1000, // tests opt into stalls + } + return f, st, h, cfg +} + +func run(t *testing.T, cfg Config) (*Loop, context.CancelFunc) { + l, err := New(cfg) + require.NoError(t, err) + ctx, cancel := context.WithCancel(context.Background()) + done := make(chan struct{}) + go func() { _ = l.Run(ctx); close(done) }() + t.Cleanup(func() { cancel(); <-done }) + return l, cancel +} + +func TestLoop_ClaimedOnWebsiteMidWatch(t *testing.T) { + f, st, h, cfg := setup(t) + l, _ := run(t, cfg) + require.Eventually(t, func() bool { return l.Snapshot().Channel == "ch1" }, 2*time.Second, 5*time.Millisecond) + + f.mu.Lock() + f.inventory = []platform.Progress{{BenefitID: "d1", MinutesWatched: 20, Claimed: true}} + f.mu.Unlock() + + require.Eventually(t, func() bool { return st.get("d1").Status == dropstate.Claimed }, 2*time.Second, 5*time.Millisecond) + require.Eventually(t, func() bool { return l.Snapshot().State == "idle" }, 2*time.Second, 5*time.Millisecond) + h.mu.Lock() + assert.Contains(t, h.recorded, "d1") + h.mu.Unlock() + f.mu.Lock() + assert.Equal(t, 0, f.claims, "no claim call for an already-claimed drop") + assert.GreaterOrEqual(t, f.stops, 1) + f.mu.Unlock() +} + +func TestLoop_ClaimsWhenClaimable(t *testing.T) { + f, st, _, cfg := setup(t) + f.progress[0].Minutes = 60 + f.inventory[0].MinutesWatched = 60 + run(t, cfg) + require.Eventually(t, func() bool { return st.get("d1").Status == dropstate.Claimed }, 2*time.Second, 5*time.Millisecond) + f.mu.Lock() + assert.Equal(t, 1, f.claims) + f.mu.Unlock() +} + +func TestLoop_NeedsLinkBlocksWithoutLooping(t *testing.T) { + f, st, _, cfg := setup(t) + f.progress[0].Minutes = 60 + f.inventory[0].MinutesWatched = 60 + f.claimRes = platform.ClaimResult{Outcome: platform.ClaimNeedsLink} + run(t, cfg) + require.Eventually(t, func() bool { return st.get("d1").Reason == dropstate.NeedsLink }, 2*time.Second, 5*time.Millisecond) + time.Sleep(100 * time.Millisecond) + f.mu.Lock() + assert.Equal(t, 1, f.claims, "needs_link is not retried") + f.mu.Unlock() +} + +func TestLoop_StallCoolsChannelAndSwaps(t *testing.T) { + _, _, _, cfg := setup(t) + cfg.StallPolls = 3 + l, _ := run(t, cfg) + // Minutes never move, so after StallPolls the loop cools ch1 and moves on. + require.Eventually(t, func() bool { return l.Snapshot().Channel == "ch2" }, 3*time.Second, 5*time.Millisecond) +} + +func TestLoop_StaleStreamDownIgnored(t *testing.T) { + f, _, _, cfg := setup(t) + l, err := New(cfg) + require.NoError(t, err) + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + // Drive the loop manually: load, reconcile, refresh, plan. + require.NoError(t, l.load(ctx)) + l.reconcile(ctx) + l.refreshLive(ctx) + l.replan(ctx) + require.Equal(t, "ch1", l.current.Channel.Channel) + l.onSession(ctx, sessionEvent("ch-old-down")) + assert.Empty(t, l.cooldowns, "event from another channel must not cool anything") + l.stopSession() + _ = f +} + +func TestLoop_BridgesManualMarksOnLoad(t *testing.T) { + _, st, h, cfg := setup(t) + st.rows["d1"] = dropstate.Row{AccountID: "acc", DropID: "d1", CampaignID: "c1", Platform: "twitch", Status: dropstate.Accruing, Minutes: 10, Required: 60, Source: dropstate.FromPlatform} + h.claimed = map[string]bool{"d1": true} + l, err := New(cfg) + require.NoError(t, err) + require.NoError(t, l.load(context.Background())) + assert.Equal(t, dropstate.Claimed, st.get("d1").Status) + assert.Equal(t, dropstate.FromUser, st.get("d1").Source) +} + +func TestNew_RejectsBackendWithoutCapabilities(t *testing.T) { + type bare struct{ platform.Backend } + _, err := New(Config{AccountID: "a", Backend: bare{}}) + require.Error(t, err) +} + +func sessionEvent(ch string) session.Event { + return session.Event{Kind: session.StreamDown, Channel: ch} +} From 2f09d1c33e9e81058e535bd4407d4b6b33c8482a Mon Sep 17 00:00:00 2001 From: Alejandro Date: Sun, 27 Sep 2026 20:45:12 -0400 Subject: [PATCH 19/56] fix(pipeline-v2): restart session on serves change, retry claim history, drop stale events Co-Authored-By: Claude Opus 5.5 (1M context) --- internal/pipeline/loop/loop.go | 122 +++++++++++--- internal/pipeline/loop/loop_test.go | 227 ++++++++++++++++++++++++++- internal/pipeline/session/session.go | 9 ++ 3 files changed, 332 insertions(+), 26 deletions(-) diff --git a/internal/pipeline/loop/loop.go b/internal/pipeline/loop/loop.go index f7315dd..2b7c99e 100644 --- a/internal/pipeline/loop/loop.go +++ b/internal/pipeline/loop/loop.go @@ -92,17 +92,19 @@ type Loop struct { discoverAt time.Time // Owned by the Run goroutine. - rows map[string]dropstate.Row - camps []platform.Campaign - progress map[string]platform.DropProgress - live map[string][]platform.Stream - forceLive []platform.Stream - cooldowns map[string]time.Time - current *planner.Decision - lastSwap time.Time - sessCancel context.CancelFunc - subscribed string - milestones map[string]int + rows map[string]dropstate.Row + camps []platform.Campaign + progress map[string]platform.DropProgress + live map[string][]platform.Stream + forceLive []platform.Stream + cooldowns map[string]time.Time + current *planner.Decision + lastSwap time.Time + sessCancel context.CancelFunc + subscribed string + milestones map[string]int + gen int // bumped on every startSession; tags session events + pendingHistory map[string]bool // drop ids whose RecordClaimIfNew failed, retried each iteration } // New assembles the loop, checking the backend's pipeline capabilities once. @@ -142,14 +144,15 @@ func New(cfg Config) (*Loop, error) { } l := &Loop{ cfg: cfg, prog: prog, claimer: cl, - nudge: make(chan struct{}, 1), - pubsub: make(chan pubsubEvent, 64), - sessEvents: make(chan session.Event, 64), - rows: map[string]dropstate.Row{}, - progress: map[string]platform.DropProgress{}, - live: map[string][]platform.Stream{}, - cooldowns: map[string]time.Time{}, - milestones: map[string]int{}, + nudge: make(chan struct{}, 1), + pubsub: make(chan pubsubEvent, 64), + sessEvents: make(chan session.Event, 64), + rows: map[string]dropstate.Row{}, + progress: map[string]platform.DropProgress{}, + live: map[string][]platform.Stream{}, + cooldowns: map[string]time.Time{}, + milestones: map[string]int{}, + pendingHistory: map[string]bool{}, } l.prober, _ = cfg.Backend.(platform.ChannelProber) l.subs, _ = cfg.Backend.(platform.ChannelSubscriber) @@ -209,6 +212,7 @@ func (l *Loop) Run(ctx context.Context) error { l.reconcile(ctx) l.refreshLive(ctx) l.claimReady(ctx) + l.retryPendingHistory(ctx) l.replan(ctx) recT := time.NewTicker(l.cfg.ReconcileEvery) liveT := time.NewTicker(l.cfg.LiveEvery) @@ -232,6 +236,7 @@ func (l *Loop) Run(ctx context.Context) error { l.onPubSub(ctx, pe) } l.claimReady(ctx) + l.retryPendingHistory(ctx) l.replan(ctx) } } @@ -314,7 +319,8 @@ func (l *Loop) commit(ctx context.Context, next dropstate.Row) { b, c := l.benefit(next.DropID) if l.cfg.History != nil { if _, err := l.cfg.History.RecordClaimIfNew(ctx, l.cfg.AccountID, b); err != nil { - slog.Warn("pipeline: record claim history failed", "kind", "error", "account", l.cfg.AccountID, "drop", next.DropID, "err", err) + slog.Warn("pipeline: record claim history failed; will retry", "kind", "error", "account", l.cfg.AccountID, "drop", next.DropID, "err", err) + l.pendingHistory[next.DropID] = true } } slog.Info("pipeline drop claimed", "kind", "claim", "account", l.cfg.AccountID, "drop", next.DropID, "source", string(next.Source)) @@ -324,6 +330,23 @@ func (l *Loop) commit(ctx context.Context, next dropstate.Row) { } } +// retryPendingHistory retries any RecordClaimIfNew calls that failed on a +// prior commit, so a transient history-store outage doesn't permanently +// drop the record (spec 4.2/7). Runs after every loop iteration. +func (l *Loop) retryPendingHistory(ctx context.Context) { + if l.cfg.History == nil || len(l.pendingHistory) == 0 { + return + } + for id := range l.pendingHistory { + b, _ := l.benefit(id) + if _, err := l.cfg.History.RecordClaimIfNew(ctx, l.cfg.AccountID, b); err != nil { + slog.Warn("pipeline: retry claim history failed", "kind", "error", "account", l.cfg.AccountID, "drop", id, "err", err) + continue + } + delete(l.pendingHistory, id) + } +} + func (l *Loop) notify(ctx context.Context, event string, c platform.Campaign, b platform.DropBenefit, extra map[string]any) { if l.cfg.Notifier == nil { return @@ -425,6 +448,9 @@ func (l *Loop) isCurrent(channel string) bool { } func (l *Loop) onSession(ctx context.Context, ev session.Event) { + if ev.Gen != l.gen { + return // stale event from a session already stopped/restarted + } now := l.cfg.Now() switch ev.Kind { case session.Progress: @@ -477,6 +503,19 @@ func (l *Loop) onPubSub(ctx context.Context, e pubsubEvent) { dp := l.progress[e.drop] dp.DropID, dp.CampaignID, dp.InstanceID = e.drop, r.CampaignID, e.instance l.progress[e.drop] = dp + if r.Required == 0 { + // Required isn't known yet (platforms that surface the reward + // only via pubsub, e.g. Kick). A claimable event is itself proof + // the drop is done; routing it through Apply/derive would read + // required<=0 as "blocked: sub_only", which is wrong here. + next := r + next.Status, next.Reason = dropstate.Claimable, dropstate.NoReason + next.RetryAfter = time.Time{} + next.Source = dropstate.FromPlatform + next.UpdatedAt = l.cfg.Now() + l.commit(ctx, next) + return + } minutes := r.Minutes if minutes < r.Required { minutes = r.Required @@ -500,10 +539,14 @@ func (l *Loop) maybeNotifyProgress(ctx context.Context, r dropstate.Row) { } m := pct / step * step last, seen := l.milestones[r.DropID] - l.milestones[r.DropID] = m - if !seen || m <= last { + if !seen { + l.milestones[r.DropID] = m return // first sight records a baseline; no restart storms } + if m <= last { + return // never regress the recorded milestone (e.g. a stale re-sync) + } + l.milestones[r.DropID] = m b, c := l.benefit(r.DropID) l.notify(ctx, "progress", c, b, map[string]any{"cur_min": r.Minutes, "req_min": r.Required}) } @@ -530,10 +573,42 @@ func (l *Loop) claimReady(ctx context.Context) { } } +// sameServeSet reports whether a and b hold the same drop ids, ignoring order. +func sameServeSet(a, b []string) bool { + if len(a) != len(b) { + return false + } + counts := make(map[string]int, len(a)) + for _, id := range a { + counts[id]++ + } + for _, id := range b { + counts[id]-- + } + for _, n := range counts { + if n != 0 { + return false + } + } + return true +} + func (l *Loop) replan(ctx context.Context) { d := planner.Plan(l.plannerInput()) if l.current != nil && l.current.Same(d) { + servesChanged := !sameServeSet(l.current.Serves, d.Serves) l.current.Serves, l.current.Reason = d.Serves, d.Reason + if servesChanged && l.current.Kind != planner.Idle { + // The running session pinned its stall/claim bookkeeping to the + // serves set it started with (e.g. a precondition drop that just + // got claimed, handing the baton to the drop it gated). Restart + // on the same channel so the session tracks the new set instead + // of reading "no progress on my old drops" as a stall. Don't + // touch lastSwap: this isn't a channel swap, so the hysteresis + // window keeps counting from when we actually landed here. + l.stopSession() + l.startSession(ctx, *l.current) + } l.updateSnapshot() return } @@ -553,6 +628,7 @@ func (l *Loop) replan(ctx context.Context) { func (l *Loop) startSession(ctx context.Context, d planner.Decision) { sctx, cancel := context.WithCancel(ctx) l.sessCancel = cancel + l.gen++ if l.subs != nil && d.Channel.ChannelID != "" { l.subs.SubscribeChannel(l.cfg.AccountID, d.Channel.ChannelID) l.subscribed = d.Channel.ChannelID @@ -560,7 +636,7 @@ func (l *Loop) startSession(ctx context.Context, d planner.Decision) { ticker := time.NewTicker(l.cfg.BeatEvery) cfg := session.Config{ Backend: l.cfg.Backend, Session: l.cfg.Session, Stream: d.Channel, - Serves: d.Serves, Ticks: ticker.C, StallPolls: l.cfg.StallPolls, + Serves: d.Serves, Ticks: ticker.C, StallPolls: l.cfg.StallPolls, Gen: l.gen, } if d.Kind == planner.ForceWatch { cfg.StallPolls = 0 // channel-points farming has no drop progress to stall on diff --git a/internal/pipeline/loop/loop_test.go b/internal/pipeline/loop/loop_test.go index b837dd4..6640821 100644 --- a/internal/pipeline/loop/loop_test.go +++ b/internal/pipeline/loop/loop_test.go @@ -2,6 +2,7 @@ package loop import ( "context" + "errors" "strings" "sync" "testing" @@ -29,6 +30,12 @@ type fakeBackend struct { watching []string stops int beatFailOn string + // claimedAfterFail simulates a "lost claim response": the platform + // actually recorded the claim server-side even though ClaimDrop reports + // claimRes (e.g. ClaimFailed) to the caller. Flips every progress item's + // Claimed flag the moment ClaimDrop is called, so the next reconcile's + // DropProgress read sees it. + claimedAfterFail bool } func (f *fakeBackend) ListActiveCampaigns(context.Context, platform.Session) ([]platform.Campaign, error) { @@ -75,6 +82,11 @@ func (f *fakeBackend) ClaimDrop(context.Context, platform.Session, platform.Drop f.mu.Lock() defer f.mu.Unlock() f.claims++ + if f.claimedAfterFail { + for i := range f.progress { + f.progress[i].Claimed = true + } + } return f.claimRes } @@ -215,12 +227,37 @@ func TestLoop_StaleStreamDownIgnored(t *testing.T) { l.refreshLive(ctx) l.replan(ctx) require.Equal(t, "ch1", l.current.Channel.Channel) - l.onSession(ctx, sessionEvent("ch-old-down")) + // Same generation as the live session: this exercises the channel check, + // not the generation check (see TestLoop_StaleGenerationIgnored for that). + l.onSession(ctx, sessionEvent(l.gen, "ch-old-down")) assert.Empty(t, l.cooldowns, "event from another channel must not cool anything") l.stopSession() _ = f } +// TestLoop_StaleGenerationIgnored covers the generation stamp itself +// (item 4): an event tagged with a generation older than the current +// session must be dropped even though it matches the current channel/kind, +// otherwise a leftover event from a just-stopped session (e.g. a +// serves-change restart) could set a bogus cooldown. +func TestLoop_StaleGenerationIgnored(t *testing.T) { + f, _, _, cfg := setup(t) + l, err := New(cfg) + require.NoError(t, err) + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + require.NoError(t, l.load(ctx)) + l.reconcile(ctx) + l.refreshLive(ctx) + l.replan(ctx) + require.Equal(t, "ch1", l.current.Channel.Channel) + staleGen := l.gen - 1 + l.onSession(ctx, sessionEvent(staleGen, "ch1")) + assert.Empty(t, l.cooldowns, "event from a stale generation must not cool the live channel") + l.stopSession() + _ = f +} + func TestLoop_BridgesManualMarksOnLoad(t *testing.T) { _, st, h, cfg := setup(t) st.rows["d1"] = dropstate.Row{AccountID: "acc", DropID: "d1", CampaignID: "c1", Platform: "twitch", Status: dropstate.Accruing, Minutes: 10, Required: 60, Source: dropstate.FromPlatform} @@ -238,6 +275,190 @@ func TestNew_RejectsBackendWithoutCapabilities(t *testing.T) { require.Error(t, err) } -func sessionEvent(ch string) session.Event { - return session.Event{Kind: session.StreamDown, Channel: ch} +func sessionEvent(gen int, ch string) session.Event { + return session.Event{Kind: session.StreamDown, Channel: ch, Gen: gen} +} + +// TestLoop_ServesChangeRestartsSessionNoFalseStall covers item 1: a +// precondition chain (d1 -> d2) on the same channel. The running session +// pins its stall bookkeeping to the serves set it started with (d1). Once +// d1 is claimed, d2 becomes mineable and the decision's Serves flips to +// [d2] while Kind/Channel stay the same ("Same" per the planner). Without +// restarting the session on the new serves set, the session keeps watching +// for gains on d1 (now absent from inventory) and never credits d2's real +// gains, so it falsely stalls and cools ch1 even though d2 is accruing. +func TestLoop_ServesChangeRestartsSessionNoFalseStall(t *testing.T) { + f := &fakeBackend{ + camps: []platform.Campaign{{ + ID: "c1", Platform: "twitch", Game: "G", Name: "Camp", Status: "active", + AccountLinked: true, AccountLinkChecked: true, + Benefits: []platform.DropBenefit{ + {ID: "d1", CampaignID: "c1", Name: "Drop1", RequiredMinutes: 60}, + {ID: "d2", CampaignID: "c1", Name: "Drop2", RequiredMinutes: 100000, Preconditions: []string{"d1"}}, + }, + }}, + progress: []platform.DropProgress{{DropID: "d1", CampaignID: "c1", Minutes: 58, Required: 60, Known: true}}, + inventory: []platform.Progress{{BenefitID: "d1", MinutesWatched: 58}}, + live: []platform.Stream{{Channel: "ch1", ViewerCount: 10}}, + claimRes: platform.ClaimResult{Outcome: platform.ClaimOK}, + } + st := &memStore{rows: map[string]dropstate.Row{}} + cfg := Config{ + AccountID: "acc", Platform: "twitch", Backend: f, Store: st, + AllowGame: func(g string) bool { return g == "G" }, + ReconcileEvery: time.Hour, LiveEvery: time.Hour, + BeatEvery: 10 * time.Millisecond, StallPolls: 3, + } + l, _ := run(t, cfg) + require.Eventually(t, func() bool { return l.Snapshot().Channel == "ch1" }, 2*time.Second, 5*time.Millisecond) + + // Push d1 to completion so it gets claimed and d2's precondition clears. + for m := 59; m <= 60; m++ { + f.mu.Lock() + f.progress[0].Minutes = m + f.inventory[0].MinutesWatched = m + f.mu.Unlock() + time.Sleep(15 * time.Millisecond) + } + require.Eventually(t, func() bool { return st.get("d1").Status == dropstate.Claimed }, 2*time.Second, 5*time.Millisecond) + + // d1 leaves inventory (claimed, as real platforms do); d2 becomes the + // only servable drop and keeps rising. Cap well under its (huge) + // required minutes so it never itself completes mid-test. + f.mu.Lock() + f.progress = []platform.DropProgress{{DropID: "d2", CampaignID: "c1", Minutes: 0, Required: 100000, Known: true}} + f.inventory = []platform.Progress{{BenefitID: "d2", MinutesWatched: 0}} + f.mu.Unlock() + + minute := 0 + deadline := time.Now().Add(500 * time.Millisecond) + for time.Now().Before(deadline) { + minute++ + f.mu.Lock() + f.progress[0].Minutes = minute + f.inventory[0].MinutesWatched = minute + f.mu.Unlock() + time.Sleep(3 * time.Millisecond) + } + + assert.Equal(t, "ch1", l.Snapshot().Channel, "must not have swapped/cooled off ch1 on a false stall") + assert.NotEqual(t, "idle", l.Snapshot().State, "must still be watching, not idled out by a false stall") + assert.Greater(t, st.get("d2").Minutes, 0, "d2 must have kept accruing after the serves-set handoff") +} + +type flakyHistory struct { + mu sync.Mutex + failed bool + recorded []string +} + +func (h *flakyHistory) RecordClaimIfNew(_ context.Context, _ string, b platform.DropBenefit) (bool, error) { + h.mu.Lock() + defer h.mu.Unlock() + if !h.failed { + h.failed = true + return false, errors.New("transient history store error") + } + h.recorded = append(h.recorded, b.ID) + return true, nil +} +func (h *flakyHistory) ClaimedBenefitIDs(context.Context, string) (map[string]bool, error) { + return nil, nil +} +func (h *flakyHistory) has(id string) bool { + h.mu.Lock() + defer h.mu.Unlock() + for _, x := range h.recorded { + if x == id { + return true + } + } + return false +} + +// TestLoop_FailedHistoryRecordIsRetried covers item 2: a RecordClaimIfNew +// failure must not be dropped on the floor. The drop id goes into +// pendingHistory and gets retried on a later loop iteration. +func TestLoop_FailedHistoryRecordIsRetried(t *testing.T) { + f, st, _, cfg := setup(t) + f.progress[0].Minutes = 60 + f.inventory[0].MinutesWatched = 60 + fh := &flakyHistory{} + cfg.History = fh + run(t, cfg) + require.Eventually(t, func() bool { return st.get("d1").Status == dropstate.Claimed }, 2*time.Second, 5*time.Millisecond) + require.Eventually(t, func() bool { return fh.has("d1") }, 2*time.Second, 5*time.Millisecond) +} + +// TestLoop_LostClaimResponseCorrectedByReconcile covers item 3: ClaimDrop +// reports failure (a "lost response" - the platform actually recorded the +// claim but the client saw an error), but the platform's own progress read +// on the post-claim reconcile shows Claimed:true. The row must still end +// up Claimed/FromPlatform. +func TestLoop_LostClaimResponseCorrectedByReconcile(t *testing.T) { + f, st, _, cfg := setup(t) + f.progress[0].Minutes = 60 + f.inventory[0].MinutesWatched = 60 + f.claimRes = platform.ClaimResult{Outcome: platform.ClaimFailed} + f.claimedAfterFail = true + run(t, cfg) + require.Eventually(t, func() bool { + r := st.get("d1") + return r.Status == dropstate.Claimed && r.Source == dropstate.FromPlatform + }, 2*time.Second, 5*time.Millisecond) + f.mu.Lock() + assert.GreaterOrEqual(t, f.claims, 1, "a claim attempt must have happened and been reported as failed") + f.mu.Unlock() +} + +type recNotifier struct { + mu sync.Mutex + events []string +} + +func (n *recNotifier) Notify(_ context.Context, event string, _ map[string]any) error { + n.mu.Lock() + defer n.mu.Unlock() + n.events = append(n.events, event) + return nil +} + +// TestMaybeNotifyProgress_MilestoneNeverLowers covers item 5: a later, +// lower reading of Minutes (e.g. a stale re-sync) must not lower the +// recorded milestone nor re-fire a notification once real progress passes it. +func TestMaybeNotifyProgress_MilestoneNeverLowers(t *testing.T) { + n := &recNotifier{} + l := &Loop{ + cfg: Config{ProgressNotifyStepPct: 10, Notifier: n}, + milestones: map[string]int{}, + } + ctx := context.Background() + l.maybeNotifyProgress(ctx, dropstate.Row{DropID: "d1", Minutes: 50, Required: 100}) // baseline @ 50% + l.maybeNotifyProgress(ctx, dropstate.Row{DropID: "d1", Minutes: 30, Required: 100}) // regressed reading + assert.Equal(t, 50, l.milestones["d1"], "milestone must not be lowered by a regressed reading") + l.maybeNotifyProgress(ctx, dropstate.Row{DropID: "d1", Minutes: 60, Required: 100}) // real forward progress + assert.Equal(t, 60, l.milestones["d1"]) + n.mu.Lock() + defer n.mu.Unlock() + assert.Equal(t, []string{"progress"}, n.events, "only genuine forward progress notifies") +} + +// TestOnPubSub_ClaimableWithUnknownRequiredSetsClaimableDirectly covers +// item 6: a claimable pubsub event for a drop whose Required is still 0 +// (not yet known) must set the row Claimable directly rather than going +// through Apply/derive, which reads required<=0 as "blocked: sub_only". +func TestOnPubSub_ClaimableWithUnknownRequiredSetsClaimableDirectly(t *testing.T) { + st := &memStore{rows: map[string]dropstate.Row{}} + l := &Loop{ + cfg: Config{AccountID: "acc", Store: st, Now: time.Now}, + rows: map[string]dropstate.Row{"d1": {AccountID: "acc", DropID: "d1", CampaignID: "c1", Status: dropstate.Eligible, Required: 0}}, + progress: map[string]platform.DropProgress{}, + } + l.onPubSub(context.Background(), pubsubEvent{kind: "claimable", drop: "d1", instance: "inst1"}) + r := l.rows["d1"] + assert.Equal(t, dropstate.Claimable, r.Status) + assert.Equal(t, dropstate.NoReason, r.Reason) + assert.True(t, r.RetryAfter.IsZero()) + assert.Equal(t, dropstate.FromPlatform, r.Source) + assert.Equal(t, "inst1", l.progress["d1"].InstanceID) } diff --git a/internal/pipeline/session/session.go b/internal/pipeline/session/session.go index 385f414..bed1662 100644 --- a/internal/pipeline/session/session.go +++ b/internal/pipeline/session/session.go @@ -22,6 +22,12 @@ type Event struct { Channel string Progress []platform.Progress Err error + // Gen is the generation stamp of the session that produced this event + // (Config.Gen, echoed verbatim). The loop bumps its generation on every + // startSession and drops events whose Gen doesn't match the current one, + // so a stale event queued by a session that already got stopped (e.g. on + // a serves-set restart) can't be mistaken for one from the live session. + Gen int } type Config struct { @@ -33,6 +39,8 @@ type Config struct { // minute per beacon, so a slower cadence under-credits. Ticks <-chan time.Time StallPolls int + // Gen is stamped onto every Event this run produces. See Event.Gen. + Gen int } // Run starts the watch, beats immediately and on every tick, and stops the @@ -41,6 +49,7 @@ func Run(ctx context.Context, cfg Config, out chan<- Event) { ch := cfg.Stream.Channel send := func(e Event) { e.Channel = ch + e.Gen = cfg.Gen select { case out <- e: case <-ctx.Done(): From d6b6d5e29ac1d194a87065fc13fa10be29ce2460 Mon Sep 17 00:00:00 2001 From: Alejandro Date: Sun, 27 Sep 2026 20:53:09 -0400 Subject: [PATCH 20/56] feat(pipeline-v2): opt-in GRUB_PIPELINE=v2 wiring and scheduler interfaces Co-Authored-By: Claude Sonnet 5 --- cmd/miner/main.go | 75 ++++++++++++++++++++++++++++ cmd/miner/main_test.go | 24 +++++++++ internal/scheduler/discovery.go | 9 +++- internal/scheduler/state.go | 20 +++++--- internal/scheduler/v2_runner_test.go | 33 ++++++++++++ internal/store/campaign_persister.go | 4 ++ 6 files changed, 155 insertions(+), 10 deletions(-) create mode 100644 internal/scheduler/v2_runner_test.go diff --git a/cmd/miner/main.go b/cmd/miner/main.go index 2f17c0c..eeb1d8f 100644 --- a/cmd/miner/main.go +++ b/cmd/miner/main.go @@ -31,6 +31,7 @@ import ( mlog "github.com/aalejandrofer/grubdrops/internal/log" "github.com/aalejandrofer/grubdrops/internal/netutil" "github.com/aalejandrofer/grubdrops/internal/notify" + "github.com/aalejandrofer/grubdrops/internal/pipeline/loop" "github.com/aalejandrofer/grubdrops/internal/platform" "github.com/aalejandrofer/grubdrops/internal/platform/kick" "github.com/aalejandrofer/grubdrops/internal/platform/twitch" @@ -336,6 +337,16 @@ func run() error { // and the /drops Past + /history views stay empty. claimRecorder := store.NewClaimRecorder(q) + // dropStore is the pipeline v2 per-account drop_state table. Backfill + // seeds it from the v1 in-progress/claims state on every boot so a + // GRUB_PIPELINE=v2 opt-in mid-lifecycle doesn't start from scratch. + dropStore := store.NewDropStateStore(q) + if n, err := store.BackfillDropState(ctx, q, time.Now()); err != nil { + logger.Warn("pipeline v2: drop_state backfill failed", "err", err) + } else if n > 0 { + logger.Info("pipeline v2: drop_state backfilled from v1 state", "rows", n) + } + // Per-account direct-Twitch backends. The direct twitch.Backend holds // per-account state (auth, userID/userLogin caches, its own PubSub // socket), so sharing ONE instance across accounts races their tokens @@ -466,6 +477,36 @@ func run() error { forceCollected := loadCollectOverrides(ctx, q) persistedSkips, skipRecorder, skipClearer := loadSkipOverrides(ctx, q) + if pipelineModeFor(ctx, q, a.ID) == "v2" { + prio, err := dropStore.StreamerPriority(ctx, a.ID) + if err != nil { + logger.Warn("pipeline v2: load streamer priority failed", "account", a.ID, "err", err) + } + var force []string + if rows, err := q.ListForceChannels(ctx, a.ID); err == nil { + for _, r := range rows { + force = append(force, r.Channel) + } + } + l, err := loop.New(loop.Config{ + AccountID: a.ID, AccountLabel: a.DisplayName, Platform: a.Platform, + Backend: b, Session: sess, + Store: dropStore, History: claimRecorder, Persister: campaignPersister, + Notifier: notifier, + AllowGame: allow, AllowChannel: matchAnyChannel(prio), GameRank: rank, + PriorityMode: priorityMode, + ForceLinked: forceLinked, + StreamerPriority: prio, + ForceWatch: force, + ProgressNotifyStepPct: progressStep, + }) + if err == nil { + logger.Info("pipeline v2 enabled for account", "account", a.ID, "platform", a.Platform) + return scheduler.NewEntry(a.ID, l), nil + } + logger.Warn("pipeline v2 unavailable, falling back to v1", "account", a.ID, "err", err) + } + acctLabel := a.DisplayName w := watcher.New(watcher.Config{ AccountID: a.ID, AccountLabel: acctLabel, Platform: a.Platform, @@ -912,6 +953,40 @@ func decodeKickChannels(s platform.Session) []string { return out } +// pipelineModeFor picks v1 or v2 for an account: kv override first, then +// GRUB_PIPELINE, default v1. +func pipelineModeFor(ctx context.Context, q *gen.Queries, accountID string) string { + if v, err := q.GetSettingString(ctx, store.PipelineOverridePrefix+accountID); err == nil { + if s := string(v); s == "v1" || s == "v2" { + return s + } + } + if os.Getenv("GRUB_PIPELINE") == "v2" { + return "v2" + } + return "v1" +} + +// matchAnyChannel reports whether any campaign channel is one of logins. +// Nil when logins is empty, so the null-game gate stays off. +func matchAnyChannel(logins []string) func([]string) bool { + if len(logins) == 0 { + return nil + } + set := make(map[string]bool, len(logins)) + for _, l := range logins { + set[strings.ToLower(l)] = true + } + return func(chs []string) bool { + for _, c := range chs { + if set[strings.ToLower(c)] { + return true + } + } + return false + } +} + // loadAccountWhitelist materialises the per-account game allow-list // into match + rank closures the watcher consumes. When the account // has NO rows of its own, falls back to the global priority list diff --git a/cmd/miner/main_test.go b/cmd/miner/main_test.go index b3eff46..84105ce 100644 --- a/cmd/miner/main_test.go +++ b/cmd/miner/main_test.go @@ -64,3 +64,27 @@ func TestGenerateMasterKey_IsAcceptedByCryptor(t *testing.T) { require.NoError(t, err) assert.NotEqual(t, key, key2, "each keygen must be unique") } + +func TestPipelineModeFor(t *testing.T) { + ctx := context.Background() + db, err := store.Open(ctx, t.TempDir()+"/t.db") + require.NoError(t, err) + t.Cleanup(func() { _ = db.Close() }) + q := gen.New(db) + + t.Setenv("GRUB_PIPELINE", "") + assert.Equal(t, "v1", pipelineModeFor(ctx, q, "acc")) + t.Setenv("GRUB_PIPELINE", "v2") + assert.Equal(t, "v2", pipelineModeFor(ctx, q, "acc")) + require.NoError(t, q.UpsertSettingString(ctx, gen.UpsertSettingStringParams{Key: store.PipelineOverridePrefix + "acc", Value: []byte("v1")})) + assert.Equal(t, "v1", pipelineModeFor(ctx, q, "acc"), "per-account override beats env") + require.NoError(t, q.UpsertSettingString(ctx, gen.UpsertSettingStringParams{Key: store.PipelineOverridePrefix + "acc", Value: []byte("junk")})) + assert.Equal(t, "v2", pipelineModeFor(ctx, q, "acc"), "invalid override falls through") +} + +func TestMatchAnyChannel(t *testing.T) { + assert.Nil(t, matchAnyChannel(nil)) + m := matchAnyChannel([]string{"Fav"}) + assert.True(t, m([]string{"x", "fav"})) + assert.False(t, m([]string{"x"})) +} diff --git a/internal/scheduler/discovery.go b/internal/scheduler/discovery.go index 49e168d..610af4d 100644 --- a/internal/scheduler/discovery.go +++ b/internal/scheduler/discovery.go @@ -6,9 +6,14 @@ import ( "time" "github.com/aalejandrofer/grubdrops/internal/platform" - "github.com/aalejandrofer/grubdrops/internal/watcher" ) +// discoveryReporter is any runner that caches its last campaign discovery. +type discoveryReporter interface { + LastDiscovery() ([]platform.Campaign, time.Time) + AllowGame() func(game string) bool +} + // AccountDiscovery is the per-account view of the most recent // ListActiveCampaigns result. CapturedAt is zero when the watcher has // never completed a successful discovery tick yet (e.g. nopRunner-backed @@ -30,7 +35,7 @@ func (s *Scheduler) WatcherDiscoveries() []AccountDiscovery { defer s.mu.Unlock() out := make([]AccountDiscovery, 0, len(s.entries)) for _, e := range s.entries { - w, ok := e.runner.(*watcher.Watcher) + w, ok := e.runner.(discoveryReporter) if !ok { out = append(out, AccountDiscovery{AccountID: e.id}) continue diff --git a/internal/scheduler/state.go b/internal/scheduler/state.go index 58e702b..849c569 100644 --- a/internal/scheduler/state.go +++ b/internal/scheduler/state.go @@ -27,17 +27,22 @@ func idleState(r runner) string { return "needs_auth" } +// snapshotter is any runner that can describe its current work. Both the v1 +// *watcher.Watcher and the pipeline v2 loop implement it. +type snapshotter interface { + Snapshot() watcher.Snapshot +} + func (s *Scheduler) Snapshot() []AccountState { s.mu.Lock() defer s.mu.Unlock() out := make([]AccountState, 0, len(s.entries)) for _, e := range s.entries { - w, ok := e.runner.(*watcher.Watcher) - if !ok { - out = append(out, AccountState{AccountID: e.id, State: idleState(e.runner)}) + if sn, ok := e.runner.(snapshotter); ok { + out = append(out, AccountState{AccountID: e.id, State: sn.Snapshot().State}) continue } - out = append(out, AccountState{AccountID: e.id, State: w.State().String()}) + out = append(out, AccountState{AccountID: e.id, State: idleState(e.runner)}) } return out } @@ -51,12 +56,11 @@ func (s *Scheduler) WatcherSnapshots() []watcher.Snapshot { defer s.mu.Unlock() out := make([]watcher.Snapshot, 0, len(s.entries)) for _, e := range s.entries { - w, ok := e.runner.(*watcher.Watcher) - if !ok { - out = append(out, watcher.Snapshot{AccountID: e.id, State: idleState(e.runner)}) + if sn, ok := e.runner.(snapshotter); ok { + out = append(out, sn.Snapshot()) continue } - out = append(out, w.Snapshot()) + out = append(out, watcher.Snapshot{AccountID: e.id, State: idleState(e.runner)}) } return out } diff --git a/internal/scheduler/v2_runner_test.go b/internal/scheduler/v2_runner_test.go new file mode 100644 index 0000000..cd9f215 --- /dev/null +++ b/internal/scheduler/v2_runner_test.go @@ -0,0 +1,33 @@ +package scheduler + +import ( + "context" + "testing" + "time" + + "github.com/stretchr/testify/assert" + + "github.com/aalejandrofer/grubdrops/internal/platform" + "github.com/aalejandrofer/grubdrops/internal/watcher" +) + +type v2Runner struct{} + +func (v2Runner) Run(ctx context.Context) error { <-ctx.Done(); return ctx.Err() } +func (v2Runner) Snapshot() watcher.Snapshot { + return watcher.Snapshot{AccountID: "acc", State: "watching", Channel: "ch1"} +} +func (v2Runner) LastDiscovery() ([]platform.Campaign, time.Time) { + return []platform.Campaign{{ID: "c1"}}, time.Unix(1, 0) +} +func (v2Runner) AllowGame() func(string) bool { return func(string) bool { return true } } + +func TestScheduler_NonWatcherRunnerWithSnapshot(t *testing.T) { + s := New(Options{}) + s.AddEntry(NewEntry("acc", v2Runner{})) + snaps := s.WatcherSnapshots() + assert.Equal(t, "ch1", snaps[0].Channel) + assert.Equal(t, "watching", s.Snapshot()[0].State, "must not fall back to needs_auth") + disc := s.WatcherDiscoveries() + assert.Len(t, disc[0].Campaigns, 1) +} diff --git a/internal/store/campaign_persister.go b/internal/store/campaign_persister.go index 8aa39dd..b10f181 100644 --- a/internal/store/campaign_persister.go +++ b/internal/store/campaign_persister.go @@ -32,6 +32,10 @@ const CollectOverridePrefix = "collect_override:" // watcher pre-loads these into its skippedBenefits set at New() time. const SkipOverridePrefix = "skip_override:" +// PipelineOverridePrefix keys a per-account pipeline choice in kv. Full key: +// PipelineOverridePrefix + accountID, value "v1" or "v2". Beats GRUB_PIPELINE. +const PipelineOverridePrefix = "pipeline_override:" + // CampaignPersister upserts every Campaign + Benefit the watcher discovers // into the local DB so the /drops page can render past + current + // upcoming tabs even before anything has been claimed. From f2d8f7f4139487fb52bd41541548acf5672d0130 Mon Sep 17 00:00:00 2001 From: Alejandro Date: Sun, 27 Sep 2026 20:59:13 -0400 Subject: [PATCH 21/56] docs(pipeline-v2): changelog and spec refinements Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/CHANGELOG.md | 18 ++++++++++++++++++ .../specs/2026-09-27-pipeline-v2-design.md | 13 +++++++++++++ 2 files changed, 31 insertions(+) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index c31ef3f..d5c277c 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -4,6 +4,24 @@ All notable changes to GrubDrops. ## [Unreleased] +### Added + +- Pipeline v2 (opt-in, `GRUB_PIPELINE=v2` or kv `pipeline_override:`): drop state now comes from each campaign's platform details, so drops claimed on the website or with a lost claim response show as claimed and mining moves on (#50). +- Streamer priority for pipeline v2: per-account priority streamers are watched first, then any live channel in the category (#49). Seeded from the existing null-game channel list. +- `drop_state` and `account_streamer_priority` tables (migration 0016), with a one-time backfill from claim history, manual marks and ghost-skips. + +### Changed + +- Twitch claim code split into status fetch + classification (behaviour unchanged). +- Scheduler reads dashboard snapshots and discoveries through interfaces, so v1 and v2 runners both report state. + +### Fixed (pipeline v2 only) + +- Claim failures back off 1m, 5m, 30m and stop after 5 attempts instead of looping forever. +- Kick link-required claims block with a reason instead of retrying every poll. +- Kick required minutes come from the platform, not an invented 120. +- Ghost-skipped drops are re-checked instead of being skipped forever. + ## [1.3.12] — 2026-08-06 ### Added diff --git a/docs/superpowers/specs/2026-09-27-pipeline-v2-design.md b/docs/superpowers/specs/2026-09-27-pipeline-v2-design.md index 0f5a416..3292486 100644 --- a/docs/superpowers/specs/2026-09-27-pipeline-v2-design.md +++ b/docs/superpowers/specs/2026-09-27-pipeline-v2-design.md @@ -391,3 +391,16 @@ verification (Twitch and Kick) on staging before any tag that enables v2. 2. Kick `/drops/progress` retention of claimed rewards. Either answer may adjust §4.2 mapping; the rest of the design is unaffected. + +Status: probes committed (behind the `live` build tag); results pending. Must also be run with a TV-client Twitch token once the #47 TV-client login lands. + +## 10. Refinements during planning + +1. Reconcile runs serially on the loop goroutine (single writer; session/PubSub events buffer meanwhile). +2. Timestamps stored as INTEGER unix seconds, 0 = unset; block_reason NOT NULL DEFAULT ''. +3. user_skip has no auto-expiry; cleared only by the user. +4. Platform interfaces named DropProgressSource, DropClaimer (ClaimDrop returns a typed ClaimResult), ChannelProber in internal/platform. +5. Every dropstate mutator leaves a platform-confirmed claim untouched; sub_only re-derives after its window like not_enrolled. +6. Session stall detection is per-drop, and the loop restarts the session on the same channel when the served-drop set changes; session events carry a generation so stale events are dropped. +7. A claimed=true from either Twitch source (details self or in-progress inventory) wins. +8. Kick rows listed without required_units fall back to campaign minutes (reporting 0 would block as sub_only). From c06365645eea37f87702820614b6df1a8ebb9e38 Mon Sep 17 00:00:00 2001 From: Alejandro Date: Sun, 27 Sep 2026 21:21:42 -0400 Subject: [PATCH 22/56] fix(pipeline-v2): final review fixes (backfill required, manual marks, unlinked, allow-list, integrity wall) - C1: unknown observations adopt the catalog requirement when the row has none; backfill seeds required minutes on skip, claim and mark rows. - I2: backfill applies manual marks after claim history so they stay user claims. - I3: campaign link block uses the new unlinked reason, lifted on the next definite answer; claim-level needs_link keeps its 24h hold. - I4: skip priority probing for restricted campaigns with no known allow-list. - I5: Twitch self:null is Known=false unless the in-progress inventory has it. - I6: integrity wall stops the session, reports auth_required, Run exits nil. - stopSession waits up to 15s for the session goroutine; same-channel restarts keep the PubSub subscription. - Claimable to Accruing/Eligible resets FailCount. - session_test.go back to mode 644. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/CHANGELOG.md | 10 +- .../specs/2026-09-27-pipeline-v2-design.md | 2 +- internal/pipeline/dropstate/dropstate.go | 17 +- internal/pipeline/dropstate/dropstate_test.go | 43 ++++- internal/pipeline/loop/loop.go | 91 ++++++++- internal/pipeline/loop/loop_test.go | 172 ++++++++++++++++++ internal/pipeline/reconcile/reconcile_test.go | 40 +++- internal/pipeline/session/session_test.go | 0 internal/platform/twitch/pipeline.go | 5 +- internal/platform/twitch/pipeline_test.go | 16 ++ internal/store/dropstate_backfill.go | 31 ++-- internal/store/dropstate_backfill_test.go | 7 + internal/store/gen/drop_state.sql.go | 21 ++- internal/store/queries/drop_state.sql | 4 +- 14 files changed, 417 insertions(+), 42 deletions(-) mode change 100755 => 100644 internal/pipeline/session/session_test.go diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index d5c277c..2d23eec 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -7,7 +7,7 @@ All notable changes to GrubDrops. ### Added - Pipeline v2 (opt-in, `GRUB_PIPELINE=v2` or kv `pipeline_override:`): drop state now comes from each campaign's platform details, so drops claimed on the website or with a lost claim response show as claimed and mining moves on (#50). -- Streamer priority for pipeline v2: per-account priority streamers are watched first, then any live channel in the category (#49). Seeded from the existing null-game channel list. +- Streamer priority for pipeline v2: per-account priority streamers are watched first, then any live channel in the category (#49). Seeded from the existing null-game channel list. Until the settings editor lands, v2 reads a one-time copy of the null-game channel list taken at upgrade. - `drop_state` and `account_streamer_priority` tables (migration 0016), with a one-time backfill from claim history, manual marks and ghost-skips. ### Changed @@ -21,6 +21,14 @@ All notable changes to GrubDrops. - Kick link-required claims block with a reason instead of retrying every poll. - Kick required minutes come from the platform, not an invented 120. - Ghost-skipped drops are re-checked instead of being skipped forever. +- Migrated Kick ghost-skips pick up the campaign's required minutes instead of staying blocked as sub-only; backfilled rows now carry required minutes. +- Manual "mark collected" marks migrate as user claims, not final platform claims. +- A campaign link block lifts on the next sync after the user links, instead of holding for 24h. +- Twitch priority streamers are not probed for restricted campaigns whose allow-list is unknown, so v2 never watches a channel the campaign doesn't credit. +- Twitch drops with no viewer record are no longer read as a definite "not enrolled". +- The Twitch integrity wall shows the account as needing re-auth and stops the loop, like v1. +- Session restarts wait for the previous watch to stop, and a restart on the same channel keeps its PubSub subscription. +- A drop that falls back from claimable resets its claim failure count. ## [1.3.12] — 2026-08-06 diff --git a/docs/superpowers/specs/2026-09-27-pipeline-v2-design.md b/docs/superpowers/specs/2026-09-27-pipeline-v2-design.md index 3292486..4a54d0b 100644 --- a/docs/superpowers/specs/2026-09-27-pipeline-v2-design.md +++ b/docs/superpowers/specs/2026-09-27-pipeline-v2-design.md @@ -101,7 +101,7 @@ One row per (account_id, drop_id). | `campaign_id` | TEXT | owning campaign | | `platform` | TEXT | `twitch` / `kick` | | `status` | TEXT | `eligible`, `accruing`, `claimable`, `claimed`, `blocked` | -| `block_reason` | TEXT NULL | `needs_link`, `claim_failed`, `sub_only`, `no_channels`, `expired`, `not_enrolled`, `user_skip` | +| `block_reason` | TEXT NULL | `needs_link` (claim-level, 24h hold), `unlinked` (campaign-level, lifted on the next definite answer), `claim_failed`, `sub_only`, `no_channels`, `expired`, `not_enrolled`, `user_skip` | | `minutes` | INTEGER | platform-reported watched minutes | | `required` | INTEGER | platform-reported required minutes | | `source` | TEXT | `platform` or `user` (who last set status) | diff --git a/internal/pipeline/dropstate/dropstate.go b/internal/pipeline/dropstate/dropstate.go index 17eaed2..91e2c0b 100644 --- a/internal/pipeline/dropstate/dropstate.go +++ b/internal/pipeline/dropstate/dropstate.go @@ -26,6 +26,10 @@ const ( Expired Reason = "expired" NotEnrolled Reason = "not_enrolled" UserSkip Reason = "user_skip" + // Unlinked is the campaign-level link block: the campaign needs an + // external account the user has not linked. Unlike the claim-level + // NeedsLink it has no hold; the next definite answer re-derives it. + Unlinked Reason = "unlinked" ) type Source string @@ -93,7 +97,7 @@ func derive(minutes, required int) (Status, Reason) { func retryFor(reason Reason, now time.Time) time.Time { switch reason { - case NeedsLink: + case NeedsLink, Unlinked: return now.Add(RetryNeedsLink) case ClaimFailed: return now.Add(RetryClaimFailed) @@ -128,6 +132,11 @@ func Apply(prev Row, obs Observation, now time.Time) Row { return r } if !obs.Known { + // A backfilled row carries no requirement; adopt the catalog's so a + // re-derive does not read (0, 0) as sub_only forever. + if prev.Required <= 0 && obs.Required > 0 { + r.Required = obs.Required + } switch { case prev.IsZero(): r.Required = obs.Required @@ -137,7 +146,7 @@ func Apply(prev Row, obs Observation, now time.Time) Row { case (prev.Status == Accruing || prev.Status == Claimable) && prev.Source == FromPlatform: return r.with(Blocked, NotEnrolled, now) case prev.Status == Blocked && (prev.Reason == NotEnrolled || prev.Reason == SubOnly) && !now.Before(prev.RetryAfter): - st, rs := derive(prev.Minutes, prev.Required) + st, rs := derive(prev.Minutes, r.Required) return r.with(st, rs, now) } return r @@ -163,7 +172,7 @@ func Apply(prev Row, obs Observation, now time.Time) Row { if st == Claimable && prev.Status == Claimable && now.Before(prev.RetryAfter) { return r // keep claim backoff } - if prev.Status == Blocked { + if prev.Status == Blocked || (prev.Status == Claimable && st != Claimable) { r.FailCount = 0 } r.Source = FromPlatform @@ -251,7 +260,7 @@ func BlockLink(prev Row, now time.Time) Row { if prev.Status == Claimed || (prev.Status == Blocked && prev.Reason == UserSkip) { return prev } - return prev.with(Blocked, NeedsLink, now) + return prev.with(Blocked, Unlinked, now) } // Mineable reports whether watching can still advance the drop. diff --git a/internal/pipeline/dropstate/dropstate_test.go b/internal/pipeline/dropstate/dropstate_test.go index 2b31ca7..5224f6a 100644 --- a/internal/pipeline/dropstate/dropstate_test.go +++ b/internal/pipeline/dropstate/dropstate_test.go @@ -132,7 +132,7 @@ func TestExpireAndBlockLink_LeaveClaimedAlone(t *testing.T) { assert.Equal(t, Claimed, Expire(c, t0).Status) assert.Equal(t, Claimed, BlockLink(c, t0).Status) assert.Equal(t, Expired, Expire(row(Accruing, NoReason, 10, 60), t0).Reason) - assert.Equal(t, NeedsLink, BlockLink(row(Eligible, NoReason, 0, 60), t0).Reason) + assert.Equal(t, Unlinked, BlockLink(row(Eligible, NoReason, 0, 60), t0).Reason) sk := row(Blocked, UserSkip, 0, 60) assert.Equal(t, UserSkip, BlockLink(sk, t0).Reason) } @@ -182,3 +182,44 @@ func TestMutators_NeverDemotePlatformClaim(t *testing.T) { assert.Equal(t, UserSkip, got.Reason) assert.Equal(t, FromUser, got.Source) } + +// C1: a backfilled ghost-skip has Required 0. When the platform stays silent +// (Kick reports unlisted rewards Known=false) but the campaign now carries the +// requirement, the re-derive must use it instead of sticking at sub_only. +func TestApply_UnknownBackfilledRowAdoptsRequired(t *testing.T) { + for _, reason := range []Reason{NotEnrolled, SubOnly} { + prev := row(Blocked, reason, 0, 0) + prev.RetryAfter = t0.Add(-time.Minute) + got := Apply(prev, Observation{Required: 120}, t0) + assert.Equal(t, Eligible, got.Status, string(reason)) + assert.Equal(t, NoReason, got.Reason, string(reason)) + assert.Equal(t, 120, got.Required, string(reason)) + } +} + +// I3: a campaign-level link block is re-derived as soon as the platform gives a +// definite answer; there is no 24h hold once the user links. +func TestApply_UnlinkedReDerivedImmediately(t *testing.T) { + prev := BlockLink(row(Accruing, NoReason, 10, 60), t0) + require.Equal(t, Unlinked, prev.Reason) + require.True(t, t0.Add(time.Minute).Before(prev.RetryAfter)) + got := Apply(prev, Observation{Known: true, Minutes: 20, Required: 60}, t0.Add(time.Minute)) + assert.Equal(t, Accruing, got.Status) + assert.Equal(t, NoReason, got.Reason) + got = Apply(prev, Observation{Known: true, Minutes: 0, Required: 60}, t0.Add(time.Minute)) + assert.Equal(t, Eligible, got.Status) +} + +// A row that falls back from Claimable to Accruing/Eligible sheds its claim +// failure count so a later claim starts a fresh backoff ladder. +func TestApply_ClaimableDemotionResetsFailCount(t *testing.T) { + prev := row(Claimable, NoReason, 60, 60) + prev.FailCount = 3 + prev.RetryAfter = t0.Add(-time.Minute) + got := Apply(prev, Observation{Known: true, Minutes: 30, Required: 120}, t0) + assert.Equal(t, Accruing, got.Status) + assert.Equal(t, 0, got.FailCount) + got = Apply(prev, Observation{Known: true, Minutes: 0, Required: 60}, t0) + assert.Equal(t, Eligible, got.Status) + assert.Equal(t, 0, got.FailCount) +} diff --git a/internal/pipeline/loop/loop.go b/internal/pipeline/loop/loop.go index 2b7c99e..07ec3df 100644 --- a/internal/pipeline/loop/loop.go +++ b/internal/pipeline/loop/loop.go @@ -5,6 +5,7 @@ package loop import ( "context" + "errors" "fmt" "log/slog" "strings" @@ -101,7 +102,9 @@ type Loop struct { current *planner.Decision lastSwap time.Time sessCancel context.CancelFunc + sessDone chan struct{} // closed by the session goroutine once session.Run returns subscribed string + authBlocked bool // integrity wall hit: Run exits, the next reload re-spins the loop milestones map[string]int gen int // bumped on every startSession; tags session events pendingHistory map[string]bool // drop ids whose RecordClaimIfNew failed, retried each iteration @@ -209,16 +212,22 @@ func (l *Loop) Run(ctx context.Context) error { if err := l.load(ctx); err != nil { return err } + defer l.stopSession() l.reconcile(ctx) + if l.authBlocked { + return nil + } l.refreshLive(ctx) l.claimReady(ctx) + if l.authBlocked { + return nil + } l.retryPendingHistory(ctx) l.replan(ctx) recT := time.NewTicker(l.cfg.ReconcileEvery) liveT := time.NewTicker(l.cfg.LiveEvery) defer recT.Stop() defer liveT.Stop() - defer l.stopSession() for { select { case <-ctx.Done(): @@ -235,7 +244,13 @@ func (l *Loop) Run(ctx context.Context) error { case pe := <-l.pubsub: l.onPubSub(ctx, pe) } + if l.authBlocked { + return nil + } l.claimReady(ctx) + if l.authBlocked { + return nil + } l.retryPendingHistory(ctx) l.replan(ctx) } @@ -280,6 +295,10 @@ func (l *Loop) reconcile(ctx context.Context) { ForceLinked: l.cfg.ForceLinked, Persister: l.cfg.Persister, }, l.rows, l.cfg.Now()) if err != nil { + if errors.Is(err, platform.ErrIntegrityBlocked) { + l.onIntegrityBlocked(ctx) + return + } slog.Warn("pipeline reconcile failed; keeping last state", "kind", "error", "account", l.cfg.AccountID, "err", err) return } @@ -295,6 +314,22 @@ func (l *Loop) reconcile(ctx context.Context) { } } +// onIntegrityBlocked mirrors v1: stop watching, surface auth_required so the +// dashboard shows a re-auth banner, and let Run exit cleanly. The next reload +// re-spins the loop once the session is refreshed. +func (l *Loop) onIntegrityBlocked(ctx context.Context) { + slog.Warn("pipeline: integrity blocked, marking account needs_auth", "kind", "auth", "account", l.cfg.AccountID) + l.authBlocked = true + l.stopSession() + l.current = nil + l.mu.Lock() + l.snap = watcher.Snapshot{AccountID: l.cfg.AccountID, State: "auth_required"} + l.mu.Unlock() + if l.cfg.Notifier != nil { + _ = l.cfg.Notifier.Notify(ctx, "state", map[string]any{"account": l.cfg.AccountID, "state": "auth_required"}) + } +} + func (l *Loop) benefit(dropID string) (platform.DropBenefit, platform.Campaign) { for _, c := range l.camps { for _, b := range c.Benefits { @@ -425,7 +460,11 @@ func (l *Loop) refreshLive(ctx context.Context) { if err != nil { slog.Debug("pipeline: list channels failed", "account", l.cfg.AccountID, "campaign", c.ID, "err", err) } - if l.prober != nil { + // A restricted campaign whose allow-list the loop can't see (Twitch + // reports only a count) would credit none of the probed priority + // streamers it doesn't list, so skip probing it. + restrictedUnknown := c.AllowedChannelCount > 0 && len(c.AllowedChannels) == 0 + if l.prober != nil && !restrictedUnknown { if logins := allowedPriority(c, l.cfg.StreamerPriority); len(logins) > 0 { if ps, err := l.prober.ProbeChannels(ctx, l.cfg.Session, c, logins); err == nil { streams = mergeStreams(ps, streams) @@ -606,13 +645,17 @@ func (l *Loop) replan(ctx context.Context) { // of reading "no progress on my old drops" as a stall. Don't // touch lastSwap: this isn't a channel swap, so the hysteresis // window keeps counting from when we actually landed here. - l.stopSession() + // Same channel, so the PubSub subscription stays. + l.haltSession() l.startSession(ctx, *l.current) } l.updateSnapshot() return } - l.stopSession() + l.haltSession() + if d.Kind == planner.Idle { + l.unsubscribe() + } l.current, l.lastSwap = &d, l.cfg.Now() slog.Info("pipeline decision", "kind", "state", "account", l.cfg.AccountID, "decision", d.Kind.String(), "channel", d.Channel.Channel, "serves", len(d.Serves), "reason", d.Reason) @@ -629,9 +672,12 @@ func (l *Loop) startSession(ctx context.Context, d planner.Decision) { sctx, cancel := context.WithCancel(ctx) l.sessCancel = cancel l.gen++ - if l.subs != nil && d.Channel.ChannelID != "" { - l.subs.SubscribeChannel(l.cfg.AccountID, d.Channel.ChannelID) - l.subscribed = d.Channel.ChannelID + if want := d.Channel.ChannelID; l.subs != nil && want != l.subscribed { + l.unsubscribe() + if want != "" { + l.subs.SubscribeChannel(l.cfg.AccountID, want) + l.subscribed = want + } } ticker := time.NewTicker(l.cfg.BeatEvery) cfg := session.Config{ @@ -641,23 +687,52 @@ func (l *Loop) startSession(ctx context.Context, d planner.Decision) { if d.Kind == planner.ForceWatch { cfg.StallPolls = 0 // channel-points farming has no drop progress to stall on } + done := make(chan struct{}) + l.sessDone = done go func() { + defer close(done) defer ticker.Stop() session.Run(sctx, cfg, l.sessEvents) }() } -func (l *Loop) stopSession() { +// sessionStopWait bounds how long haltSession waits for the session +// goroutine (and its StopWatch) to finish. +const sessionStopWait = 15 * time.Second + +// haltSession cancels the running session and waits, bounded, for its +// goroutine to exit so StopWatch never overlaps the next StartWatch. It +// leaves the PubSub subscription alone. +func (l *Loop) haltSession() { if l.sessCancel != nil { l.sessCancel() l.sessCancel = nil } + if l.sessDone != nil { + t := time.NewTimer(sessionStopWait) + select { + case <-l.sessDone: + case <-t.C: + slog.Warn("pipeline: session did not stop in time", "kind", "error", "account", l.cfg.AccountID) + } + t.Stop() + l.sessDone = nil + } +} + +func (l *Loop) unsubscribe() { if l.subs != nil && l.subscribed != "" { l.subs.UnsubscribeChannel(l.cfg.AccountID, l.subscribed) l.subscribed = "" } } +// stopSession is the final stop: halt the session and drop the subscription. +func (l *Loop) stopSession() { + l.haltSession() + l.unsubscribe() +} + // stateString maps a decision onto the v1 dashboard state vocabulary. func stateString(d *planner.Decision) string { if d == nil { diff --git a/internal/pipeline/loop/loop_test.go b/internal/pipeline/loop/loop_test.go index 6640821..edfd8c7 100644 --- a/internal/pipeline/loop/loop_test.go +++ b/internal/pipeline/loop/loop_test.go @@ -3,6 +3,7 @@ package loop import ( "context" "errors" + "fmt" "strings" "sync" "testing" @@ -462,3 +463,174 @@ func TestOnPubSub_ClaimableWithUnknownRequiredSetsClaimableDirectly(t *testing.T assert.Equal(t, dropstate.FromPlatform, r.Source) assert.Equal(t, "inst1", l.progress["d1"].InstanceID) } + +// probeBackend adds a ChannelProber that reports a priority streamer live. +type probeBackend struct { + *fakeBackend + pmu sync.Mutex + probed []string // campaign ids probed +} + +func (p *probeBackend) ProbeChannels(_ context.Context, _ platform.Session, c platform.Campaign, _ []string) ([]platform.Stream, error) { + p.pmu.Lock() + defer p.pmu.Unlock() + p.probed = append(p.probed, c.ID) + return []platform.Stream{{Channel: "prio", ViewerCount: 1}}, nil +} + +// I4: a restricted campaign whose allow-list the loop can't see (Twitch +// reports only a count) must not get priority streamers probed in, or the +// planner would watch a channel the campaign doesn't credit. +func TestLoop_RestrictedCampaignWithoutAllowListSkipsPriorityProbe(t *testing.T) { + f, _, _, cfg := setup(t) + f.camps[0].AllowedChannelCount = 3 + pb := &probeBackend{fakeBackend: f} + cfg.Backend = pb + cfg.StreamerPriority = []string{"prio"} + l, err := New(cfg) + require.NoError(t, err) + ctx := context.Background() + require.NoError(t, l.load(ctx)) + l.reconcile(ctx) + l.refreshLive(ctx) + l.replan(ctx) + defer l.stopSession() + + pb.pmu.Lock() + assert.NotContains(t, pb.probed, "c1", "restricted campaign with unknown allow-list must not be probed") + pb.pmu.Unlock() + for _, s := range l.live["c1"] { + assert.NotEqual(t, "prio", s.Channel) + } + require.NotNil(t, l.current) + assert.Equal(t, "ch1", l.current.Channel.Channel) +} + +// Control for the above: an unrestricted campaign still gets priority probing. +func TestLoop_UnrestrictedCampaignProbesPriority(t *testing.T) { + f, _, _, cfg := setup(t) + pb := &probeBackend{fakeBackend: f} + cfg.Backend = pb + cfg.StreamerPriority = []string{"prio"} + l, err := New(cfg) + require.NoError(t, err) + ctx := context.Background() + require.NoError(t, l.load(ctx)) + l.reconcile(ctx) + l.refreshLive(ctx) + pb.pmu.Lock() + assert.Contains(t, pb.probed, "c1") + pb.pmu.Unlock() +} + +// integrityBackend fails discovery with the integrity wall. +type integrityBackend struct{ *fakeBackend } + +func (integrityBackend) ListActiveCampaigns(context.Context, platform.Session) ([]platform.Campaign, error) { + return nil, fmt.Errorf("gql: %w", platform.ErrIntegrityBlocked) +} + +// I6: the integrity wall surfaces as auth_required and the loop exits cleanly +// (like v1) so the next reload re-spins it once the session is refreshed. +func TestLoop_IntegrityBlockedExitsAuthRequired(t *testing.T) { + f, _, _, cfg := setup(t) + cfg.Backend = integrityBackend{f} + l, err := New(cfg) + require.NoError(t, err) + done := make(chan error, 1) + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + go func() { done <- l.Run(ctx) }() + select { + case err := <-done: + assert.NoError(t, err) + case <-time.After(2 * time.Second): + t.Fatal("Run did not exit on integrity wall") + } + assert.Equal(t, "auth_required", l.Snapshot().State) + f.mu.Lock() + assert.Empty(t, f.watching, "no session starts behind an integrity wall") + f.mu.Unlock() +} + +// subBackend records PubSub subscriptions and detects StopWatch overlapping +// the next StartWatch. +type subBackend struct { + *fakeBackend + smu sync.Mutex + subs []string + unsubs []string + inStop bool + overlaps int + starts int +} + +func (s *subBackend) SubscribeChannel(_, id string) { + s.smu.Lock() + defer s.smu.Unlock() + s.subs = append(s.subs, id) +} +func (s *subBackend) UnsubscribeChannel(_, id string) { + s.smu.Lock() + defer s.smu.Unlock() + s.unsubs = append(s.unsubs, id) +} +func (s *subBackend) StartWatch(ctx context.Context, sess platform.Session, st platform.Stream) (platform.WatchHandle, error) { + s.smu.Lock() + if s.inStop { + s.overlaps++ + } + s.starts++ + s.smu.Unlock() + return s.fakeBackend.StartWatch(ctx, sess, st) +} +func (s *subBackend) StopWatch(ctx context.Context, h platform.WatchHandle) error { + s.smu.Lock() + s.inStop = true + s.smu.Unlock() + time.Sleep(50 * time.Millisecond) + s.smu.Lock() + s.inStop = false + s.smu.Unlock() + return s.fakeBackend.StopWatch(ctx, h) +} +func (s *subBackend) counts() (subs, unsubs, starts, overlaps int) { + s.smu.Lock() + defer s.smu.Unlock() + return len(s.subs), len(s.unsubs), s.starts, s.overlaps +} + +// A serves-change restart on the same channel waits for the old session's +// StopWatch before the new StartWatch, and keeps the PubSub subscription. +func TestLoop_ServesRestartWaitsForStopAndKeepsSubscription(t *testing.T) { + f, _, _, cfg := setup(t) + f.live = []platform.Stream{{Channel: "ch1", ChannelID: "id1", ViewerCount: 10}} + sb := &subBackend{fakeBackend: f} + cfg.Backend = sb + l, err := New(cfg) + require.NoError(t, err) + ctx := context.Background() + require.NoError(t, l.load(ctx)) + l.reconcile(ctx) + l.refreshLive(ctx) + l.replan(ctx) + require.Eventually(t, func() bool { _, _, starts, _ := sb.counts(); return starts == 1 }, 2*time.Second, 5*time.Millisecond) + + // Force a serves-set change on the same decision. + l.current.Serves = []string{"stale"} + l.replan(ctx) + require.Eventually(t, func() bool { _, _, starts, _ := sb.counts(); return starts == 2 }, 2*time.Second, 5*time.Millisecond) + + subs, unsubs, _, overlaps := sb.counts() + assert.Equal(t, 0, overlaps, "StartWatch must not run while the previous StopWatch is in flight") + assert.Equal(t, 1, subs, "same-channel restart must not resubscribe") + assert.Equal(t, 0, unsubs, "same-channel restart must not unsubscribe") + + l.stopSession() + subs, unsubs, _, _ = sb.counts() + assert.Equal(t, 1, subs) + assert.Equal(t, 1, unsubs, "final stop unsubscribes") + f.mu.Lock() + assert.Equal(t, 2, f.stops, "final stop waited for StopWatch") + f.mu.Unlock() +} diff --git a/internal/pipeline/reconcile/reconcile_test.go b/internal/pipeline/reconcile/reconcile_test.go index f1e9333..7406e08 100644 --- a/internal/pipeline/reconcile/reconcile_test.go +++ b/internal/pipeline/reconcile/reconcile_test.go @@ -103,7 +103,7 @@ func TestRun_UnlinkedCampaignBlocked_UnlessForced(t *testing.T) { progress: []platform.DropProgress{{DropID: "d1", CampaignID: "c1", Required: 60, Known: true}}} res, err := Run(context.Background(), cfg(f, &fakePersister{}), nil, now) require.NoError(t, err) - assert.Equal(t, dropstate.NeedsLink, byDrop(res.Rows)["d1"].Reason) + assert.Equal(t, dropstate.Unlinked, byDrop(res.Rows)["d1"].Reason) cf := cfg(f, &fakePersister{}) cf.ForceLinked = func(id string) bool { return id == "c1" } @@ -112,6 +112,44 @@ func TestRun_UnlinkedCampaignBlocked_UnlessForced(t *testing.T) { assert.Equal(t, dropstate.Eligible, byDrop(res.Rows)["d1"].Status) } +// I3: once the user links, the next sync lifts the campaign link block at once. +func TestRun_LinkedAfterUnlinkedUnblocksImmediately(t *testing.T) { + f := &fakeBackend{camps: []platform.Campaign{camp("c1", "G", "d1", "d2")}, + progress: []platform.DropProgress{ + {DropID: "d1", CampaignID: "c1", Minutes: 20, Required: 60, Known: true}, + {DropID: "d2", CampaignID: "c1", Minutes: 0, Required: 60, Known: true}, + }} + blocked := func(id string) dropstate.Row { + return dropstate.Row{AccountID: "a", DropID: id, CampaignID: "c1", Platform: "twitch", + Status: dropstate.Blocked, Reason: dropstate.Unlinked, Minutes: 10, Required: 60, + Source: dropstate.FromPlatform, RetryAfter: now.Add(dropstate.RetryNeedsLink - time.Minute)} + } + prev := map[string]dropstate.Row{"d1": blocked("d1"), "d2": blocked("d2")} + res, err := Run(context.Background(), cfg(f, &fakePersister{}), prev, now) + require.NoError(t, err) + assert.Equal(t, dropstate.Accruing, byDrop(res.Rows)["d1"].Status) + assert.Equal(t, dropstate.Eligible, byDrop(res.Rows)["d2"].Status) +} + +// C1: a backfilled ghost-skip (Required 0, immediate retry) on Kick, where the +// platform reports unlisted rewards Known=false, becomes mineable again. +func TestRun_BackfilledKickSkipBecomesEligible(t *testing.T) { + c := camp("c1", "G", "d1") + c.Platform = "kick" + f := &fakeBackend{camps: []platform.Campaign{c}, + progress: []platform.DropProgress{{DropID: "d1", CampaignID: "c1", Required: 120, Known: false}}} + prev := map[string]dropstate.Row{"d1": {AccountID: "a", DropID: "d1", CampaignID: "c1", Platform: "kick", + Status: dropstate.Blocked, Reason: dropstate.NotEnrolled, Source: dropstate.FromPlatform, RetryAfter: now}} + cf := cfg(f, &fakePersister{}) + cf.Platform = "kick" + res, err := Run(context.Background(), cf, prev, now) + require.NoError(t, err) + r := byDrop(res.Rows)["d1"] + assert.Equal(t, dropstate.Eligible, r.Status) + assert.Equal(t, dropstate.NoReason, r.Reason) + assert.Equal(t, 120, r.Required) +} + func TestRun_ExpiredCampaign(t *testing.T) { c := camp("c1", "G", "d1") c.Status = "expired" diff --git a/internal/pipeline/session/session_test.go b/internal/pipeline/session/session_test.go old mode 100755 new mode 100644 diff --git a/internal/platform/twitch/pipeline.go b/internal/platform/twitch/pipeline.go index 078a5e1..496648a 100644 --- a/internal/platform/twitch/pipeline.go +++ b/internal/platform/twitch/pipeline.go @@ -31,7 +31,9 @@ func (d *discovery) dropProgress(ctx context.Context, sess platform.Session, cam continue } seen[td.ID] = true - dp := platform.DropProgress{DropID: td.ID, CampaignID: campaignID, Required: td.RequiredMinutesWatched, Known: true} + // self:null means Twitch said nothing about this viewer, which does + // not prove "not enrolled"; only a self object is a definite answer. + dp := platform.DropProgress{DropID: td.ID, CampaignID: campaignID, Required: td.RequiredMinutesWatched, Known: td.Self != nil} if td.Self != nil { dp.Minutes = td.Self.CurrentMinutesWatched dp.Claimed = td.Self.IsClaimed @@ -74,6 +76,7 @@ func (b *Backend) DropProgress(ctx context.Context, s platform.Session, camps [] if !ok { continue } + dps[i].Known = true // present in the in-progress inventory if p.MinutesWatched > dps[i].Minutes { dps[i].Minutes = p.MinutesWatched } diff --git a/internal/platform/twitch/pipeline_test.go b/internal/platform/twitch/pipeline_test.go index c97e6c1..afbb1c8 100644 --- a/internal/platform/twitch/pipeline_test.go +++ b/internal/platform/twitch/pipeline_test.go @@ -62,7 +62,23 @@ func TestDropProgress_NullSelfIsUnclaimed(t *testing.T) { require.Len(t, got, 1) assert.False(t, got[0].Claimed) assert.Equal(t, 0, got[0].Minutes) + assert.False(t, got[0].Known, "self:null proves nothing about enrollment") +} + +// A null self in details but a hit in the in-progress inventory is a definite +// answer: the inventory entry proves enrollment. +func TestDropProgress_NullSelfKnownViaInventory(t *testing.T) { + b := pipelineTestBackend(t, map[string]string{"c1": `{"data":{"user":{"dropCampaign":{"timeBasedDrops":[ + {"id":"d1","requiredMinutesWatched":60,"benefitEdges":[],"self":null} + ]}}}}`}, `{"data":{"currentUser":{"inventory":{"dropCampaignsInProgress":[ + {"id":"c1","timeBasedDrops":[{"id":"d1","self":{"currentMinutesWatched":7,"isClaimed":false,"dropInstanceID":""}}]} + ]}}}}`, 0) + got, err := b.DropProgress(context.Background(), platform.Session{AccessToken: "t"}, + []platform.Campaign{{ID: "c1", Platform: "twitch"}}) + require.NoError(t, err) + require.Len(t, got, 1) assert.True(t, got[0].Known) + assert.Equal(t, 7, got[0].Minutes) } func TestDropProgress_DedupesMultiItemDrop(t *testing.T) { diff --git a/internal/store/dropstate_backfill.go b/internal/store/dropstate_backfill.go index b871fcc..d9f4011 100644 --- a/internal/store/dropstate_backfill.go +++ b/internal/store/dropstate_backfill.go @@ -16,9 +16,12 @@ import ( const DropStateBackfilledKey = "drop_state_backfilled" // BackfillDropState seeds drop_state from v1 state: ghost-skips become -// not_enrolled with an immediate retry, manual marks become user claims, and -// claim history becomes platform claims. Later sources win. Runs once; the v1 -// kv keys are left in place so rolling back to v1 is safe. +// not_enrolled with an immediate retry, claim history becomes platform +// claims, and manual marks become user claims. Later sources win: manual +// marks go last because the mark-collected handler also writes a claims row, +// and a user assertion must not become a final platform claim. Every row +// carries the benefit's required minutes. Runs once; the v1 kv keys are left +// in place so rolling back to v1 is safe. func BackfillDropState(ctx context.Context, q *gen.Queries, now time.Time) (int, error) { if v, err := q.GetSettingString(ctx, DropStateBackfilledKey); err == nil && string(v) == "1" { return 0, nil @@ -26,7 +29,7 @@ func BackfillDropState(ctx context.Context, q *gen.Queries, now time.Time) (int, rows := map[string]dropstate.Row{} put := func(r dropstate.Row) { rows[r.AccountID+"|"+r.DropID] = r } - fromKV := func(prefix string, mk func(acct, drop, campaign, plat string) dropstate.Row) error { + fromKV := func(prefix string, mk func(acct, drop, campaign, plat string, required int) dropstate.Row) error { kvs, err := q.ListKVByPrefix(ctx, sql.NullString{String: prefix, Valid: true}) if err != nil { return fmt.Errorf("list %s: %w", prefix, err) @@ -49,20 +52,14 @@ func BackfillDropState(ctx context.Context, q *gen.Queries, now time.Time) (int, } return fmt.Errorf("get benefit campaign %s: %w", drop, err) } - put(mk(acct, drop, bc.CampaignID, bc.Platform)) + put(mk(acct, drop, bc.CampaignID, bc.Platform, int(bc.RequiredMinutes))) } return nil } - if err := fromKV(SkipOverridePrefix, func(acct, drop, campaign, plat string) dropstate.Row { + if err := fromKV(SkipOverridePrefix, func(acct, drop, campaign, plat string, required int) dropstate.Row { return dropstate.Row{AccountID: acct, DropID: drop, CampaignID: campaign, Platform: plat, Status: dropstate.Blocked, Reason: dropstate.NotEnrolled, Source: dropstate.FromPlatform, - RetryAfter: now, UpdatedAt: now} - }); err != nil { - return 0, err - } - if err := fromKV(CollectOverridePrefix, func(acct, drop, campaign, plat string) dropstate.Row { - return dropstate.Row{AccountID: acct, DropID: drop, CampaignID: campaign, Platform: plat, - Status: dropstate.Claimed, Source: dropstate.FromUser, UpdatedAt: now} + Required: required, RetryAfter: now, UpdatedAt: now} }); err != nil { return 0, err } @@ -72,7 +69,13 @@ func BackfillDropState(ctx context.Context, q *gen.Queries, now time.Time) (int, } for _, c := range claims { put(dropstate.Row{AccountID: c.AccountID, DropID: c.BenefitID, CampaignID: c.CampaignID, Platform: c.Platform, - Status: dropstate.Claimed, Source: dropstate.FromPlatform, UpdatedAt: now}) + Status: dropstate.Claimed, Source: dropstate.FromPlatform, Required: int(c.RequiredMinutes), UpdatedAt: now}) + } + if err := fromKV(CollectOverridePrefix, func(acct, drop, campaign, plat string, required int) dropstate.Row { + return dropstate.Row{AccountID: acct, DropID: drop, CampaignID: campaign, Platform: plat, + Status: dropstate.Claimed, Source: dropstate.FromUser, Required: required, UpdatedAt: now} + }); err != nil { + return 0, err } st := NewDropStateStore(q) diff --git a/internal/store/dropstate_backfill_test.go b/internal/store/dropstate_backfill_test.go index 7627989..2347a48 100644 --- a/internal/store/dropstate_backfill_test.go +++ b/internal/store/dropstate_backfill_test.go @@ -30,6 +30,8 @@ func TestBackfillDropState(t *testing.T) { rec := NewClaimRecorder(q) require.NoError(t, rec.RecordClaim(ctx, "acc-1", platform.DropBenefit{ID: "claimed1"})) require.NoError(t, rec.RecordClaim(ctx, "acc-1", platform.DropBenefit{ID: "both1"})) + // The mark-collected handler writes a claims row AND a collect_override. + require.NoError(t, rec.RecordClaim(ctx, "acc-1", platform.DropBenefit{ID: "marked1"})) for _, k := range []string{ CollectOverridePrefix + "marked1:acc-1", SkipOverridePrefix + "skipped1:acc-1", @@ -59,6 +61,11 @@ func TestBackfillDropState(t *testing.T) { assert.Equal(t, now, by["skipped1"].RetryAfter, "skips get one fresh evaluation") assert.Equal(t, dropstate.Claimed, by["both1"].Status) assert.Equal(t, "c1", by["claimed1"].CampaignID) + // Seeded rows carry the benefit's requirement so a later re-derive does + // not read Required 0 as sub_only. + for _, id := range []string{"claimed1", "marked1", "skipped1", "both1"} { + assert.Equal(t, 60, by[id].Required, id) + } // Idempotent: second run is a no-op. n, err = BackfillDropState(ctx, q, now) diff --git a/internal/store/gen/drop_state.sql.go b/internal/store/gen/drop_state.sql.go index bf3485b..bfececa 100644 --- a/internal/store/gen/drop_state.sql.go +++ b/internal/store/gen/drop_state.sql.go @@ -10,36 +10,38 @@ import ( ) const getBenefitCampaign = `-- name: GetBenefitCampaign :one -SELECT b.campaign_id, cp.platform +SELECT b.campaign_id, cp.platform, b.required_minutes FROM benefits b JOIN campaigns cp ON cp.id = b.campaign_id WHERE b.id = ? ` type GetBenefitCampaignRow struct { - CampaignID string `json:"campaign_id"` - Platform string `json:"platform"` + CampaignID string `json:"campaign_id"` + Platform string `json:"platform"` + RequiredMinutes int64 `json:"required_minutes"` } func (q *Queries) GetBenefitCampaign(ctx context.Context, id string) (GetBenefitCampaignRow, error) { row := q.db.QueryRowContext(ctx, getBenefitCampaign, id) var i GetBenefitCampaignRow - err := row.Scan(&i.CampaignID, &i.Platform) + err := row.Scan(&i.CampaignID, &i.Platform, &i.RequiredMinutes) return i, err } const listClaimsForBackfill = `-- name: ListClaimsForBackfill :many -SELECT c.account_id, c.benefit_id, b.campaign_id, a.platform +SELECT c.account_id, c.benefit_id, b.campaign_id, a.platform, b.required_minutes FROM claims c JOIN benefits b ON b.id = c.benefit_id JOIN accounts a ON a.id = c.account_id ` type ListClaimsForBackfillRow struct { - AccountID string `json:"account_id"` - BenefitID string `json:"benefit_id"` - CampaignID string `json:"campaign_id"` - Platform string `json:"platform"` + AccountID string `json:"account_id"` + BenefitID string `json:"benefit_id"` + CampaignID string `json:"campaign_id"` + Platform string `json:"platform"` + RequiredMinutes int64 `json:"required_minutes"` } func (q *Queries) ListClaimsForBackfill(ctx context.Context) ([]ListClaimsForBackfillRow, error) { @@ -56,6 +58,7 @@ func (q *Queries) ListClaimsForBackfill(ctx context.Context) ([]ListClaimsForBac &i.BenefitID, &i.CampaignID, &i.Platform, + &i.RequiredMinutes, ); err != nil { return nil, err } diff --git a/internal/store/queries/drop_state.sql b/internal/store/queries/drop_state.sql index c545b5a..2ed034a 100644 --- a/internal/store/queries/drop_state.sql +++ b/internal/store/queries/drop_state.sql @@ -20,13 +20,13 @@ ON CONFLICT(account_id, drop_id) DO UPDATE SET updated_at = excluded.updated_at; -- name: ListClaimsForBackfill :many -SELECT c.account_id, c.benefit_id, b.campaign_id, a.platform +SELECT c.account_id, c.benefit_id, b.campaign_id, a.platform, b.required_minutes FROM claims c JOIN benefits b ON b.id = c.benefit_id JOIN accounts a ON a.id = c.account_id; -- name: GetBenefitCampaign :one -SELECT b.campaign_id, cp.platform +SELECT b.campaign_id, cp.platform, b.required_minutes FROM benefits b JOIN campaigns cp ON cp.id = b.campaign_id WHERE b.id = ?; From 06cf7f67f78c303d08331e4cd8df1a72abfa5b8a Mon Sep 17 00:00:00 2001 From: Alejandro Date: Mon, 28 Sep 2026 01:38:43 -0400 Subject: [PATCH 23/56] fix(pipeline): plumb whitelisted games into v2 Session.Games TV-client Twitch sessions discover channel-first and need Session.Games; v2 never set it, so TV accounts discovered nothing. Co-Authored-By: Claude Opus 5.5 (1M context) --- cmd/miner/main.go | 1 + internal/pipeline/loop/loop.go | 10 +++++++++- internal/pipeline/loop/loop_test.go | 23 ++++++++++++++++++++++- 3 files changed, 32 insertions(+), 2 deletions(-) diff --git a/cmd/miner/main.go b/cmd/miner/main.go index 29a82e3..fafd829 100644 --- a/cmd/miner/main.go +++ b/cmd/miner/main.go @@ -494,6 +494,7 @@ func run() error { Store: dropStore, History: claimRecorder, Persister: campaignPersister, Notifier: notifier, AllowGame: allow, AllowChannel: matchAnyChannel(prio), GameRank: rank, + Games: names, PriorityMode: priorityMode, ForceLinked: forceLinked, StreamerPriority: prio, diff --git a/internal/pipeline/loop/loop.go b/internal/pipeline/loop/loop.go index 07ec3df..86da6b7 100644 --- a/internal/pipeline/loop/loop.go +++ b/internal/pipeline/loop/loop.go @@ -46,7 +46,12 @@ type Config struct { Persister reconcile.CampaignPersister Notifier Notifier - AllowGame func(game string) bool + AllowGame func(game string) bool + // Games is the whitelisted game display names, copied into + // Session.Games when the session carries none. TV-client Twitch + // sessions walk one game directory per name (chandisc.go); Android + // sessions ignore it. Mirrors watcher.Config.Games. + Games []string AllowChannel func(channels []string) bool GameRank func(game string) int PriorityMode string @@ -142,6 +147,9 @@ func New(cfg Config) (*Loop, error) { cfg.DownCooldown = 2 * time.Minute } cfg.Session.AccountID = cfg.AccountID + if cfg.Session.Games == nil { + cfg.Session.Games = cfg.Games + } if cfg.AllowGame != nil { cfg.Session.GameFilter = cfg.AllowGame } diff --git a/internal/pipeline/loop/loop_test.go b/internal/pipeline/loop/loop_test.go index edfd8c7..6c53840 100644 --- a/internal/pipeline/loop/loop_test.go +++ b/internal/pipeline/loop/loop_test.go @@ -37,11 +37,14 @@ type fakeBackend struct { // Claimed flag the moment ClaimDrop is called, so the next reconcile's // DropProgress read sees it. claimedAfterFail bool + // listGames records the Session.Games each ListActiveCampaigns saw. + listGames [][]string } -func (f *fakeBackend) ListActiveCampaigns(context.Context, platform.Session) ([]platform.Campaign, error) { +func (f *fakeBackend) ListActiveCampaigns(_ context.Context, s platform.Session) ([]platform.Campaign, error) { f.mu.Lock() defer f.mu.Unlock() + f.listGames = append(f.listGames, s.Games) return f.camps, nil } func (f *fakeBackend) DropProgress(context.Context, platform.Session, []platform.Campaign) ([]platform.DropProgress, error) { @@ -634,3 +637,21 @@ func TestLoop_ServesRestartWaitsForStopAndKeepsSubscription(t *testing.T) { assert.Equal(t, 2, f.stops, "final stop waited for StopWatch") f.mu.Unlock() } + +// TV-client Twitch sessions discover channel-first and need Session.Games +// (the whitelisted names); Config.Games must reach every discovery call. +func TestLoop_ConfigGamesReachSession(t *testing.T) { + f, _, _, cfg := setup(t) + cfg.Games = []string{"G", "Other Game"} + run(t, cfg) + require.Eventually(t, func() bool { + f.mu.Lock() + defer f.mu.Unlock() + return len(f.listGames) > 0 + }, 2*time.Second, 5*time.Millisecond) + f.mu.Lock() + defer f.mu.Unlock() + for _, g := range f.listGames { + assert.Equal(t, []string{"G", "Other Game"}, g) + } +} From 793dccd1312c39389bb060b288cdc9586d759ab3 Mon Sep 17 00:00:00 2001 From: Alejandro Date: Mon, 28 Sep 2026 01:39:30 -0400 Subject: [PATCH 24/56] fix(twitch): v2 details progress reports sub-gated drops as Required 0 Co-Authored-By: Claude Opus 5.5 (1M context) --- internal/platform/twitch/pipeline.go | 12 +++++++++++- internal/platform/twitch/pipeline_test.go | 15 +++++++++++++++ 2 files changed, 26 insertions(+), 1 deletion(-) diff --git a/internal/platform/twitch/pipeline.go b/internal/platform/twitch/pipeline.go index 496648a..5584195 100644 --- a/internal/platform/twitch/pipeline.go +++ b/internal/platform/twitch/pipeline.go @@ -12,6 +12,16 @@ import ( // Twitch can't report progress for them. func isSyntheticID(id string) bool { return strings.ContainsAny(id, "| ") } +// requiredMinutes maps a drop's watch requirement to the app-wide marker: +// sub-gated drops (requiredSubs > 0) are not watch-earnable whatever minutes +// Twitch lists, so they report 0 (issue #47, same rule as fetchDetails). +func requiredMinutes(minutes, subs int) int { + if subs > 0 { + return 0 + } + return minutes +} + // dropProgress reads DropCampaignDetails uncached, because self changes as // the viewer watches and claims. One entry per drop id. func (d *discovery) dropProgress(ctx context.Context, sess platform.Session, campaignID string) ([]platform.DropProgress, error) { @@ -33,7 +43,7 @@ func (d *discovery) dropProgress(ctx context.Context, sess platform.Session, cam seen[td.ID] = true // self:null means Twitch said nothing about this viewer, which does // not prove "not enrolled"; only a self object is a definite answer. - dp := platform.DropProgress{DropID: td.ID, CampaignID: campaignID, Required: td.RequiredMinutesWatched, Known: td.Self != nil} + dp := platform.DropProgress{DropID: td.ID, CampaignID: campaignID, Required: requiredMinutes(td.RequiredMinutesWatched, td.RequiredSubs), Known: td.Self != nil} if td.Self != nil { dp.Minutes = td.Self.CurrentMinutesWatched dp.Claimed = td.Self.IsClaimed diff --git a/internal/platform/twitch/pipeline_test.go b/internal/platform/twitch/pipeline_test.go index afbb1c8..bc21cc7 100644 --- a/internal/platform/twitch/pipeline_test.go +++ b/internal/platform/twitch/pipeline_test.go @@ -153,3 +153,18 @@ func TestBackend_SatisfiesPipelineInterfaces(t *testing.T) { var _ platform.DropClaimer = (*Backend)(nil) var _ platform.ChannelProber = (*Backend)(nil) } + +// #47: a sub-gated drop can't be earned by watching, whatever minutes Twitch +// lists. The details path reports Required 0 (the "not watch-earnable" marker). +func TestDropProgress_SubGatedDetailsRequiredZero(t *testing.T) { + b := pipelineTestBackend(t, map[string]string{"c1": `{"data":{"user":{"dropCampaign":{"timeBasedDrops":[ + {"id":"d1","requiredMinutesWatched":60,"requiredSubs":1,"benefitEdges":[],"self":{"currentMinutesWatched":0,"isClaimed":false,"dropInstanceID":""}}, + {"id":"d2","requiredMinutesWatched":60,"requiredSubs":0,"benefitEdges":[],"self":{"currentMinutesWatched":0,"isClaimed":false,"dropInstanceID":""}} + ]}}}}`}, emptyInventory, 0) + got, err := b.DropProgress(context.Background(), platform.Session{AccessToken: "t"}, + []platform.Campaign{{ID: "c1", Platform: "twitch"}}) + require.NoError(t, err) + require.Len(t, got, 2) + assert.Equal(t, 0, got[0].Required, "sub-gated drop is not watch-earnable") + assert.Equal(t, 60, got[1].Required) +} From a60a715cee75034071e3c64fa95bbe3c6c175141 Mon Sep 17 00:00:00 2001 From: Alejandro Date: Mon, 28 Sep 2026 01:40:39 -0400 Subject: [PATCH 25/56] fix(twitch): v2 reads TV-session drop progress from Inventory DropCampaignDetails returns dropCampaign:null for TV-client tokens, so v2 DropProgress skips it for TV sessions and builds observations from Inventory (minutes, requirement, claimed, instance id). v1 inventory() is untouched. Co-Authored-By: Claude Opus 5.5 (1M context) --- internal/platform/twitch/campaigns.go | 35 +++++++++++++++ internal/platform/twitch/pipeline.go | 27 +++++++++++- internal/platform/twitch/pipeline_test.go | 53 +++++++++++++++++++++++ 3 files changed, 114 insertions(+), 1 deletion(-) diff --git a/internal/platform/twitch/campaigns.go b/internal/platform/twitch/campaigns.go index 9f319da..c6b719f 100644 --- a/internal/platform/twitch/campaigns.go +++ b/internal/platform/twitch/campaigns.go @@ -474,6 +474,41 @@ func (d *discovery) drainAllowed() map[string][]string { return out } +// inventoryDropProgress reads the Inventory query as pipeline v2 per-drop +// observations keyed by campaign id. Used for TV-client sessions, whose +// DropCampaignDetails returns dropCampaign:null (#48): Inventory is the +// only per-viewer progress source. Unlike inventory() it keeps the +// campaign id and the drop's requirement (Required 0 for sub-gated drops, +// #47). Every entry is Known — Inventory always carries self for in-progress +// campaigns. Duplicate drop ids within a campaign are collapsed. +func (d *discovery) inventoryDropProgress(ctx context.Context, sess platform.Session) (map[string][]platform.DropProgress, error) { + var inv inventoryData + if err := d.c.gql(ctx, sess.AccessToken, OpInventory, nil, &inv); err != nil { + return nil, fmt.Errorf("inventory: %w", err) + } + out := map[string][]platform.DropProgress{} + seen := map[string]bool{} + for _, camp := range inv.CurrentUser.Inventory.DropCampaignsInProgress { + for _, td := range camp.TimeBasedDrops { + key := camp.ID + "\x00" + td.ID + if td.ID == "" || seen[key] { + continue + } + seen[key] = true + out[camp.ID] = append(out[camp.ID], platform.DropProgress{ + DropID: td.ID, + CampaignID: camp.ID, + Minutes: td.Self.CurrentMinutesWatched, + Required: requiredMinutes(td.RequiredMinutesWatched, td.RequiredSubs), + Claimed: td.Self.IsClaimed, + InstanceID: td.Self.DropInstanceID, + Known: true, + }) + } + } + return out, nil +} + // inventory returns the current watch progress for all in-progress drop campaigns. func (d *discovery) inventory(ctx context.Context, sess platform.Session) ([]platform.Progress, error) { var inv inventoryData diff --git a/internal/platform/twitch/pipeline.go b/internal/platform/twitch/pipeline.go index 5584195..8bb9a8c 100644 --- a/internal/platform/twitch/pipeline.go +++ b/internal/platform/twitch/pipeline.go @@ -59,8 +59,11 @@ func (d *discovery) dropProgress(ctx context.Context, sess platform.Session, cam // inventory); the in-progress inventory fills fresher minutes and the // instance id, and a claimed=true from either source wins — isClaimed is // a positive statement from the same self object and never reverts on -// Twitch. +// Twitch. TV-client sessions read Inventory only (see tvDropProgress). func (b *Backend) DropProgress(ctx context.Context, s platform.Session, camps []platform.Campaign) ([]platform.DropProgress, error) { + if s.ClientID == ClientTV { + return b.tvDropProgress(ctx, s, camps) + } inv, err := b.disc.inventory(ctx, s) if err != nil { return nil, err @@ -100,6 +103,28 @@ func (b *Backend) DropProgress(ctx context.Context, s platform.Session, camps [] return out, nil } +// tvDropProgress serves TV-client sessions from Inventory alone: Twitch +// returns dropCampaign:null for DropCampaignDetails on TV tokens (#48), so +// the details call is never made. Drops missing from Inventory get no entry, +// which the reconciler treats as unknown. +func (b *Backend) tvDropProgress(ctx context.Context, s platform.Session, camps []platform.Campaign) ([]platform.DropProgress, error) { + byCamp, err := b.disc.inventoryDropProgress(ctx, s) + if err != nil { + return nil, err + } + var out []platform.DropProgress + for _, c := range camps { + if isSyntheticID(c.ID) { + for _, bf := range c.Benefits { + out = append(out, platform.DropProgress{DropID: bf.ID, CampaignID: c.ID, Known: true, Unmineable: true}) + } + continue + } + out = append(out, byCamp[c.ID]...) + } + return out, nil +} + // classifyClaimStatus maps Twitch's claimDropRewards.status to an outcome. func classifyClaimStatus(status string) platform.ClaimResult { switch status { diff --git a/internal/platform/twitch/pipeline_test.go b/internal/platform/twitch/pipeline_test.go index bc21cc7..2d4a5d2 100644 --- a/internal/platform/twitch/pipeline_test.go +++ b/internal/platform/twitch/pipeline_test.go @@ -154,6 +154,59 @@ func TestBackend_SatisfiesPipelineInterfaces(t *testing.T) { var _ platform.ChannelProber = (*Backend)(nil) } +// tvPipelineBackend serves only Inventory; any DropCampaignDetails request +// fails the test (Twitch returns dropCampaign:null for TV-client tokens). +func tvPipelineBackend(t *testing.T, inventory string) *Backend { + t.Helper() + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + var req gqlRequest + require.NoError(t, json.NewDecoder(r.Body).Decode(&req)) + switch req.OperationName { + case OpInventory.Name: + _, _ = w.Write([]byte(inventory)) + default: + t.Errorf("unexpected op %q for a TV session", req.OperationName) + w.WriteHeader(http.StatusInternalServerError) + } + })) + t.Cleanup(srv.Close) + c := newTestClient(srv.URL) + return &Backend{c: c, disc: &discovery{c: c, userLogin: "testuser"}} +} + +func TestDropProgress_TVSessionReadsInventoryNoDetails(t *testing.T) { + b := tvPipelineBackend(t, `{"data":{"currentUser":{"inventory":{"dropCampaignsInProgress":[ + {"id":"c1","timeBasedDrops":[ + {"id":"d1","requiredMinutesWatched":60,"requiredSubs":0,"self":{"currentMinutesWatched":60,"isClaimed":true,"dropInstanceID":"i1"}}, + {"id":"d2","requiredMinutesWatched":120,"requiredSubs":0,"self":{"currentMinutesWatched":30,"isClaimed":false,"dropInstanceID":""}}, + {"id":"d3","requiredMinutesWatched":60,"requiredSubs":1,"self":{"currentMinutesWatched":0,"isClaimed":false,"dropInstanceID":""}} + ]}, + {"id":"cOther","timeBasedDrops":[ + {"id":"dx","requiredMinutesWatched":60,"self":{"currentMinutesWatched":5,"isClaimed":false,"dropInstanceID":""}} + ]} + ]}}}}`) + got, err := b.DropProgress(context.Background(), platform.Session{AccessToken: "t", ClientID: ClientTV}, + []platform.Campaign{ + {ID: "c1", Platform: "twitch", Benefits: []platform.DropBenefit{{ID: "d1"}, {ID: "d2"}, {ID: "d3"}, {ID: "d4"}}}, + {ID: "c2", Platform: "twitch", Benefits: []platform.DropBenefit{{ID: "e1"}}}, + }) + require.NoError(t, err) + assert.ElementsMatch(t, []platform.DropProgress{ + {DropID: "d1", CampaignID: "c1", Minutes: 60, Required: 60, Claimed: true, InstanceID: "i1", Known: true}, + {DropID: "d2", CampaignID: "c1", Minutes: 30, Required: 120, Known: true}, + {DropID: "d3", CampaignID: "c1", Minutes: 0, Required: 0, Known: true}, // sub-gated + }, got, "only requested campaigns' drops present in Inventory; absent drops get no entry") +} + +func TestDropProgress_TVSessionSyntheticStillUnmineable(t *testing.T) { + b := tvPipelineBackend(t, emptyInventory) + got, err := b.DropProgress(context.Background(), platform.Session{AccessToken: "t", ClientID: ClientTV}, + []platform.Campaign{{ID: "Game|Name", Platform: "twitch", Benefits: []platform.DropBenefit{{ID: "Game|Name_default"}}}}) + require.NoError(t, err) + require.Len(t, got, 1) + assert.True(t, got[0].Unmineable) +} + // #47: a sub-gated drop can't be earned by watching, whatever minutes Twitch // lists. The details path reports Required 0 (the "not watch-earnable" marker). func TestDropProgress_SubGatedDetailsRequiredZero(t *testing.T) { From afe4fd04d9374e51c860a63b0eda089e4d93bfa4 Mon Sep 17 00:00:00 2001 From: Alejandro Date: Mon, 28 Sep 2026 01:41:32 -0400 Subject: [PATCH 26/56] fix(twitch): bind session client profile in v2 entry points DropProgress, ClaimDrop and ProbeChannels can run before any v1 entry point bound the token, so a TV session went out under the Android Client-Id. Co-Authored-By: Claude Opus 5.5 (1M context) --- internal/platform/twitch/pipeline.go | 3 + internal/platform/twitch/pipeline_test.go | 69 ++++++++++++++++++++++- 2 files changed, 71 insertions(+), 1 deletion(-) diff --git a/internal/platform/twitch/pipeline.go b/internal/platform/twitch/pipeline.go index 8bb9a8c..139d04d 100644 --- a/internal/platform/twitch/pipeline.go +++ b/internal/platform/twitch/pipeline.go @@ -61,6 +61,7 @@ func (d *discovery) dropProgress(ctx context.Context, sess platform.Session, cam // a positive statement from the same self object and never reverts on // Twitch. TV-client sessions read Inventory only (see tvDropProgress). func (b *Backend) DropProgress(ctx context.Context, s platform.Session, camps []platform.Campaign) ([]platform.DropProgress, error) { + b.c.bind(s) if s.ClientID == ClientTV { return b.tvDropProgress(ctx, s, camps) } @@ -141,6 +142,7 @@ func classifyClaimStatus(status string) platform.ClaimResult { // before claiming (reconciler reads self.isAccountConnected), so Twitch has // no link outcome here. func (b *Backend) ClaimDrop(ctx context.Context, s platform.Session, d platform.DropProgress) platform.ClaimResult { + b.c.bind(s) userID, _ := b.watch.resolveUserID(ctx, s) status, err := b.claim.claimStatus(ctx, s, platform.DropBenefit{ID: d.DropID, CampaignID: d.CampaignID, InstanceID: d.InstanceID}, userID) if err != nil { @@ -152,6 +154,7 @@ func (b *Backend) ClaimDrop(ctx context.Context, s platform.Session, d platform. // ProbeChannels satisfies platform.ChannelProber: parallel live check of the // given logins, filtered to c.Game when set. func (b *Backend) ProbeChannels(ctx context.Context, s platform.Session, c platform.Campaign, logins []string) ([]platform.Stream, error) { + b.c.bind(s) return b.chans.listEligible(ctx, s, c, logins) } diff --git a/internal/platform/twitch/pipeline_test.go b/internal/platform/twitch/pipeline_test.go index 2d4a5d2..e04a283 100644 --- a/internal/platform/twitch/pipeline_test.go +++ b/internal/platform/twitch/pipeline_test.go @@ -5,6 +5,7 @@ import ( "encoding/json" "net/http" "net/http/httptest" + "sync" "testing" "github.com/stretchr/testify/assert" @@ -34,7 +35,8 @@ func pipelineTestBackend(t *testing.T, details map[string]string, inventory stri } })) t.Cleanup(srv.Close) - return &Backend{disc: &discovery{c: newTestClient(srv.URL), userLogin: "testuser"}} + c := newTestClient(srv.URL) + return &Backend{c: c, disc: &discovery{c: c, userLogin: "testuser"}} } const emptyInventory = `{"data":{"currentUser":{"inventory":{"dropCampaignsInProgress":[]}}}}` @@ -207,6 +209,71 @@ func TestDropProgress_TVSessionSyntheticStillUnmineable(t *testing.T) { assert.True(t, got[0].Unmineable) } +// clientIDRecorder is a gql server that records every request's Client-Id +// and answers each op with a minimal valid body. +func clientIDRecorder(t *testing.T) (*Backend, func() []string) { + t.Helper() + var mu sync.Mutex + var ids []string + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() + ids = append(ids, r.Header.Get("Client-Id")) + mu.Unlock() + var req gqlRequest + _ = json.NewDecoder(r.Body).Decode(&req) + switch req.OperationName { + case "CurrentUser": + _, _ = w.Write([]byte(`{"data":{"currentUser":{"id":"123","login":"u"}}}`)) + case OpClaimDrop.Name: + _, _ = w.Write([]byte(`{"data":{"claimDropRewards":{"status":"ELIGIBLE_FOR_ALL"}}}`)) + case OpInventory.Name: + _, _ = w.Write([]byte(emptyInventory)) + default: + _, _ = w.Write([]byte(`{"data":{"user":{"stream":null}}}`)) + } + })) + t.Cleanup(srv.Close) + return newForTest(srv.URL), func() []string { + mu.Lock() + defer mu.Unlock() + return append([]string(nil), ids...) + } +} + +func assertAllClientIDs(t *testing.T, ids []string, want string) { + t.Helper() + require.NotEmpty(t, ids, "no requests reached the server") + for _, id := range ids { + assert.Equal(t, want, id) + } +} + +// The v2 loop can call ClaimDrop before any v1 entry point bound the token, +// so ClaimDrop itself must bind the session's client profile. +func TestClaimDrop_TVSessionSendsTVClientID(t *testing.T) { + b, ids := clientIDRecorder(t) + res := b.ClaimDrop(context.Background(), platform.Session{AccessToken: "tv-tok", ClientID: ClientTV}, + platform.DropProgress{DropID: "d1", CampaignID: "c1", InstanceID: "i1"}) + assert.Equal(t, platform.ClaimOK, res.Outcome) + assertAllClientIDs(t, ids(), profileTV.ID) +} + +func TestDropProgress_TVSessionSendsTVClientID(t *testing.T) { + b, ids := clientIDRecorder(t) + _, err := b.DropProgress(context.Background(), platform.Session{AccessToken: "tv-tok", ClientID: ClientTV}, + []platform.Campaign{{ID: "c1", Platform: "twitch"}}) + require.NoError(t, err) + assertAllClientIDs(t, ids(), profileTV.ID) +} + +func TestProbeChannels_TVSessionSendsTVClientID(t *testing.T) { + b, ids := clientIDRecorder(t) + _, err := b.ProbeChannels(context.Background(), platform.Session{AccessToken: "tv-tok", ClientID: ClientTV}, + platform.Campaign{}, []string{"somechan"}) + require.NoError(t, err) + assertAllClientIDs(t, ids(), profileTV.ID) +} + // #47: a sub-gated drop can't be earned by watching, whatever minutes Twitch // lists. The details path reports Required 0 (the "not watch-earnable" marker). func TestDropProgress_SubGatedDetailsRequiredZero(t *testing.T) { From 98a94c6b4914ad1a87920a34a10835929ec60509 Mon Sep 17 00:00:00 2001 From: Alejandro Date: Mon, 28 Sep 2026 01:41:44 -0400 Subject: [PATCH 27/56] docs: changelog for v2 TV-client fixes Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/CHANGELOG.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index b9a8cc2..6a16599 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -29,6 +29,9 @@ All notable changes to GrubDrops. - The Twitch integrity wall shows the account as needing re-auth and stops the loop, like v1. - Session restarts wait for the previous watch to stop, and a restart on the same channel keeps its PubSub subscription. - A drop that falls back from claimable resets its claim failure count. +- v2 discovers campaigns for TV-client Twitch accounts (the whitelisted games now reach the session). +- v2 reads TV-account Twitch progress from the inventory, and sends every v2 request under the account's own Twitch client. +- v2 no longer mines subscription-only Twitch drops. ## [1.4.0] — 2026-09-28 From 0f4820a8c74a989a815679b1ceaa7be82d2ea3b2 Mon Sep 17 00:00:00 2001 From: Alejandro Date: Mon, 28 Sep 2026 02:59:15 -0400 Subject: [PATCH 28/56] feat(pipeline): log v1 heartbeat line from v2 session Dashboard HEARTBEATS/HR counts kind=heartbeat log lines; v2 logged none, so v2 accounts showed 0. session.Config gains AccountID for the field. Co-Authored-By: Claude Opus 5.5 (1M context) --- internal/pipeline/session/session.go | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/internal/pipeline/session/session.go b/internal/pipeline/session/session.go index bed1662..df131f5 100644 --- a/internal/pipeline/session/session.go +++ b/internal/pipeline/session/session.go @@ -4,6 +4,7 @@ package session import ( "context" + "log/slog" "time" "github.com/aalejandrofer/grubdrops/internal/platform" @@ -41,6 +42,8 @@ type Config struct { StallPolls int // Gen is stamped onto every Event this run produces. See Event.Gen. Gen int + // AccountID tags the heartbeat log line the dashboard counts. + AccountID string } // Run starts the watch, beats immediately and on every tick, and stops the @@ -79,6 +82,9 @@ func Run(ctx context.Context, cfg Config, out chan<- Event) { } return false } + // Same line as v1's watcher: the dashboard HEARTBEATS/HR card + // counts kind=heartbeat log lines. + slog.Info("watcher heartbeat sent", "kind", "heartbeat", "account", cfg.AccountID, "channel", ch) prog, err := cfg.Backend.InventoryProgress(ctx, cfg.Session) if err != nil { return true // transient; the next tick retries From b29a7c6bfcfbf75d2bd27662aa82a9a62a251713 Mon Sep 17 00:00:00 2001 From: Alejandro Date: Mon, 28 Sep 2026 02:59:22 -0400 Subject: [PATCH 29/56] fix(pipeline): Twitch stall claim-probe for drops claimed outside app MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit DropCampaignDetails returns self: null for claimed campaigns, so a drop claimed outside GrubDrops whose campaign left Inventory stayed Eligible and was watched, stalled and cooled forever. On stall, the loop now sends one claim per served Eligible drop absent from the last inventory read: OK/ALREADY_CLAIMED marks it claimed, NeedsLink blocks needs_link, failure blocks not_enrolled (1h re-check, FailCount untouched). Twitch only (StallClaimProbe); Kick keeps claimed rewards listed. Loop passes AccountID to the session heartbeat log. Spec §9 results + §10 item 9, changelog. Co-Authored-By: Claude Opus 5.5 (1M context) --- cmd/miner/main.go | 3 + docs/CHANGELOG.md | 2 + .../specs/2026-09-27-pipeline-v2-design.md | 11 ++- internal/pipeline/dropstate/dropstate.go | 13 +++ internal/pipeline/dropstate/dropstate_test.go | 23 +++++ internal/pipeline/loop/loop.go | 54 +++++++++++ internal/pipeline/loop/loop_test.go | 91 +++++++++++++++++++ 7 files changed, 196 insertions(+), 1 deletion(-) diff --git a/cmd/miner/main.go b/cmd/miner/main.go index 7ca61cf..26adb97 100644 --- a/cmd/miner/main.go +++ b/cmd/miner/main.go @@ -485,6 +485,9 @@ func run() error { StreamerPriority: prio, ForceWatch: force, ProgressNotifyStepPct: progressStep, + // Twitch hides drops claimed outside the app once their + // campaign leaves the Inventory; Kick keeps them listed. + StallClaimProbe: a.Platform == "twitch", }) if err == nil { logger.Info("pipeline v2 enabled for account", "account", a.ID, "platform", a.Platform) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index c82db16..cd490e9 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -32,6 +32,8 @@ All notable changes to GrubDrops. - v2 discovers campaigns for TV-client Twitch accounts (the whitelisted games now reach the session). - v2 reads TV-account Twitch progress from the inventory, and sends every v2 request under the account's own Twitch client. - v2 no longer mines subscription-only Twitch drops. +- A Twitch drop claimed outside GrubDrops whose campaign left the inventory is detected with one claim check when watching it stalls, instead of being watched again and again. +- v2 logs the per-beat heartbeat line, so v2 accounts show on the dashboard HEARTBEATS/HR card instead of 0. ## [1.4.1] — 2026-09-28 diff --git a/docs/superpowers/specs/2026-09-27-pipeline-v2-design.md b/docs/superpowers/specs/2026-09-27-pipeline-v2-design.md index 4a54d0b..83ea54c 100644 --- a/docs/superpowers/specs/2026-09-27-pipeline-v2-design.md +++ b/docs/superpowers/specs/2026-09-27-pipeline-v2-design.md @@ -392,7 +392,15 @@ verification (Twitch and Kick) on staging before any tag that enables v2. Either answer may adjust §4.2 mapping; the rest of the design is unaffected. -Status: probes committed (behind the `live` build tag); results pending. Must also be run with a TV-client Twitch token once the #47 TV-client login lands. +Results (2026-09-28, probes behind the `live` build tag): + +- Kick `/drops/progress` keeps claimed rewards listed with `claimed=true` + (15/15 on a prod account). +- Twitch `DropCampaignDetails` (Android client) returns `self: null` for + drops of campaigns the account has already claimed; the TV client has no + details at all. So Twitch claimed-outside-app detection for campaigns that + left Inventory relies on the stall claim-probe (§10 item 9). Active-campaign + `self` presence is untested (no in-progress campaigns on the probe account). ## 10. Refinements during planning @@ -404,3 +412,4 @@ Status: probes committed (behind the `live` build tag); results pending. Must al 6. Session stall detection is per-drop, and the loop restarts the session on the same channel when the served-drop set changes; session events carry a generation so stale events are dropped. 7. A claimed=true from either Twitch source (details self or in-progress inventory) wins. 8. Kick rows listed without required_units fall back to campaign minutes (reporting 0 would block as sub_only). +9. Twitch stall claim-probe: when a watched Eligible drop is absent from Inventory and the session stalls, the loop sends one claim; ALREADY_CLAIMED/OK marks it claimed, failure blocks it not_enrolled (1h re-check) without counting a claim failure. diff --git a/internal/pipeline/dropstate/dropstate.go b/internal/pipeline/dropstate/dropstate.go index 91e2c0b..3a3160e 100644 --- a/internal/pipeline/dropstate/dropstate.go +++ b/internal/pipeline/dropstate/dropstate.go @@ -216,6 +216,19 @@ func ClaimFailedAttempt(prev Row, now time.Time) Row { return r } +// ProbeNotEnrolled records a failed stall claim-probe: the platform neither +// tracks the drop nor accepts a claim for it, so block it not_enrolled (1h +// re-check). A probe is not a claim of a completed drop, so FailCount is +// left alone. Platform-confirmed claims and user skips are kept. +func ProbeNotEnrolled(prev Row, now time.Time) Row { + if (prev.Status == Claimed && prev.Source == FromPlatform) || (prev.Status == Blocked && prev.Reason == UserSkip) { + return prev + } + r := prev.with(Blocked, NotEnrolled, now) + r.Source = FromPlatform + return r +} + // MarkCollected is the user asserting the drop is claimed. func MarkCollected(prev Row, now time.Time) Row { if prev.Status == Claimed { diff --git a/internal/pipeline/dropstate/dropstate_test.go b/internal/pipeline/dropstate/dropstate_test.go index 5224f6a..aa5e971 100644 --- a/internal/pipeline/dropstate/dropstate_test.go +++ b/internal/pipeline/dropstate/dropstate_test.go @@ -223,3 +223,26 @@ func TestApply_ClaimableDemotionResetsFailCount(t *testing.T) { assert.Equal(t, Eligible, got.Status) assert.Equal(t, 0, got.FailCount) } + +// ProbeNotEnrolled is the stall claim-probe's failure outcome: blocked with the +// 1h re-check window, never counted as a claim failure. +func TestProbeNotEnrolled(t *testing.T) { + normal := row(Eligible, NoReason, 0, 60) + normal.Source = FromUser + got := ProbeNotEnrolled(normal, t0) + assert.Equal(t, Blocked, got.Status) + assert.Equal(t, NotEnrolled, got.Reason) + assert.Equal(t, FromPlatform, got.Source) + assert.Equal(t, t0.Add(RetryNotEnrolled), got.RetryAfter) + assert.Equal(t, 0, got.FailCount, "a probe is not a claim failure") + + withFails := row(Eligible, NoReason, 0, 60) + withFails.FailCount = 2 + assert.Equal(t, 2, ProbeNotEnrolled(withFails, t0).FailCount, "FailCount carried, never incremented") + + claimed := row(Claimed, NoReason, 60, 60) + assert.Equal(t, claimed, ProbeNotEnrolled(claimed, t0), "platform-confirmed claim untouched") + skipped := row(Blocked, UserSkip, 0, 60) + skipped.Source = FromUser + assert.Equal(t, skipped, ProbeNotEnrolled(skipped, t0), "user skip untouched") +} diff --git a/internal/pipeline/loop/loop.go b/internal/pipeline/loop/loop.go index 86da6b7..cb8f9b7 100644 --- a/internal/pipeline/loop/loop.go +++ b/internal/pipeline/loop/loop.go @@ -59,6 +59,12 @@ type Config struct { StreamerPriority []string ForceWatch []string ProgressNotifyStepPct int + // StallClaimProbe (Twitch): on a stall, send one claim for each served + // Eligible drop that was absent from the last inventory read. Twitch + // drops claimed outside the app whose campaign left the Inventory are + // otherwise invisible (DropCampaignDetails self: null) and would be + // watched forever. Kick keeps claimed rewards listed, so it leaves this off. + StallClaimProbe bool Now func() time.Time ReconcileEvery time.Duration // default 15m @@ -113,6 +119,9 @@ type Loop struct { milestones map[string]int gen int // bumped on every startSession; tags session events pendingHistory map[string]bool // drop ids whose RecordClaimIfNew failed, retried each iteration + // lastInv is the set of drop ids in the current session's most recent + // Progress event (the full inventory read). Reset on every session start. + lastInv map[string]bool } // New assembles the loop, checking the backend's pipeline capabilities once. @@ -501,6 +510,11 @@ func (l *Loop) onSession(ctx context.Context, ev session.Event) { now := l.cfg.Now() switch ev.Kind { case session.Progress: + inv := make(map[string]bool, len(ev.Progress)) + for _, p := range ev.Progress { + inv[p.BenefitID] = true + } + l.lastInv = inv l.applyProgress(ctx, ev.Progress) case session.StreamDown: if l.isCurrent(ev.Channel) { @@ -508,12 +522,50 @@ func (l *Loop) onSession(ctx context.Context, ev session.Event) { } case session.Stalled: if l.isCurrent(ev.Channel) { + if l.cfg.StallClaimProbe { + l.stallClaimProbe(ctx, now) + } l.cooldowns[strings.ToLower(ev.Channel)] = now.Add(l.cfg.StallCooldown) l.reconcile(ctx) } } } +// stallClaimProbe sends one claim for each served Eligible drop the last +// inventory read did not list. On Twitch such a drop is usually one claimed +// outside the app whose campaign left the Inventory: nothing reports it, so +// the claim answer is the only signal. OK/ALREADY_CLAIMED marks it claimed; +// a failure blocks it not_enrolled (1h re-check) without counting a claim +// failure, since it was never a completed drop. +func (l *Loop) stallClaimProbe(ctx context.Context, now time.Time) { + if l.current == nil { + return + } + for _, id := range l.current.Serves { + r, ok := l.rows[id] + if !ok || r.Status != dropstate.Eligible || l.lastInv[id] { + continue + } + dp, ok := l.progress[id] + if !ok || dp.DropID == "" { + dp = platform.DropProgress{DropID: id, CampaignID: r.CampaignID} + } + res := l.claimer.ClaimDrop(ctx, l.cfg.Session, dp) + var next dropstate.Row + switch res.Outcome { + case platform.ClaimOK, platform.ClaimAlready: + next = dropstate.ClaimOK(r, now) + case platform.ClaimNeedsLink: + next = dropstate.ClaimNeedsLink(r, now) + default: + next = dropstate.ProbeNotEnrolled(r, now) + } + slog.Info("pipeline stall claim-probe", "kind", "claim", "account", l.cfg.AccountID, "drop", id, + "outcome", res.Outcome.String(), "detail", res.Detail) + l.commit(ctx, next) + } +} + func (l *Loop) applyProgress(ctx context.Context, progs []platform.Progress) { now := l.cfg.Now() for _, p := range progs { @@ -680,6 +732,7 @@ func (l *Loop) startSession(ctx context.Context, d planner.Decision) { sctx, cancel := context.WithCancel(ctx) l.sessCancel = cancel l.gen++ + l.lastInv = nil if want := d.Channel.ChannelID; l.subs != nil && want != l.subscribed { l.unsubscribe() if want != "" { @@ -691,6 +744,7 @@ func (l *Loop) startSession(ctx context.Context, d planner.Decision) { cfg := session.Config{ Backend: l.cfg.Backend, Session: l.cfg.Session, Stream: d.Channel, Serves: d.Serves, Ticks: ticker.C, StallPolls: l.cfg.StallPolls, Gen: l.gen, + AccountID: l.cfg.AccountID, } if d.Kind == planner.ForceWatch { cfg.StallPolls = 0 // channel-points farming has no drop progress to stall on diff --git a/internal/pipeline/loop/loop_test.go b/internal/pipeline/loop/loop_test.go index 6c53840..2efdf75 100644 --- a/internal/pipeline/loop/loop_test.go +++ b/internal/pipeline/loop/loop_test.go @@ -655,3 +655,94 @@ func TestLoop_ConfigGamesReachSession(t *testing.T) { assert.Equal(t, []string{"G", "Other Game"}, g) } } + +// probeSetup scripts the Twitch "claimed outside the app, campaign left the +// Inventory" case: the platform reports nothing for d1 (DropCampaignDetails +// self: null, not in Inventory), so reconcile derives it Eligible and the +// session never sees a gain on it. +func probeSetup(t *testing.T) (*fakeBackend, *memStore, *memHistory, Config) { + f, st, h, cfg := setup(t) + f.progress = nil + f.inventory = nil + cfg.StallPolls = 3 + cfg.StallClaimProbe = true + return f, st, h, cfg +} + +func (f *fakeBackend) claimCount() int { + f.mu.Lock() + defer f.mu.Unlock() + return f.claims +} + +// (a) An Eligible served drop absent from the inventory gets one claim on +// stall; ALREADY_CLAIMED marks it claimed from the platform, records history +// and the planner moves on instead of re-watching it. +func TestLoop_StallClaimProbe_AlreadyClaimedMarksClaimed(t *testing.T) { + f, st, h, cfg := probeSetup(t) + f.claimRes = platform.ClaimResult{Outcome: platform.ClaimAlready} + l, _ := run(t, cfg) + require.Eventually(t, func() bool { + r := st.get("d1") + return r.Status == dropstate.Claimed && r.Source == dropstate.FromPlatform + }, 2*time.Second, 5*time.Millisecond) + require.Eventually(t, func() bool { return l.Snapshot().State == "idle" }, 2*time.Second, 5*time.Millisecond) + time.Sleep(50 * time.Millisecond) + assert.Equal(t, 1, f.claimCount(), "exactly one probe claim") + h.mu.Lock() + assert.Contains(t, h.recorded, "d1") + h.mu.Unlock() +} + +// (b) A failed probe blocks the drop not_enrolled without counting a claim +// failure, and it is not probed again. +func TestLoop_StallClaimProbe_FailedBlocksNotEnrolled(t *testing.T) { + f, st, _, cfg := probeSetup(t) + f.claimRes = platform.ClaimResult{Outcome: platform.ClaimFailed} + l, _ := run(t, cfg) + require.Eventually(t, func() bool { + r := st.get("d1") + return r.Status == dropstate.Blocked && r.Reason == dropstate.NotEnrolled + }, 2*time.Second, 5*time.Millisecond) + require.Eventually(t, func() bool { return l.Snapshot().State == "idle" }, 2*time.Second, 5*time.Millisecond) + time.Sleep(50 * time.Millisecond) + r := st.get("d1") + assert.Equal(t, 0, r.FailCount, "a probe is not a claim failure") + assert.Equal(t, dropstate.FromPlatform, r.Source) + assert.Equal(t, 1, f.claimCount(), "exactly one probe claim") +} + +// (c) A drop present in the inventory (really tracked, just frozen) is a +// plain stall: no probe claim. +func TestLoop_StallClaimProbe_SkipsDropInInventory(t *testing.T) { + f, st, _, cfg := probeSetup(t) + f.inventory = []platform.Progress{{BenefitID: "d1", MinutesWatched: 0}} + l, _ := run(t, cfg) + require.Eventually(t, func() bool { return l.Snapshot().Channel == "ch2" }, 3*time.Second, 5*time.Millisecond) + time.Sleep(50 * time.Millisecond) + assert.Equal(t, 0, f.claimCount()) + assert.Equal(t, dropstate.Eligible, st.get("d1").Status) +} + +// (d) Probe disabled (Kick): stall behaviour unchanged, no claim. +func TestLoop_StallClaimProbe_DisabledNoClaim(t *testing.T) { + f, st, _, cfg := probeSetup(t) + cfg.StallClaimProbe = false + l, _ := run(t, cfg) + require.Eventually(t, func() bool { return l.Snapshot().Channel == "ch2" }, 3*time.Second, 5*time.Millisecond) + time.Sleep(50 * time.Millisecond) + assert.Equal(t, 0, f.claimCount()) + assert.Equal(t, dropstate.Eligible, st.get("d1").Status) +} + +// (e) Only Eligible rows are probed: an Accruing row missing from the +// inventory is not claimed. +func TestLoop_StallClaimProbe_SkipsAccruing(t *testing.T) { + f, st, _, cfg := probeSetup(t) + f.progress = []platform.DropProgress{{DropID: "d1", CampaignID: "c1", Minutes: 10, Required: 60, Known: true}} + l, _ := run(t, cfg) + require.Eventually(t, func() bool { return l.Snapshot().Channel == "ch2" }, 3*time.Second, 5*time.Millisecond) + time.Sleep(50 * time.Millisecond) + assert.Equal(t, 0, f.claimCount()) + assert.Equal(t, dropstate.Accruing, st.get("d1").Status) +} From 502dc4cf76f72305613b58e24058073b9cc3a933 Mon Sep 17 00:00:00 2001 From: Alejandro Date: Mon, 28 Sep 2026 03:06:01 -0400 Subject: [PATCH 30/56] fix(pipeline): stricter stall claim-probe (second stall, empty status is failure) Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/CHANGELOG.md | 3 +- .../specs/2026-09-27-pipeline-v2-design.md | 2 +- internal/pipeline/loop/loop.go | 31 ++++-- internal/pipeline/loop/loop_test.go | 100 ++++++++++++++++-- internal/pipeline/session/session.go | 6 +- internal/platform/twitch/pipeline.go | 8 +- internal/platform/twitch/pipeline_test.go | 6 +- 7 files changed, 130 insertions(+), 26 deletions(-) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index cd490e9..1bbf607 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -33,7 +33,8 @@ All notable changes to GrubDrops. - v2 reads TV-account Twitch progress from the inventory, and sends every v2 request under the account's own Twitch client. - v2 no longer mines subscription-only Twitch drops. - A Twitch drop claimed outside GrubDrops whose campaign left the inventory is detected with one claim check when watching it stalls, instead of being watched again and again. -- v2 logs the per-beat heartbeat line, so v2 accounts show on the dashboard HEARTBEATS/HR card instead of 0. +- v2 heartbeats appear in the dashboard events feed, like v1's. +- The stall claim check only runs on a drop's second stall, and an empty Twitch claim status counts as a failure, so a blind check can never mark an unenrolled drop claimed. ## [1.4.1] — 2026-09-28 diff --git a/docs/superpowers/specs/2026-09-27-pipeline-v2-design.md b/docs/superpowers/specs/2026-09-27-pipeline-v2-design.md index 83ea54c..3f24aab 100644 --- a/docs/superpowers/specs/2026-09-27-pipeline-v2-design.md +++ b/docs/superpowers/specs/2026-09-27-pipeline-v2-design.md @@ -412,4 +412,4 @@ Results (2026-09-28, probes behind the `live` build tag): 6. Session stall detection is per-drop, and the loop restarts the session on the same channel when the served-drop set changes; session events carry a generation so stale events are dropped. 7. A claimed=true from either Twitch source (details self or in-progress inventory) wins. 8. Kick rows listed without required_units fall back to campaign minutes (reporting 0 would block as sub_only). -9. Twitch stall claim-probe: when a watched Eligible drop is absent from Inventory and the session stalls, the loop sends one claim; ALREADY_CLAIMED/OK marks it claimed, failure blocks it not_enrolled (1h re-check) without counting a claim failure. +9. Twitch stall claim-probe: when a watched Eligible drop is absent from Inventory and the session stalls on it for the second time (the first stall is a plain cooldown), the loop sends one claim; ALREADY_CLAIMED/OK marks it claimed, failure (including an empty claim status) blocks it not_enrolled (1h re-check) without counting a claim failure. diff --git a/internal/pipeline/loop/loop.go b/internal/pipeline/loop/loop.go index cb8f9b7..6494e56 100644 --- a/internal/pipeline/loop/loop.go +++ b/internal/pipeline/loop/loop.go @@ -122,6 +122,10 @@ type Loop struct { // lastInv is the set of drop ids in the current session's most recent // Progress event (the full inventory read). Reset on every session start. lastInv map[string]bool + // stallCount counts current-channel stalls per probe-qualifying drop + // (Eligible, absent from lastInv). The probe fires on the second stall; + // an entry is dropped once probed or once the row leaves Eligible. + stallCount map[string]int } // New assembles the loop, checking the backend's pipeline capabilities once. @@ -173,6 +177,7 @@ func New(cfg Config) (*Loop, error) { cooldowns: map[string]time.Time{}, milestones: map[string]int{}, pendingHistory: map[string]bool{}, + stallCount: map[string]int{}, } l.prober, _ = cfg.Backend.(platform.ChannelProber) l.subs, _ = cfg.Backend.(platform.ChannelSubscriber) @@ -362,6 +367,9 @@ func (l *Loop) benefit(dropID string) (platform.DropBenefit, platform.Campaign) func (l *Loop) commit(ctx context.Context, next dropstate.Row) { prev := l.rows[next.DropID] l.rows[next.DropID] = next + if next.Status != dropstate.Eligible { + delete(l.stallCount, next.DropID) + } if err := l.cfg.Store.Upsert(ctx, next); err != nil { slog.Warn("pipeline: persist drop state failed", "kind", "error", "account", l.cfg.AccountID, "drop", next.DropID, "err", err) } @@ -531,12 +539,13 @@ func (l *Loop) onSession(ctx context.Context, ev session.Event) { } } -// stallClaimProbe sends one claim for each served Eligible drop the last -// inventory read did not list. On Twitch such a drop is usually one claimed -// outside the app whose campaign left the Inventory: nothing reports it, so -// the claim answer is the only signal. OK/ALREADY_CLAIMED marks it claimed; -// a failure blocks it not_enrolled (1h re-check) without counting a claim -// failure, since it was never a completed drop. +// stallClaimProbe sends one claim for a served Eligible drop the last +// inventory read did not list, on that drop's second stall (the first is a +// plain cooldown). On Twitch such a drop is usually one claimed outside the +// app whose campaign left the Inventory: nothing reports it, so the claim +// answer is the only signal. OK/ALREADY_CLAIMED marks it claimed; a failure +// blocks it not_enrolled (1h re-check) without counting a claim failure, +// since it was never a completed drop. func (l *Loop) stallClaimProbe(ctx context.Context, now time.Time) { if l.current == nil { return @@ -546,21 +555,27 @@ func (l *Loop) stallClaimProbe(ctx context.Context, now time.Time) { if !ok || r.Status != dropstate.Eligible || l.lastInv[id] { continue } + l.stallCount[id]++ + if l.stallCount[id] < 2 { + continue + } + delete(l.stallCount, id) dp, ok := l.progress[id] if !ok || dp.DropID == "" { dp = platform.DropProgress{DropID: id, CampaignID: r.CampaignID} } res := l.claimer.ClaimDrop(ctx, l.cfg.Session, dp) var next dropstate.Row + kind := "probe" switch res.Outcome { case platform.ClaimOK, platform.ClaimAlready: - next = dropstate.ClaimOK(r, now) + next, kind = dropstate.ClaimOK(r, now), "claim" case platform.ClaimNeedsLink: next = dropstate.ClaimNeedsLink(r, now) default: next = dropstate.ProbeNotEnrolled(r, now) } - slog.Info("pipeline stall claim-probe", "kind", "claim", "account", l.cfg.AccountID, "drop", id, + slog.Info("pipeline stall claim-probe", "kind", kind, "account", l.cfg.AccountID, "drop", id, "outcome", res.Outcome.String(), "detail", res.Detail) l.commit(ctx, next) } diff --git a/internal/pipeline/loop/loop_test.go b/internal/pipeline/loop/loop_test.go index 2efdf75..11fab85 100644 --- a/internal/pipeline/loop/loop_test.go +++ b/internal/pipeline/loop/loop_test.go @@ -39,6 +39,8 @@ type fakeBackend struct { claimedAfterFail bool // listGames records the Session.Games each ListActiveCampaigns saw. listGames [][]string + // claimWatchLen is len(watching) at the first ClaimDrop call. + claimWatchLen int } func (f *fakeBackend) ListActiveCampaigns(_ context.Context, s platform.Session) ([]platform.Campaign, error) { @@ -86,6 +88,9 @@ func (f *fakeBackend) ClaimDrop(context.Context, platform.Session, platform.Drop f.mu.Lock() defer f.mu.Unlock() f.claims++ + if f.claims == 1 { + f.claimWatchLen = len(f.watching) + } if f.claimedAfterFail { for i := range f.progress { f.progress[i].Claimed = true @@ -675,12 +680,22 @@ func (f *fakeBackend) claimCount() int { return f.claims } -// (a) An Eligible served drop absent from the inventory gets one claim on -// stall; ALREADY_CLAIMED marks it claimed from the platform, records history -// and the planner moves on instead of re-watching it. +// watchedChannels returns every channel a session has started on, in order. +func (f *fakeBackend) watchedChannels() []string { + f.mu.Lock() + defer f.mu.Unlock() + return append([]string(nil), f.watching...) +} + +// (a) An Eligible served drop absent from the inventory is probed on its +// SECOND stall (ch1 stalls: plain cooldown; ch2 stalls: one claim). +// ALREADY_CLAIMED marks it claimed from the platform, records history, +// notifies once, and the planner moves on instead of re-watching it. func TestLoop_StallClaimProbe_AlreadyClaimedMarksClaimed(t *testing.T) { f, st, h, cfg := probeSetup(t) f.claimRes = platform.ClaimResult{Outcome: platform.ClaimAlready} + n := &recNotifier{} + cfg.Notifier = n l, _ := run(t, cfg) require.Eventually(t, func() bool { r := st.get("d1") @@ -689,13 +704,68 @@ func TestLoop_StallClaimProbe_AlreadyClaimedMarksClaimed(t *testing.T) { require.Eventually(t, func() bool { return l.Snapshot().State == "idle" }, 2*time.Second, 5*time.Millisecond) time.Sleep(50 * time.Millisecond) assert.Equal(t, 1, f.claimCount(), "exactly one probe claim") + f.mu.Lock() + assert.Equal(t, 2, f.claimWatchLen, "claim sent only after the second stall (ch1, then ch2)") + f.mu.Unlock() + assert.Equal(t, []string{"ch1", "ch2"}, f.watchedChannels()) h.mu.Lock() assert.Contains(t, h.recorded, "d1") h.mu.Unlock() + n.mu.Lock() + claimEvents := 0 + for _, e := range n.events { + if e == "claim" { + claimEvents++ + } + } + n.mu.Unlock() + assert.Equal(t, 1, claimEvents, "exactly one claim notification") +} + +// A single stall is a plain cooldown: no claim is sent. +func TestLoop_StallClaimProbe_FirstStallNoClaim(t *testing.T) { + f, st, _, cfg := probeSetup(t) + f.live = []platform.Stream{{Channel: "ch1", ViewerCount: 10}} // one channel: one stall, then idle + l, _ := run(t, cfg) + require.Eventually(t, func() bool { return len(f.watchedChannels()) == 1 }, 2*time.Second, 5*time.Millisecond) + require.Eventually(t, func() bool { return l.Snapshot().State == "idle" }, 2*time.Second, 5*time.Millisecond) + time.Sleep(50 * time.Millisecond) + assert.Equal(t, 0, f.claimCount(), "first stall must not probe") + assert.Equal(t, dropstate.Eligible, st.get("d1").Status) +} + +// A drop's stall count is cleared once its row leaves Eligible, so a later +// return to Eligible starts counting from zero again. +func TestLoop_StallClaimProbe_CountClearedWhenRowLeavesEligible(t *testing.T) { + f, _, _, cfg := probeSetup(t) + cfg.StallPolls = 1000 // the live session must not stall on its own + l, err := New(cfg) + require.NoError(t, err) + ctx := context.Background() + require.NoError(t, l.load(ctx)) + l.reconcile(ctx) + l.refreshLive(ctx) + l.replan(ctx) + defer l.stopSession() + require.Equal(t, "ch1", l.current.Channel.Channel) + require.Equal(t, dropstate.Eligible, l.rows["d1"].Status) + stall := func() { l.onSession(ctx, session.Event{Kind: session.Stalled, Channel: "ch1", Gen: l.gen}) } + + stall() // count 1 + assert.Equal(t, 0, f.claimCount()) + r := l.rows["d1"] + r.Status = dropstate.Accruing + l.commit(ctx, r) // leaves Eligible: count cleared + r.Status = dropstate.Eligible + l.commit(ctx, r) + stall() // count 1 again, not 2 + assert.Equal(t, 0, f.claimCount(), "count must restart after the row left Eligible") + stall() // count 2: probe + assert.Equal(t, 1, f.claimCount()) } // (b) A failed probe blocks the drop not_enrolled without counting a claim -// failure, and it is not probed again. +// failure, survives the follow-up reconcile, and is not probed again. func TestLoop_StallClaimProbe_FailedBlocksNotEnrolled(t *testing.T) { f, st, _, cfg := probeSetup(t) f.claimRes = platform.ClaimResult{Outcome: platform.ClaimFailed} @@ -707,19 +777,29 @@ func TestLoop_StallClaimProbe_FailedBlocksNotEnrolled(t *testing.T) { require.Eventually(t, func() bool { return l.Snapshot().State == "idle" }, 2*time.Second, 5*time.Millisecond) time.Sleep(50 * time.Millisecond) r := st.get("d1") + assert.Equal(t, dropstate.Blocked, r.Status, "block survives the follow-up reconcile") + assert.Equal(t, dropstate.NotEnrolled, r.Reason) assert.Equal(t, 0, r.FailCount, "a probe is not a claim failure") assert.Equal(t, dropstate.FromPlatform, r.Source) assert.Equal(t, 1, f.claimCount(), "exactly one probe claim") } +// waitBothStalled waits until ch1 and ch2 have both been watched and the +// loop is idle, i.e. the drop has stalled twice. +func waitBothStalled(t *testing.T, f *fakeBackend, l *Loop) { + t.Helper() + require.Eventually(t, func() bool { return len(f.watchedChannels()) >= 2 }, 3*time.Second, 5*time.Millisecond) + require.Eventually(t, func() bool { return l.Snapshot().State == "idle" }, 3*time.Second, 5*time.Millisecond) + time.Sleep(50 * time.Millisecond) +} + // (c) A drop present in the inventory (really tracked, just frozen) is a -// plain stall: no probe claim. +// plain stall: no probe claim, even on the second stall. func TestLoop_StallClaimProbe_SkipsDropInInventory(t *testing.T) { f, st, _, cfg := probeSetup(t) f.inventory = []platform.Progress{{BenefitID: "d1", MinutesWatched: 0}} l, _ := run(t, cfg) - require.Eventually(t, func() bool { return l.Snapshot().Channel == "ch2" }, 3*time.Second, 5*time.Millisecond) - time.Sleep(50 * time.Millisecond) + waitBothStalled(t, f, l) assert.Equal(t, 0, f.claimCount()) assert.Equal(t, dropstate.Eligible, st.get("d1").Status) } @@ -729,8 +809,7 @@ func TestLoop_StallClaimProbe_DisabledNoClaim(t *testing.T) { f, st, _, cfg := probeSetup(t) cfg.StallClaimProbe = false l, _ := run(t, cfg) - require.Eventually(t, func() bool { return l.Snapshot().Channel == "ch2" }, 3*time.Second, 5*time.Millisecond) - time.Sleep(50 * time.Millisecond) + waitBothStalled(t, f, l) assert.Equal(t, 0, f.claimCount()) assert.Equal(t, dropstate.Eligible, st.get("d1").Status) } @@ -741,8 +820,7 @@ func TestLoop_StallClaimProbe_SkipsAccruing(t *testing.T) { f, st, _, cfg := probeSetup(t) f.progress = []platform.DropProgress{{DropID: "d1", CampaignID: "c1", Minutes: 10, Required: 60, Known: true}} l, _ := run(t, cfg) - require.Eventually(t, func() bool { return l.Snapshot().Channel == "ch2" }, 3*time.Second, 5*time.Millisecond) - time.Sleep(50 * time.Millisecond) + waitBothStalled(t, f, l) assert.Equal(t, 0, f.claimCount()) assert.Equal(t, dropstate.Accruing, st.get("d1").Status) } diff --git a/internal/pipeline/session/session.go b/internal/pipeline/session/session.go index df131f5..cf02de9 100644 --- a/internal/pipeline/session/session.go +++ b/internal/pipeline/session/session.go @@ -42,7 +42,7 @@ type Config struct { StallPolls int // Gen is stamped onto every Event this run produces. See Event.Gen. Gen int - // AccountID tags the heartbeat log line the dashboard counts. + // AccountID tags the heartbeat log line shown in the dashboard events feed. AccountID string } @@ -82,8 +82,8 @@ func Run(ctx context.Context, cfg Config, out chan<- Event) { } return false } - // Same line as v1's watcher: the dashboard HEARTBEATS/HR card - // counts kind=heartbeat log lines. + // Same line as v1's watcher, so v2 heartbeats appear in the + // dashboard events feed like v1's. slog.Info("watcher heartbeat sent", "kind", "heartbeat", "account", cfg.AccountID, "channel", ch) prog, err := cfg.Backend.InventoryProgress(ctx, cfg.Session) if err != nil { diff --git a/internal/platform/twitch/pipeline.go b/internal/platform/twitch/pipeline.go index 139d04d..c79fe0a 100644 --- a/internal/platform/twitch/pipeline.go +++ b/internal/platform/twitch/pipeline.go @@ -127,9 +127,15 @@ func (b *Backend) tvDropProgress(ctx context.Context, s platform.Session, camps } // classifyClaimStatus maps Twitch's claimDropRewards.status to an outcome. +// An empty status is a failure: v2 sends blind stall claim-probes, and +// reading "" as OK would permanently mark an unenrolled drop claimed. A +// real claim that returned "" is corrected by the post-claim reconcile. +// (v1's claim() in claim.go keeps its own mapping.) func classifyClaimStatus(status string) platform.ClaimResult { switch status { - case "ELIGIBLE_FOR_ALL", "": + case "": + return platform.ClaimResult{Outcome: platform.ClaimFailed, Detail: "empty claim status"} + case "ELIGIBLE_FOR_ALL": return platform.ClaimResult{Outcome: platform.ClaimOK} case "DROP_INSTANCE_ALREADY_CLAIMED": return platform.ClaimResult{Outcome: platform.ClaimAlready} diff --git a/internal/platform/twitch/pipeline_test.go b/internal/platform/twitch/pipeline_test.go index e04a283..be9edda 100644 --- a/internal/platform/twitch/pipeline_test.go +++ b/internal/platform/twitch/pipeline_test.go @@ -143,7 +143,11 @@ func TestDropProgress_DetailsErrorFailsWhole(t *testing.T) { func TestClassifyClaimStatus(t *testing.T) { assert.Equal(t, platform.ClaimOK, classifyClaimStatus("ELIGIBLE_FOR_ALL").Outcome) - assert.Equal(t, platform.ClaimOK, classifyClaimStatus("").Outcome) + // An empty status proves nothing; a blind stall claim-probe must not + // read it as a claim. + empty := classifyClaimStatus("") + assert.Equal(t, platform.ClaimFailed, empty.Outcome) + assert.Equal(t, "empty claim status", empty.Detail) assert.Equal(t, platform.ClaimAlready, classifyClaimStatus("DROP_INSTANCE_ALREADY_CLAIMED").Outcome) r := classifyClaimStatus("DROP_INSTANCE_ALREADY_EXPIRED") assert.Equal(t, platform.ClaimFailed, r.Outcome) From a8a944c38fe2f1c5662c66712e111dec197fac38 Mon Sep 17 00:00:00 2001 From: Alejandro Date: Tue, 29 Sep 2026 18:53:56 -0400 Subject: [PATCH 31/56] fix(pipeline): bridge claim history after every reconcile, not just load v1->v2 account switches leave drop_state with no row yet for a drop v1 already claimed. load()'s bridge only touches rows that already exist, so the first reconcile creates the row from the platform's silent answer and nothing ever corrects it: Twitch gives no definite signal for a drop claimed outside the app whose campaign left the Inventory, so it stayed eligible and got watched forever (22 Rust drops on prod). Extract bridgeClaimHistory() and call it from both load() and the end of reconcile(), reusing dropstate.MarkCollected + commit() so a later definite platform answer still overwrites a user mark, and a row already Claimed is a no-op. Co-Authored-By: Claude Sonnet 5 --- docs/CHANGELOG.md | 1 + internal/pipeline/loop/loop.go | 31 ++++++++++++++++++++--------- internal/pipeline/loop/loop_test.go | 31 +++++++++++++++++++++++++++++ 3 files changed, 54 insertions(+), 9 deletions(-) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 5c3249c..e35e85b 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -35,6 +35,7 @@ All notable changes to GrubDrops. - A Twitch drop claimed outside GrubDrops whose campaign left the inventory is detected with one claim check when watching it stalls, instead of being watched again and again. - v2 heartbeats appear in the dashboard events feed, like v1's. - The stall claim check only runs on a drop's second stall, and an empty Twitch claim status counts as a failure, so a blind check can never mark an unenrolled drop claimed. +- Drops already claimed under v1 no longer look unclaimed to v2 when an account switches over; v2 reads the claim history after every sync, not only at startup. ## [1.4.2] — 2026-09-28 diff --git a/internal/pipeline/loop/loop.go b/internal/pipeline/loop/loop.go index 6494e56..f18a870 100644 --- a/internal/pipeline/loop/loop.go +++ b/internal/pipeline/loop/loop.go @@ -288,25 +288,37 @@ func (l *Loop) load(ctx context.Context) error { for _, r := range rows { l.rows[r.DropID] = r } + l.bridgeClaimHistory(ctx) + return nil +} + +// bridgeClaimHistory marks any row the claim history already lists as +// claimed but the local row hasn't caught up to. It runs at load (for claims +// a v1-era mark-collected click, or v1 itself, wrote before v2 started) and +// again after every reconcile (for rows reconcile itself just created: an +// account's first sync under v2 has no drop_state rows yet, so a claim v1 +// recorded for one of them would otherwise never be bridged — see +// bridge-brief 2026-09-27). A later definite platform answer still overwrites +// a user mark (dropstate.Apply, spec 4.2); MarkCollected already leaves a +// platform-confirmed claim untouched, so re-running this on an +// already-claimed row is a no-op. +func (l *Loop) bridgeClaimHistory(ctx context.Context) { if l.cfg.History == nil { - return nil + return } ids, err := l.cfg.History.ClaimedBenefitIDs(ctx, l.cfg.AccountID) if err != nil { slog.Warn("pipeline: read claim history failed", "kind", "error", "account", l.cfg.AccountID, "err", err) - return nil + return } now := l.cfg.Now() for id := range ids { - if r, ok := l.rows[id]; ok && r.Status != dropstate.Claimed { - next := dropstate.MarkCollected(r, now) - l.rows[id] = next - if err := l.cfg.Store.Upsert(ctx, next); err != nil { - slog.Warn("pipeline: persist bridged mark failed", "kind", "error", "account", l.cfg.AccountID, "drop", id, "err", err) - } + r, ok := l.rows[id] + if !ok || r.Status == dropstate.Claimed { + continue } + l.commit(ctx, dropstate.MarkCollected(r, now)) } - return nil } func (l *Loop) reconcile(ctx context.Context) { @@ -334,6 +346,7 @@ func (l *Loop) reconcile(ctx context.Context) { for _, r := range res.Rows { l.commit(ctx, r) } + l.bridgeClaimHistory(ctx) } // onIntegrityBlocked mirrors v1: stop watching, surface auth_required so the diff --git a/internal/pipeline/loop/loop_test.go b/internal/pipeline/loop/loop_test.go index 11fab85..8983481 100644 --- a/internal/pipeline/loop/loop_test.go +++ b/internal/pipeline/loop/loop_test.go @@ -278,6 +278,37 @@ func TestLoop_BridgesManualMarksOnLoad(t *testing.T) { assert.Equal(t, dropstate.FromUser, st.get("d1").Source) } +// TestLoop_BridgesManualMarksAfterReconcile covers the gap TestLoop_BridgesManualMarksOnLoad +// doesn't: an account switching from v1 to v2 has claims rows in history +// before drop_state ever has a row for that drop. load() finds nothing to +// bridge (no row yet); the first reconcile creates the row from the +// platform's silent (Known=false) answer, so the bridge must also run after +// reconcile, on the row it just created, or the drop is stuck looking +// unclaimed forever. Revert-proof: remove the post-reconcile bridge call and +// this fails (d1 stays Eligible/FromPlatform, and the planner mines it). +func TestLoop_BridgesManualMarksAfterReconcile(t *testing.T) { + f, st, h, cfg := setup(t) + f.progress = nil // d1 absent from the platform's DropProgress answer (Known=false) + h.claimed = map[string]bool{"d1": true} + l, err := New(cfg) + require.NoError(t, err) + ctx := context.Background() + require.NoError(t, l.load(ctx)) + require.True(t, st.get("d1").IsZero(), "no row exists yet; nothing for load() to bridge") + + l.reconcile(ctx) + + assert.Equal(t, dropstate.Claimed, st.get("d1").Status, "claim history must be bridged onto the row reconcile just created") + assert.Equal(t, dropstate.FromUser, st.get("d1").Source) + + l.refreshLive(ctx) + l.replan(ctx) + assert.Equal(t, "idle", l.Snapshot().State, "an already-claimed drop must not be planned for mining") + f.mu.Lock() + assert.Empty(t, f.watching, "no session should start for a drop claimed under v1") + f.mu.Unlock() +} + func TestNew_RejectsBackendWithoutCapabilities(t *testing.T) { type bare struct{ platform.Backend } _, err := New(Config{AccountID: "a", Backend: bare{}}) From 9df122bbd468a3a1956166e53554ffe7ad0c3e31 Mon Sep 17 00:00:00 2001 From: Alejandro Date: Tue, 29 Sep 2026 19:01:00 -0400 Subject: [PATCH 32/56] fix(pipeline): bridge claim history only for rows new this sync Co-Authored-By: Claude Opus 5.5 (1M context) --- internal/pipeline/loop/loop.go | 42 ++++++++++++++++++++--------- internal/pipeline/loop/loop_test.go | 35 ++++++++++++++++++++++++ 2 files changed, 64 insertions(+), 13 deletions(-) diff --git a/internal/pipeline/loop/loop.go b/internal/pipeline/loop/loop.go index f18a870..3b57f4c 100644 --- a/internal/pipeline/loop/loop.go +++ b/internal/pipeline/loop/loop.go @@ -288,21 +288,27 @@ func (l *Loop) load(ctx context.Context) error { for _, r := range rows { l.rows[r.DropID] = r } - l.bridgeClaimHistory(ctx) + l.bridgeClaimHistory(ctx, nil) return nil } -// bridgeClaimHistory marks any row the claim history already lists as -// claimed but the local row hasn't caught up to. It runs at load (for claims -// a v1-era mark-collected click, or v1 itself, wrote before v2 started) and -// again after every reconcile (for rows reconcile itself just created: an -// account's first sync under v2 has no drop_state rows yet, so a claim v1 -// recorded for one of them would otherwise never be bridged — see -// bridge-brief 2026-09-27). A later definite platform answer still overwrites -// a user mark (dropstate.Apply, spec 4.2); MarkCollected already leaves a -// platform-confirmed claim untouched, so re-running this on an -// already-claimed row is a no-op. -func (l *Loop) bridgeClaimHistory(ctx context.Context) { +// bridgeClaimHistory marks rows the claim history already lists as claimed +// but the local row hasn't caught up to. only, when non-nil, restricts which +// history ids are eligible: reconcile passes a filter that keeps only ids +// whose row didn't exist before that reconcile (an account's first sync +// under v2 has no drop_state rows yet, so a claim v1 recorded for one of +// them would otherwise never be bridged — see bridge-brief 2026-09-27). +// +// A row that already existed is deliberately left alone here: the next +// definite platform answer must overwrite a user mark (dropstate.Apply, spec +// 4.2), and re-asserting the bridge on every reconcile would fight that rule +// forever — Apply demotes Claimed/FromUser to Accruing/FromPlatform on a +// definite answer, then this bridge would immediately re-claim it from the +// same (still-present) history id, in the same reconcile and every one +// after. load() bridges every existing row unconditionally (only=nil): a +// v1-era mark-collected click, or v1 itself, reloads the scheduler on write, +// so load() runs again right after such a claim lands. +func (l *Loop) bridgeClaimHistory(ctx context.Context, only func(id string) bool) { if l.cfg.History == nil { return } @@ -313,6 +319,9 @@ func (l *Loop) bridgeClaimHistory(ctx context.Context) { } now := l.cfg.Now() for id := range ids { + if only != nil && !only(id) { + continue + } r, ok := l.rows[id] if !ok || r.Status == dropstate.Claimed { continue @@ -343,10 +352,17 @@ func (l *Loop) reconcile(ctx context.Context) { l.mu.Lock() l.discovery, l.discoverAt = res.Campaigns, l.cfg.Now() l.mu.Unlock() + existed := make(map[string]bool, len(l.rows)) + for id := range l.rows { + existed[id] = true + } for _, r := range res.Rows { l.commit(ctx, r) } - l.bridgeClaimHistory(ctx) + // Only bridge rows this reconcile just created (not in existed): an + // already-existing row is left to dropstate.Apply's own rules so a + // definite platform answer can overwrite a stale user mark and stick. + l.bridgeClaimHistory(ctx, func(id string) bool { return !existed[id] }) } // onIntegrityBlocked mirrors v1: stop watching, surface auth_required so the diff --git a/internal/pipeline/loop/loop_test.go b/internal/pipeline/loop/loop_test.go index 8983481..248c314 100644 --- a/internal/pipeline/loop/loop_test.go +++ b/internal/pipeline/loop/loop_test.go @@ -309,6 +309,41 @@ func TestLoop_BridgesManualMarksAfterReconcile(t *testing.T) { f.mu.Unlock() } +// TestLoop_PostReconcileBridgeDoesNotRefightDefiniteAnswer covers review +// finding round 1 (Critical): the post-reconcile bridge must only apply to +// rows that didn't exist before that reconcile. d1 is bridged to +// Claimed/FromUser at load. A later sync then gives a *definite* platform +// answer (Known=true, Claimed=false) contradicting that user mark; per spec +// 4.2 (dropstate.Apply) that answer must win and stick — a second reconcile +// must not flip it back to Claimed just because history still lists it. +// Revert-proof: drop the "only rows new this sync" filter (bridge every +// history id unconditionally on every reconcile) and this fails, because the +// post-reconcile bridge re-claims d1 in the same reconcile that just +// demoted it, and again on the following one. +func TestLoop_PostReconcileBridgeDoesNotRefightDefiniteAnswer(t *testing.T) { + f, st, h, cfg := setup(t) + st.rows["d1"] = dropstate.Row{AccountID: "acc", DropID: "d1", CampaignID: "c1", Platform: "twitch", Status: dropstate.Accruing, Minutes: 10, Required: 60, Source: dropstate.FromPlatform} + h.claimed = map[string]bool{"d1": true} + l, err := New(cfg) + require.NoError(t, err) + ctx := context.Background() + require.NoError(t, l.load(ctx)) + require.Equal(t, dropstate.Claimed, st.get("d1").Status, "bridged at load") + require.Equal(t, dropstate.FromUser, st.get("d1").Source, "bridged at load") + + f.mu.Lock() + f.progress = []platform.DropProgress{{DropID: "d1", CampaignID: "c1", Minutes: 5, Required: 60, Known: true, Claimed: false}} + f.mu.Unlock() + + l.reconcile(ctx) + assert.Equal(t, dropstate.Accruing, st.get("d1").Status, "a definite platform answer must overwrite the user mark") + assert.Equal(t, dropstate.FromPlatform, st.get("d1").Source) + + l.reconcile(ctx) + assert.Equal(t, dropstate.Accruing, st.get("d1").Status, "post-reconcile bridge must not re-assert history on a row that already existed") + assert.Equal(t, dropstate.FromPlatform, st.get("d1").Source) +} + func TestNew_RejectsBackendWithoutCapabilities(t *testing.T) { type bare struct{ platform.Backend } _, err := New(Config{AccountID: "a", Backend: bare{}}) From 5fdf3422d238ffe3838ae939a2a27ffe885d5e92 Mon Sep 17 00:00:00 2001 From: Alejandro Date: Tue, 29 Sep 2026 20:37:15 -0400 Subject: [PATCH 33/56] fix(twitch): skip TV AvailableDrops campaigns for other or no game AvailableDrops returns streamers' own game-less channel campaigns and other games' campaigns; listByChannels persisted them all (753 'no game' rows flooding /drops). Keep only the game being walked; Inventory campaigns are unaffected. Co-Authored-By: Claude Opus 5.5 (1M context) --- internal/platform/twitch/chandisc.go | 6 ++++ internal/platform/twitch/chandisc_test.go | 37 +++++++++++++++++++++++ 2 files changed, 43 insertions(+) diff --git a/internal/platform/twitch/chandisc.go b/internal/platform/twitch/chandisc.go index efc505e..c431f3a 100644 --- a/internal/platform/twitch/chandisc.go +++ b/internal/platform/twitch/chandisc.go @@ -159,6 +159,12 @@ func (d *discovery) listByChannels(ctx context.Context, sess platform.Session, c continue } for _, vc := range resp.Channel.ViewerDropCampaigns { + // AvailableDrops also returns the streamer's own channel + // campaigns (no game) and campaigns for other games. Keep + // only the game being walked; they flooded /drops otherwise. + if vc.Game.Name == "" || gameslug.Slug(vc.Game.Name) != slug { + continue + } start, end := parseISO(vc.StartAt), parseISO(vc.EndAt) add(platform.Campaign{ ID: vc.ID, Platform: "twitch", Game: vc.Game.Name, Name: vc.Name, diff --git a/internal/platform/twitch/chandisc_test.go b/internal/platform/twitch/chandisc_test.go index c7a7048..e5f5079 100644 --- a/internal/platform/twitch/chandisc_test.go +++ b/internal/platform/twitch/chandisc_test.go @@ -470,3 +470,40 @@ func TestListByChannels_InventoryFailureKeepsDirectoryResults(t *testing.T) { assert.Equal(t, "campA", camps[0].ID) assert.Equal(t, 1, b.AllowedChannelCount("campA")) } + +// TestListByChannels_SkipsAvailableDropsForOtherGames: AvailableDrops also +// returns streamers' own channel campaigns, which carry no game (or a +// different game than the directory being walked). Those flooded /drops +// with hundreds of "no game" rows. Only campaigns for the queried game are +// kept; Inventory campaigns are unaffected. +func TestListByChannels_SkipsAvailableDropsForOtherGames(t *testing.T) { + const availMixed = `{"data":{"channel":{"viewerDropCampaigns":[ + {"id":"campNoGame","name":"Streamer Channel Drop","game":null,"endAt":"2030-01-01T00:00:00Z", + "timeBasedDrops":[{"id":"dN","name":"N","requiredMinutesWatched":30,"requiredSubs":0,"benefitEdges":[{"benefit":{"id":"bN","name":"N","imageAssetURL":""}}]}]}, + {"id":"campOther","name":"Other Game Drop","game":{"id":"1","name":"Apex Legends"},"endAt":"2030-01-01T00:00:00Z", + "timeBasedDrops":[{"id":"dO","name":"O","requiredMinutesWatched":30,"requiredSubs":0,"benefitEdges":[{"benefit":{"id":"bO","name":"O","imageAssetURL":""}}]}]}, + {"id":"campA","name":"Rust Isles General","game":{"id":"263490","name":"Rust"},"endAt":"2030-01-01T00:00:00Z", + "timeBasedDrops":[{"id":"dWatch","name":"Box","requiredMinutesWatched":60,"requiredSubs":0,"benefitEdges":[{"benefit":{"id":"bW","name":"Box","imageAssetURL":""}}]}]}]}}}` + srv := fakeGQL(t, map[string]func(map[string]any) string{ + "DirectoryPage_Game": func(v map[string]any) string { return dirRust }, + "DropsHighlightService_AvailableDrops": func(v map[string]any) string { + return availMixed + }, + "Inventory": func(map[string]any) string { return inventoryTV }, + }) + defer srv.Close() + + b := newForTest(srv.URL) + sess := platform.Session{AccessToken: "tv", ClientID: ClientTV, Games: []string{"Rust"}} + camps, err := b.ListActiveCampaigns(context.Background(), sess) + require.NoError(t, err) + ids := map[string]bool{} + for _, c := range camps { + ids[c.ID] = true + } + assert.True(t, ids["campA"], "matching-game AvailableDrops campaign kept") + assert.True(t, ids["campB"], "Inventory campaign unaffected") + assert.False(t, ids["campNoGame"], "game-less channel campaign skipped") + assert.False(t, ids["campOther"], "campaign for a different game skipped") + assert.Len(t, camps, 2) +} From 9624d5d0540dd4b2da8e2e80c414121508821cac Mon Sep 17 00:00:00 2001 From: Alejandro Date: Tue, 29 Sep 2026 20:41:39 -0400 Subject: [PATCH 34/56] feat(twitch): shared campaign catalog for TV-login discovery Twitch hides ViewerDropsDashboard/DropCampaignDetails from TV tokens and AvailableDrops returns one campaign per channel, so a global badge campaign masked every other campaign of a game (R6 S2 watch drops hidden from TV accounts). Non-TV backends now publish their campaign list + allow-lists to one process-wide twitch.Catalog (link flags made account-neutral); TV backends merge fresh (45m TTL), active, whitelisted campaigns they didn't find themselves. Inventory stays authoritative. Watch/claim requests keep each session's own client. Co-Authored-By: Claude Opus 5.5 (1M context) --- cmd/miner/main.go | 6 + internal/platform/twitch/backend.go | 32 ++++ internal/platform/twitch/catalog.go | 185 ++++++++++++++++++ internal/platform/twitch/catalog_test.go | 232 +++++++++++++++++++++++ 4 files changed, 455 insertions(+) create mode 100644 internal/platform/twitch/catalog.go create mode 100644 internal/platform/twitch/catalog_test.go diff --git a/cmd/miner/main.go b/cmd/miner/main.go index 0659ee1..f6fbb0a 100644 --- a/cmd/miner/main.go +++ b/cmd/miner/main.go @@ -253,6 +253,11 @@ func run() error { } else { twitchBackend = twitch.New() } + // One shared Twitch campaign catalog for the whole process: backends on + // the Android/legacy login publish the full campaign list, TV-login + // backends merge the campaigns Twitch hides from the TV client. + twitchCatalog := twitch.NewCatalog() + twitchBackend.SetCatalog(twitchCatalog) if twitchBrowserEnabled && browserClient != nil { // proxyTransport is nil when no proxy is configured; the browser // backend then dials the Spade beacon direct, same as before. @@ -378,6 +383,7 @@ func run() error { } else { bk = twitch.New() } + bk.SetCatalog(twitchCatalog) twitchBackends[a.ID] = bk return bk, true } diff --git a/internal/platform/twitch/backend.go b/internal/platform/twitch/backend.go index b73f3bd..f953dbb 100644 --- a/internal/platform/twitch/backend.go +++ b/internal/platform/twitch/backend.go @@ -119,6 +119,10 @@ type Backend struct { tvDetails map[string][]platform.DropBenefit tvDetailsAt time.Time + // catalog is the process-wide shared campaign list: non-TV sessions + // publish to it, TV sessions merge from it. Nil = no sharing. + catalog *Catalog + // PubSub WebSocket — one per backend (per platform-account). Lazy // init on first ListActiveCampaigns once we have the user_id + // auth token. Real-time progress / claim / stream-down events feed @@ -255,6 +259,10 @@ func (b *Backend) ListActiveCampaigns(ctx context.Context, s platform.Session) ( if err != nil { return nil, err } + b.mu.Lock() + cat := b.catalog + b.mu.Unlock() + camps = mergeCatalog(cat, s, camps, allowed) details := make(map[string][]platform.DropBenefit, len(camps)) for _, c := range camps { details[c.ID] = c.Benefits @@ -280,13 +288,37 @@ func (b *Backend) ListActiveCampaigns(ctx context.Context, s platform.Session) ( for cid, logins := range allowed { b.allowedLoginsByCampaign[cid] = logins } + cat := b.catalog b.mu.Unlock() + // Share the full list with TV-client backends (which can't see the + // dashboard). Publish copies account-neutral link state. + if cat != nil { + cat.Publish(b.catalogSource(s), camps, allowed) + } // Best-effort PubSub bootstrap. Once-only — subsequent calls noop. // Failures are non-fatal: the watcher falls back to polling. b.ensurePubSub(s) return camps, nil } +// SetCatalog attaches the process-wide shared campaign catalog. Non-TV +// sessions publish their discovered campaigns to it; TV sessions merge +// campaigns hidden from the TV client from it. Nil disables sharing. +func (b *Backend) SetCatalog(c *Catalog) { + b.mu.Lock() + b.catalog = c + b.mu.Unlock() +} + +// catalogSource keys this backend's catalog entry: the account when the +// scheduler set it, else the backend instance (one per account in prod). +func (b *Backend) catalogSource(s platform.Session) string { + if s.AccountID != "" { + return s.AccountID + } + return fmt.Sprintf("%p", b) +} + // SetPubSubHandlers wires real-time callbacks. Must be called before // the first ListActiveCampaigns (which lazily connects). Safe to leave // nil — PubSub still connects + logs events for diagnostic purposes. diff --git a/internal/platform/twitch/catalog.go b/internal/platform/twitch/catalog.go new file mode 100644 index 0000000..55e61df --- /dev/null +++ b/internal/platform/twitch/catalog.go @@ -0,0 +1,185 @@ +package twitch + +import ( + "log/slog" + "sort" + "sync" + "time" + + "github.com/aalejandrofer/grubdrops/internal/gameslug" + "github.com/aalejandrofer/grubdrops/internal/platform" +) + +// catalogTTL bounds how long a published campaign list is trusted. The +// non-TV discovery pass republishes every tick, so an older entry means +// its publisher stopped (account removed, session dead). +const catalogTTL = 45 * time.Minute + +// Catalog is a process-wide, read-only store of the Twitch campaign list +// found by non-TV (Android/legacy) sessions through ViewerDropsDashboard + +// DropCampaignDetails. Twitch hides both from TV-client tokens, and the +// TV channel-first path (AvailableDrops) returns one campaign per channel, +// so a global badge campaign can mask every other campaign of a game. TV +// backends merge this catalog into their own discovery. The data is +// campaign-level only: watch/claim requests keep each session's own +// client profile. +type Catalog struct { + mu sync.Mutex + sources map[string]catalogEntry // publisher key -> its latest list +} + +type catalogEntry struct { + campaigns []platform.Campaign + allowed map[string][]string + at time.Time +} + +// NewCatalog returns an empty catalog. +func NewCatalog() *Catalog { + return &Catalog{sources: map[string]catalogEntry{}} +} + +// Publish replaces source's campaign list. The stored copy is made +// account-neutral: link flags become optimistic (AccountLinked=true, +// AccountLinkChecked=false) so one account's link state never leaks to +// another; the campaign-level AccountLinkURL is kept. +func (c *Catalog) Publish(source string, camps []platform.Campaign, allowed map[string][]string) { + if c == nil { + return + } + cp := make([]platform.Campaign, 0, len(camps)) + for _, camp := range camps { + camp.AccountLinked = true + camp.AccountLinkChecked = false + camp.Benefits = append([]platform.DropBenefit(nil), camp.Benefits...) + camp.AllowedChannels = append([]string(nil), camp.AllowedChannels...) + cp = append(cp, camp) + } + al := make(map[string][]string, len(allowed)) + for id, logins := range allowed { + al[id] = append([]string(nil), logins...) + } + c.mu.Lock() + c.sources[source] = catalogEntry{campaigns: cp, allowed: al, at: time.Now()} + c.mu.Unlock() +} + +// snapshot unions every fresh source. For a campaign more than one source +// carries, the entry with the most benefits wins (a publisher whose +// whitelist lacks the game emits a benefit-less shell), newest breaking +// ties; its allow-list rides along. age is the freshest source's age; ok +// is false when no source is fresh. +func (c *Catalog) snapshot() (camps []platform.Campaign, allowed map[string][]string, age time.Duration, ok bool) { + c.mu.Lock() + defer c.mu.Unlock() + keys := make([]string, 0, len(c.sources)) + for k := range c.sources { + keys = append(keys, k) + } + sort.Strings(keys) // deterministic order across map iteration + type pick struct { + camp platform.Campaign + allowed []string + hasAl bool + at time.Time + } + best := map[string]pick{} + var order []string + var newest time.Time + for _, k := range keys { + e := c.sources[k] + if time.Since(e.at) >= catalogTTL { + continue + } + ok = true + if e.at.After(newest) { + newest = e.at + } + for _, camp := range e.campaigns { + al, hasAl := e.allowed[camp.ID] + cur, seen := best[camp.ID] + if !seen { + order = append(order, camp.ID) + } else if len(camp.Benefits) < len(cur.camp.Benefits) || + (len(camp.Benefits) == len(cur.camp.Benefits) && !e.at.After(cur.at)) { + continue + } + best[camp.ID] = pick{camp: camp, allowed: al, hasAl: hasAl, at: e.at} + } + } + if !ok { + return nil, nil, 0, false + } + allowed = map[string][]string{} + for _, id := range order { + p := best[id] + camp := p.camp + camp.Benefits = append([]platform.DropBenefit(nil), camp.Benefits...) + camp.AllowedChannels = append([]string(nil), camp.AllowedChannels...) + camps = append(camps, camp) + if p.hasAl { + allowed[id] = append([]string(nil), p.allowed...) + } + } + return camps, allowed, time.Since(newest), true +} + +// sessionWantsGame reports whether game is on the session's whitelist: +// GameFilter when set, else a gameslug match against Session.Games. +func sessionWantsGame(s platform.Session, game string) bool { + if game == "" { + return false + } + if s.GameFilter != nil { + return s.GameFilter(game) + } + slug := gameslug.Slug(game) + for _, g := range s.Games { + if gameslug.Slug(g) == slug { + return true + } + } + return false +} + +// mergeCatalog appends to camps the fresh catalog campaigns the TV pass +// didn't find: not already present by ID (so the TV account's own +// Inventory stays authoritative), active with a future end time, and on +// the session's whitelist. Their catalog allow-lists are added to allowed +// when the TV pass supplied none. +func mergeCatalog(cat *Catalog, s platform.Session, camps []platform.Campaign, allowed map[string][]string) []platform.Campaign { + if cat == nil { + return camps + } + cc, cal, age, ok := cat.snapshot() + if !ok { + return camps + } + have := make(map[string]struct{}, len(camps)) + for _, c := range camps { + have[c.ID] = struct{}{} + } + now := time.Now() + merged := 0 + for _, c := range cc { + if _, dup := have[c.ID]; dup { + continue + } + if c.Status != "active" || c.EndsAt.IsZero() || !c.EndsAt.After(now) { + continue + } + if !sessionWantsGame(s, c.Game) { + continue + } + camps = append(camps, c) + have[c.ID] = struct{}{} + if al, has := cal[c.ID]; has { + if _, tvHas := allowed[c.ID]; !tvHas { + allowed[c.ID] = al + } + } + merged++ + } + slog.Info("tv discovery: merged shared catalog", "merged", merged, "catalog_age", age.Round(time.Second)) + return camps +} diff --git a/internal/platform/twitch/catalog_test.go b/internal/platform/twitch/catalog_test.go new file mode 100644 index 0000000..cb9138a --- /dev/null +++ b/internal/platform/twitch/catalog_test.go @@ -0,0 +1,232 @@ +package twitch + +import ( + "context" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/aalejandrofer/grubdrops/internal/platform" +) + +// dashboardAndroid is the ViewerDropsDashboard an Android-client session +// sees: the watch campaign campR6 (hidden from the TV client behind a +// badge campaign), a non-whitelisted game, an expired campaign, and an +// ACTIVE-labelled campaign whose end time already passed. The Android +// account is NOT linked for campR6 — that flag must never leak to other +// accounts through the catalog. +const dashboardAndroid = `{"data":{"currentUser":{"dropCampaigns":[ + {"id":"campR6","name":"Rust S2 Watch","status":"ACTIVE","startAt":"2020-01-01T00:00:00Z","endAt":"2030-01-01T00:00:00Z", + "accountLinkURL":"https://link/r6","self":{"isAccountConnected":false},"game":{"id":"263490","displayName":"Rust"}}, + {"id":"campApex","name":"Apex Watch","status":"ACTIVE","startAt":"2020-01-01T00:00:00Z","endAt":"2030-01-01T00:00:00Z", + "self":{"isAccountConnected":true},"game":{"id":"1","displayName":"Apex Legends"}}, + {"id":"campOld","name":"Rust S1","status":"EXPIRED","startAt":"2020-01-01T00:00:00Z","endAt":"2020-02-01T00:00:00Z", + "self":{"isAccountConnected":true},"game":{"id":"263490","displayName":"Rust"}}, + {"id":"campPast","name":"Rust Ended","status":"ACTIVE","startAt":"2020-01-01T00:00:00Z","endAt":"2020-02-01T00:00:00Z", + "self":{"isAccountConnected":true},"game":{"id":"263490","displayName":"Rust"}}]}}}` + +const detailsR6 = `{"data":{"user":{"dropCampaign":{"id":"campR6","name":"Rust S2 Watch", + "allow":{"isEnabled":true,"channels":[{"id":"c7","name":"r6chan"}]}, + "timeBasedDrops":[{"id":"dR1","name":"Charm","requiredMinutesWatched":120,"requiredSubs":0, + "benefitEdges":[{"benefit":{"id":"bR1","name":"Charm","imageAssetURL":""}}]}]}}}}` + +const detailsEmpty = `{"data":{"user":{"dropCampaign":{"id":"x","timeBasedDrops":[]}}}}` + +// availBadge mimics the live masking: AvailableDrops returns ONE campaign +// per channel, a global 0-minute chat badge, hiding campR6. +const availBadge = `{"data":{"channel":{"viewerDropCampaigns":[{"id":"campBadge","name":"Chat Badge","game":{"id":"263490","name":"Rust"},"endAt":"2030-01-01T00:00:00Z", + "timeBasedDrops":[{"id":"dB","name":"Badge","requiredMinutesWatched":0,"requiredSubs":0,"benefitEdges":[{"benefit":{"id":"bB","name":"Badge","imageAssetURL":""}}]}]}]}}}` + +const inventoryEmpty = `{"data":{"currentUser":{"inventory":{"dropCampaignsInProgress":[],"gameEventDrops":[]}}}}` + +func androidBackend(t *testing.T) *Backend { + t.Helper() + srv := fakeGQL(t, map[string]func(map[string]any) string{ + "ViewerDropsDashboard": func(map[string]any) string { return dashboardAndroid }, + "DropCampaignDetails": func(v map[string]any) string { + if v["dropID"] == "campR6" { + return detailsR6 + } + return detailsEmpty + }, + }) + t.Cleanup(srv.Close) + return newForTest(srv.URL) +} + +func tvBackend(t *testing.T, inventory string) *Backend { + t.Helper() + srv := fakeGQL(t, map[string]func(map[string]any) string{ + "DirectoryPage_Game": func(map[string]any) string { return dirRust }, + "DropsHighlightService_AvailableDrops": func(map[string]any) string { + return availBadge + }, + "Inventory": func(map[string]any) string { return inventory }, + }) + t.Cleanup(srv.Close) + return newForTest(srv.URL) +} + +func tvSess() platform.Session { + return platform.Session{AccessToken: "tv", ClientID: ClientTV, Games: []string{"Rust"}} +} + +func campsByID(camps []platform.Campaign) map[string]platform.Campaign { + out := map[string]platform.Campaign{} + for _, c := range camps { + out[c.ID] = c + } + return out +} + +// publishFromAndroid runs a non-TV ListActiveCampaigns against cat. +func publishFromAndroid(t *testing.T, cat *Catalog) { + t.Helper() + ab := androidBackend(t) + ab.SetCatalog(cat) + _, err := ab.ListActiveCampaigns(context.Background(), platform.Session{AccessToken: "android"}) + require.NoError(t, err) +} + +// (a) A campaign the TV client can't see (masked by a badge campaign) is +// found through the catalog an Android account published: optimistic, +// non-account-specific link state, its allow-list feeds the TV backend's +// eligible-channel cache, and its benefits serve TV CampaignDetails. +func TestCatalog_TVMergesAndroidCampaigns(t *testing.T) { + cat := NewCatalog() + publishFromAndroid(t, cat) + + tb := tvBackend(t, inventoryEmpty) + tb.SetCatalog(cat) + camps, err := tb.ListActiveCampaigns(context.Background(), tvSess()) + require.NoError(t, err) + byID := campsByID(camps) + require.Contains(t, byID, "campBadge", "the TV path's own campaign is kept") + require.Contains(t, byID, "campR6", "catalog campaign hidden from TV must be merged") + + r6 := byID["campR6"] + assert.True(t, r6.AccountLinked, "catalog copy must not carry the Android account's link flags") + assert.False(t, r6.AccountLinkChecked, "link state is unknown for another account: optimistic") + assert.Equal(t, "https://link/r6", r6.AccountLinkURL, "campaign-level link URL kept") + assert.Equal(t, "active", r6.Status) + require.Len(t, r6.Benefits, 1) + assert.Equal(t, "dR1", r6.Benefits[0].ID) + assert.Equal(t, 120, r6.Benefits[0].RequiredMinutes) + + assert.Equal(t, 1, tb.AllowedChannelCount("campR6"), "catalog allow-list feeds ListEligibleChannels") + tb.mu.Lock() + assert.Equal(t, []string{"r6chan"}, tb.allowedLoginsByCampaign["campR6"]) + tb.mu.Unlock() + + benefits, err := tb.CampaignDetails(context.Background(), tvSess(), "campR6") + require.NoError(t, err) + require.Len(t, benefits, 1, "merged campaigns serve /drops CampaignDetails") +} + +// (b) The TV account's own Inventory is authoritative for link state and +// status of a campaign the catalog also carries. +func TestCatalog_InventoryWinsOverCatalog(t *testing.T) { + const invR6 = `{"data":{"currentUser":{"inventory":{"dropCampaignsInProgress":[ + {"id":"campR6","name":"Rust S2 Watch","game":{"id":"263490","name":"Rust"},"endAt":"2030-01-01T00:00:00Z", + "status":"ACTIVE","self":{"isAccountConnected":false},"accountLinkURL":"https://link/r6", + "timeBasedDrops":[{"id":"dR1","name":"Charm","requiredMinutesWatched":120,"requiredSubs":0, + "benefitEdges":[{"benefit":{"id":"bR1","name":"Charm","imageAssetURL":""}}], + "self":{"currentMinutesWatched":5,"isClaimed":false,"dropInstanceID":""}}]}], + "gameEventDrops":[]}}}}` + cat := NewCatalog() + publishFromAndroid(t, cat) + + tb := tvBackend(t, invR6) + tb.SetCatalog(cat) + camps, err := tb.ListActiveCampaigns(context.Background(), tvSess()) + require.NoError(t, err) + n := 0 + for _, c := range camps { + if c.ID == "campR6" { + n++ + } + } + assert.Equal(t, 1, n, "catalog must not duplicate an Inventory campaign") + r6 := campsByID(camps)["campR6"] + assert.False(t, r6.AccountLinked, "Inventory's link state wins over the catalog's optimistic one") + assert.True(t, r6.AccountLinkChecked) + require.Len(t, r6.Benefits, 1) +} + +// (c) Non-whitelisted, expired, and past-end catalog campaigns are not +// merged; a session GameFilter takes precedence over Session.Games. +func TestCatalog_FiltersGameAndWindow(t *testing.T) { + cat := NewCatalog() + publishFromAndroid(t, cat) + + tb := tvBackend(t, inventoryEmpty) + tb.SetCatalog(cat) + camps, err := tb.ListActiveCampaigns(context.Background(), tvSess()) + require.NoError(t, err) + byID := campsByID(camps) + assert.Contains(t, byID, "campR6") + assert.NotContains(t, byID, "campApex", "non-whitelisted game not merged") + assert.NotContains(t, byID, "campOld", "expired campaign not merged") + assert.NotContains(t, byID, "campPast", "campaign past its end time not merged") + + sess := tvSess() + sess.GameFilter = func(string) bool { return false } + tb2 := tvBackend(t, inventoryEmpty) + tb2.SetCatalog(cat) + camps, err = tb2.ListActiveCampaigns(context.Background(), sess) + require.NoError(t, err) + assert.NotContains(t, campsByID(camps), "campR6", "GameFilter, when set, decides membership") +} + +// (d) A catalog older than catalogTTL is ignored. +func TestCatalog_StaleIgnored(t *testing.T) { + cat := NewCatalog() + publishFromAndroid(t, cat) + cat.mu.Lock() + for k, e := range cat.sources { + e.at = e.at.Add(-catalogTTL - time.Minute) + cat.sources[k] = e + } + cat.mu.Unlock() + + tb := tvBackend(t, inventoryEmpty) + tb.SetCatalog(cat) + camps, err := tb.ListActiveCampaigns(context.Background(), tvSess()) + require.NoError(t, err) + assert.NotContains(t, campsByID(camps), "campR6", "stale catalog must be ignored") + assert.Equal(t, 0, tb.AllowedChannelCount("campR6")) +} + +// (e) No catalog set: TV discovery is unchanged, and a non-TV backend +// without a catalog still lists normally. +func TestCatalog_NoneSetUnchanged(t *testing.T) { + ab := androidBackend(t) + _, err := ab.ListActiveCampaigns(context.Background(), platform.Session{AccessToken: "android"}) + require.NoError(t, err) + + tb := tvBackend(t, inventoryEmpty) + camps, err := tb.ListActiveCampaigns(context.Background(), tvSess()) + require.NoError(t, err) + require.Len(t, camps, 1) + assert.Equal(t, "campBadge", camps[0].ID) +} + +// A publisher whose whitelist lacks a game emits that campaign as a +// benefit-less shell; it must not wipe another publisher's full entry. +func TestCatalog_PrefersEntryWithBenefits(t *testing.T) { + cat := NewCatalog() + full := platform.Campaign{ID: "c1", Game: "Rust", Status: "active", EndsAt: time.Now().Add(time.Hour), + Benefits: []platform.DropBenefit{{ID: "d1", CampaignID: "c1", RequiredMinutes: 60}}} + shell := full + shell.Benefits = nil + cat.Publish("acc-a", []platform.Campaign{full}, map[string][]string{"c1": {"x"}}) + cat.Publish("acc-b", []platform.Campaign{shell}, nil) + + camps, allowed, _, ok := cat.snapshot() + require.True(t, ok) + require.Len(t, camps, 1) + assert.Len(t, camps[0].Benefits, 1) + assert.Equal(t, []string{"x"}, allowed["c1"]) +} From 7ac6b6bcc4559698e8d4916edf4833bcb061a85e Mon Sep 17 00:00:00 2001 From: Alejandro Date: Tue, 29 Sep 2026 20:41:39 -0400 Subject: [PATCH 35/56] test(drops): lock ends_at grouping; changelog for TV catalog + noise fix /drops tabs already group by ends_at, not stored status; this test pins that an ended campaign with status=active lands in Past, never Current. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/CHANGELOG.md | 5 +++ internal/api/handlers_drops_collect_test.go | 41 +++++++++++++++++++++ 2 files changed, 46 insertions(+) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index e35e85b..81a24a5 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -9,12 +9,17 @@ All notable changes to GrubDrops. - Pipeline v2 (opt-in, `GRUB_PIPELINE=v2` or kv `pipeline_override:`): drop state now comes from each campaign's platform details, so drops claimed on the website or with a lost claim response show as claimed and mining moves on (#50). - Streamer priority for pipeline v2: per-account priority streamers are watched first, then any live channel in the category (#49). Seeded from the existing null-game channel list. Until the settings editor lands, v2 reads a one-time copy of the null-game channel list taken at upgrade. - `drop_state` and `account_streamer_priority` tables (migration 0016), with a one-time backfill from claim history, manual marks and ghost-skips. +- Accounts on the Twitch TV login share the full campaign list found by any Android-login account in the same GrubDrops, so campaigns hidden from the TV client (for example behind a global badge campaign) are still found. ### Changed - Twitch claim code split into status fetch + classification (behaviour unchanged). - Scheduler reads dashboard snapshots and discoveries through interfaces, so v1 and v2 runners both report state. +### Fixed + +- Streamers' own channel campaigns with no game no longer flood the drops page, and campaigns past their end time show as ended. + ### Fixed (pipeline v2 only) - Claim failures back off 1m, 5m, 30m and stop after 5 attempts instead of looping forever. diff --git a/internal/api/handlers_drops_collect_test.go b/internal/api/handlers_drops_collect_test.go index 5b0aa77..bf1d95d 100644 --- a/internal/api/handlers_drops_collect_test.go +++ b/internal/api/handlers_drops_collect_test.go @@ -276,3 +276,44 @@ func renderCampaignItems_render(t *testing.T, detail campaignDetailRow) string { require.NoError(t, tmpl.ExecuteTemplate(&buf, "drops_campaign_items", detail)) return buf.String() } + +// TestCollectAll_EndedCampaignIsPastWhateverItsStatus: a campaign whose end +// time has passed belongs to Past even when its stored status still says +// "active" (e.g. TV AvailableDrops rows never refreshed to expired). It +// must never show in Current or the Current Discoverable list. +func TestCollectAll_EndedCampaignIsPastWhateverItsStatus(t *testing.T) { + ctx := context.Background() + db, err := store.Open(ctx, filepath.Join(t.TempDir(), "test.db")) + require.NoError(t, err) + t.Cleanup(func() { _ = db.Close() }) + q := gen.New(db) + + now := time.Now().Unix() + for _, c := range []struct{ id, game string }{{"ended-wl", "Rust"}, {"ended-nogame", ""}} { + require.NoError(t, q.UpsertCampaign(ctx, gen.UpsertCampaignParams{ + ID: c.id, Platform: "twitch", Game: c.game, Name: c.id, + StartsAt: now - 7200, EndsAt: now - 60, Status: "active", + RawJson: "{}", DiscoveredAt: now, Kind: "drop", AccountLinked: 1, + })) + } + + d := &dropsDeps{q: q, loc: timeutil.NewZone(time.UTC)} + allow := map[string]struct{}{"rust": {}} + ids := func(rows []dropsRow) map[string]bool { + out := map[string]bool{} + for _, r := range rows { + out[r.CampaignID] = true + } + return out + } + + past, current, _, unlistedCur, err := d.collectAll(ctx, allow, true, now, 200, tabCurrent) + require.NoError(t, err) + require.True(t, ids(past)["ended-wl"], "ended whitelisted campaign shows as past") + require.False(t, ids(current)["ended-wl"], "ended campaign must not show as current despite status=active") + require.False(t, ids(unlistedCur)["ended-nogame"], "ended game-less campaign must not show in Current Discoverable") + + _, _, _, unlistedPast, err := d.collectAll(ctx, allow, true, now, 200, tabPast) + require.NoError(t, err) + require.True(t, ids(unlistedPast)["ended-nogame"], "ended game-less campaign is listed under Past") +} From c004c4ca001ed11a1241f5a5561a55abe718657b Mon Sep 17 00:00:00 2001 From: Alejandro Date: Tue, 29 Sep 2026 20:52:27 -0400 Subject: [PATCH 36/56] fix(pipeline): a campaign link block only lifts in reconcile, not on any Known observation Prod: a TV-login account mined a drop whose campaign reported self.isAccountConnected=false, because dropstate.Apply re-derived Blocked/Unlinked on the next Known observation regardless of source. The session's own Progress events (loop.applyProgress -> Apply with Known=true) lifted the block between reconciles just like a real reconcile did, so the planner mined a drop whose claim could never succeed until the next sync re-blocked it. - dropstate.Apply: Blocked/Unlinked now behaves like Blocked/UserSkip - only a claimed observation moves it; every other Known observation leaves it alone. Lifting it is the reconciler's job. - reconcile.Run: when the campaign is linked and the prev row is Blocked/Unlinked, lift it first with dropstate.Retry (re-derives from minutes/required) before folding in the observation. Still-unlinked rows keep BlockLink after Apply as before. - Updated the superseded I3 test (Apply re-derives Unlinked immediately on any Known observation) to the new rule; the reconcile-level lift test it was standing in for already exists and continues to pass unchanged (TestRun_LinkedAfterUnlinkedUnblocksImmediately). Co-Authored-By: Claude Sonnet 5 --- docs/CHANGELOG.md | 1 + internal/pipeline/dropstate/dropstate.go | 9 +++-- internal/pipeline/dropstate/dropstate_test.go | 32 +++++++++++---- internal/pipeline/loop/loop_test.go | 40 +++++++++++++++++++ internal/pipeline/reconcile/reconcile.go | 10 ++++- internal/pipeline/reconcile/reconcile_test.go | 19 +++++++++ 6 files changed, 100 insertions(+), 11 deletions(-) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 81a24a5..a6504af 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -41,6 +41,7 @@ All notable changes to GrubDrops. - v2 heartbeats appear in the dashboard events feed, like v1's. - The stall claim check only runs on a drop's second stall, and an empty Twitch claim status counts as a failure, so a blind check can never mark an unenrolled drop claimed. - Drops already claimed under v1 no longer look unclaimed to v2 when an account switches over; v2 reads the claim history after every sync, not only at startup. +- v2 no longer watches drops from campaigns that need an unlinked account between syncs. ## [1.4.2] — 2026-09-28 diff --git a/internal/pipeline/dropstate/dropstate.go b/internal/pipeline/dropstate/dropstate.go index 3a3160e..78078f5 100644 --- a/internal/pipeline/dropstate/dropstate.go +++ b/internal/pipeline/dropstate/dropstate.go @@ -27,8 +27,11 @@ const ( NotEnrolled Reason = "not_enrolled" UserSkip Reason = "user_skip" // Unlinked is the campaign-level link block: the campaign needs an - // external account the user has not linked. Unlike the claim-level - // NeedsLink it has no hold; the next definite answer re-derives it. + // external account the user has not linked. Like UserSkip, a non-claimed + // observation never lifts it on its own — only the reconciler may, once + // it re-checks the campaign's link status and confirms it is linked + // (dropstate.Retry). A platform-confirmed claim still wins, same as any + // other block. Unlinked Reason = "unlinked" ) @@ -162,7 +165,7 @@ func Apply(prev Row, obs Observation, now time.Time) Row { r.Source, r.FailCount = FromPlatform, 0 return r.with(Claimed, NoReason, now) } - if prev.Status == Blocked && prev.Reason == UserSkip { + if prev.Status == Blocked && (prev.Reason == UserSkip || prev.Reason == Unlinked) { return r } if prev.Status == Blocked && (prev.Reason == NeedsLink || prev.Reason == ClaimFailed) && now.Before(prev.RetryAfter) { diff --git a/internal/pipeline/dropstate/dropstate_test.go b/internal/pipeline/dropstate/dropstate_test.go index aa5e971..6d9b198 100644 --- a/internal/pipeline/dropstate/dropstate_test.go +++ b/internal/pipeline/dropstate/dropstate_test.go @@ -197,17 +197,35 @@ func TestApply_UnknownBackfilledRowAdoptsRequired(t *testing.T) { } } -// I3: a campaign-level link block is re-derived as soon as the platform gives a -// definite answer; there is no 24h hold once the user links. -func TestApply_UnlinkedReDerivedImmediately(t *testing.T) { +// I3 (superseded, see below): a campaign-level link block used to re-derive +// on the next Known observation regardless of source. That let a mid-watch +// session Progress event (loop.applyProgress, also Known=true) lift the +// block between reconciles even though the campaign was still unlinked, so +// the planner mined a drop whose claim could never succeed. Lifting an +// unlinked block is now the reconciler's job alone (dropstate.Retry once it +// confirms the campaign is linked); see TestApply_UnlinkedNeverLiftedByObservation +// and reconcile.TestRun_LinkedAfterUnlinkedUnblocksImmediately. + +// A campaign-level link block behaves like user_skip: only a claimed +// observation can move it. Any other Known observation (including one from +// a mid-watch session Progress event) must leave it Blocked/Unlinked; only +// the reconciler (via dropstate.Retry, once it has confirmed the campaign +// is linked) may lift it. +func TestApply_UnlinkedNeverLiftedByObservation(t *testing.T) { prev := BlockLink(row(Accruing, NoReason, 10, 60), t0) require.Equal(t, Unlinked, prev.Reason) - require.True(t, t0.Add(time.Minute).Before(prev.RetryAfter)) + got := Apply(prev, Observation{Known: true, Minutes: 20, Required: 60}, t0.Add(time.Minute)) - assert.Equal(t, Accruing, got.Status) - assert.Equal(t, NoReason, got.Reason) + assert.Equal(t, Blocked, got.Status) + assert.Equal(t, Unlinked, got.Reason) + got = Apply(prev, Observation{Known: true, Minutes: 0, Required: 60}, t0.Add(time.Minute)) - assert.Equal(t, Eligible, got.Status) + assert.Equal(t, Blocked, got.Status) + assert.Equal(t, Unlinked, got.Reason) + + got = Apply(prev, Observation{Known: true, Claimed: true, Minutes: 60, Required: 60}, t0.Add(time.Minute)) + assert.Equal(t, Claimed, got.Status) + assert.Equal(t, NoReason, got.Reason) } // A row that falls back from Claimable to Accruing/Eligible sheds its claim diff --git a/internal/pipeline/loop/loop_test.go b/internal/pipeline/loop/loop_test.go index 248c314..e72a659 100644 --- a/internal/pipeline/loop/loop_test.go +++ b/internal/pipeline/loop/loop_test.go @@ -267,6 +267,46 @@ func TestLoop_StaleGenerationIgnored(t *testing.T) { _ = f } +// TestLoop_UnlinkedCampaign_SessionProgressNeverLiftsBlock covers the prod +// bug (2026-09-30): an unlinked campaign's Blocked/Unlinked drop was re-derived +// to Accruing by the next mid-watch session Progress event (loop.applyProgress, +// a Known=true observation just like reconcile's), so the planner mined it +// between syncs even though its claim could never succeed. Revert-proof: +// revert the dropstate.Apply Unlinked guard and this fails — applyProgress +// flips d1 back to Accruing and a following replan starts watching ch1 again. +func TestLoop_UnlinkedCampaign_SessionProgressNeverLiftsBlock(t *testing.T) { + f, st, _, cfg := setup(t) + f.camps[0].AccountLinked = false + l, err := New(cfg) + require.NoError(t, err) + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + require.NoError(t, l.load(ctx)) + l.reconcile(ctx) + require.Equal(t, dropstate.Blocked, st.get("d1").Status) + require.Equal(t, dropstate.Unlinked, st.get("d1").Reason) + + l.refreshLive(ctx) + l.replan(ctx) + assert.Equal(t, "idle", l.Snapshot().State, "the only drop is blocked, so there is nothing to mine") + + // A session Progress event (as loop.applyProgress folds mid-watch, the + // same path a live watch session feeds between reconciles) must not lift + // the campaign link block on its own. + l.applyProgress(ctx, []platform.Progress{{BenefitID: "d1", MinutesWatched: 50}}) + assert.Equal(t, dropstate.Blocked, st.get("d1").Status, "session progress must not lift a campaign link block") + assert.Equal(t, dropstate.Unlinked, st.get("d1").Reason) + + l.refreshLive(ctx) + l.replan(ctx) + assert.Equal(t, "idle", l.Snapshot().State, "the planner must still not pick the unlinked drop") + f.mu.Lock() + assert.Empty(t, f.watching, "no StartWatch for an unlinked campaign's drop") + f.mu.Unlock() + l.stopSession() +} + func TestLoop_BridgesManualMarksOnLoad(t *testing.T) { _, st, h, cfg := setup(t) st.rows["d1"] = dropstate.Row{AccountID: "acc", DropID: "d1", CampaignID: "c1", Platform: "twitch", Status: dropstate.Accruing, Minutes: 10, Required: 60, Source: dropstate.FromPlatform} diff --git a/internal/pipeline/reconcile/reconcile.go b/internal/pipeline/reconcile/reconcile.go index e2021bf..cbf28d4 100644 --- a/internal/pipeline/reconcile/reconcile.go +++ b/internal/pipeline/reconcile/reconcile.go @@ -120,7 +120,15 @@ func Run(ctx context.Context, cfg Config, prev map[string]dropstate.Row, now tim if known { req = o.Required } - next = dropstate.Apply(p, dropstate.Observation{ + applyFrom := p + if linked && p.Status == dropstate.Blocked && p.Reason == dropstate.Unlinked { + // Only the reconciler may lift a campaign link block, and + // only once it has confirmed the campaign is linked: a + // non-claimed observation alone (e.g. a mid-watch session + // Progress event) must never lift it. + applyFrom = dropstate.Retry(p, now) + } + next = dropstate.Apply(applyFrom, dropstate.Observation{ Known: known && o.Known, Unmineable: o.Unmineable, Claimed: o.Claimed, Minutes: o.Minutes, Required: req, }, now) diff --git a/internal/pipeline/reconcile/reconcile_test.go b/internal/pipeline/reconcile/reconcile_test.go index 7406e08..8d6351e 100644 --- a/internal/pipeline/reconcile/reconcile_test.go +++ b/internal/pipeline/reconcile/reconcile_test.go @@ -131,6 +131,25 @@ func TestRun_LinkedAfterUnlinkedUnblocksImmediately(t *testing.T) { assert.Equal(t, dropstate.Eligible, byDrop(res.Rows)["d2"].Status) } +// Companion to TestRun_LinkedAfterUnlinkedUnblocksImmediately: when the +// campaign is still unlinked, a Blocked/Unlinked row must stay that way +// (dropstate.Apply alone no longer lifts it, and reconcile.Run re-applies +// BlockLink after Apply). +func TestRun_StillUnlinkedStaysBlocked(t *testing.T) { + c := camp("c1", "G", "d1") + c.AccountLinked = false + f := &fakeBackend{camps: []platform.Campaign{c}, + progress: []platform.DropProgress{{DropID: "d1", CampaignID: "c1", Minutes: 20, Required: 60, Known: true}}} + prev := map[string]dropstate.Row{"d1": {AccountID: "a", DropID: "d1", CampaignID: "c1", Platform: "twitch", + Status: dropstate.Blocked, Reason: dropstate.Unlinked, Minutes: 10, Required: 60, + Source: dropstate.FromPlatform, RetryAfter: now.Add(dropstate.RetryNeedsLink - time.Minute)}} + res, err := Run(context.Background(), cfg(f, &fakePersister{}), prev, now) + require.NoError(t, err) + r := byDrop(res.Rows)["d1"] + assert.Equal(t, dropstate.Blocked, r.Status) + assert.Equal(t, dropstate.Unlinked, r.Reason) +} + // C1: a backfilled ghost-skip (Required 0, immediate retry) on Kick, where the // platform reports unlisted rewards Known=false, becomes mineable again. func TestRun_BackfilledKickSkipBecomesEligible(t *testing.T) { From 96c9db76871e1218db96db0574ef9dc0a77727f1 Mon Sep 17 00:00:00 2001 From: Alejandro Date: Tue, 29 Sep 2026 21:22:03 -0400 Subject: [PATCH 37/56] feat(ui): dashboard/drops banner for Twitch TV-login discovery limit Twitch's TV OAuth client (every login since #48/v1.4.0) can't see the drops dashboard, so those accounts only find campaigns for whitelisted games. Add a banner on the dashboard and /drops when at least one enabled Twitch account has a TV-client session, pointing at the game whitelist (/priority); it renders as a warning when that account's effective whitelist (account games, falling back to global) is empty. Reuses the existing dashAlert banner component (now factored into a shared _dash_alerts.html partial so /drops gets it too) instead of a new one, per the design system's flat/no-new-component rule. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/CHANGELOG.md | 1 + internal/api/handlers_dashboard.go | 22 +- internal/api/handlers_drops.go | 8 + internal/api/server.go | 1 + internal/api/tv_discovery.go | 70 ++++++ internal/api/tv_discovery_test.go | 257 +++++++++++++++++++++++ internal/i18n/locales/en.json | 4 + internal/i18n/locales/es.json | 4 + internal/i18n/locales/zh-CN.json | 4 + internal/web/static/css/app.css | 10 + internal/web/templates/_dash_alerts.html | 19 ++ internal/web/templates/dashboard.html | 15 +- internal/web/templates/drops.html | 2 + 13 files changed, 399 insertions(+), 18 deletions(-) create mode 100644 internal/api/tv_discovery.go create mode 100644 internal/api/tv_discovery_test.go create mode 100644 internal/web/templates/_dash_alerts.html diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index a6504af..d8a14e5 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -10,6 +10,7 @@ All notable changes to GrubDrops. - Streamer priority for pipeline v2: per-account priority streamers are watched first, then any live channel in the category (#49). Seeded from the existing null-game channel list. Until the settings editor lands, v2 reads a one-time copy of the null-game channel list taken at upgrade. - `drop_state` and `account_streamer_priority` tables (migration 0016), with a one-time backfill from claim history, manual marks and ghost-skips. - Accounts on the Twitch TV login share the full campaign list found by any Android-login account in the same GrubDrops, so campaigns hidden from the TV client (for example behind a global badge campaign) are still found. +- A banner explains that Twitch limits campaign discovery for new (TV) logins and points to the game whitelist. ### Changed diff --git a/internal/api/handlers_dashboard.go b/internal/api/handlers_dashboard.go index ede2534..64835d6 100644 --- a/internal/api/handlers_dashboard.go +++ b/internal/api/handlers_dashboard.go @@ -62,6 +62,10 @@ type dashboardDeps struct { s *store.Settings start time.Time loc *timeutil.Zone // display timezone (live; setting → TZ env → UTC) + // sessions loads decrypted platform sessions, used only to detect a + // Twitch TV-client login for the tv_discovery banner. Nil-safe: + // tvDiscoveryAlert returns nil when unset. + sessions *store.SessionStore // channelCounters is keyed by platform name ("twitch", "kick"). Nil // or missing entries make the dashboard fall back to zero for that // platform — safer than panicking when a backend isn't wired up. @@ -250,10 +254,15 @@ type dashPage struct { } type dashAlert struct { - Kind string // "needs_auth" | "no_drops" - Account string // display @login - URL string // direct CTA link - Action string // button label + Kind string // "needs_auth" | "no_drops" | "tv_discovery" | ... + Account string // display @login, or (for account-agnostic alerts like + // tv_discovery) the alert's bold title text + URL string // direct CTA link + Action string // button label + // Warning styles the alert with the red/warning accent instead of the + // default info accent (same copy, different emphasis). Used by + // tv_discovery when the affected account's effective whitelist is empty. + Warning bool } func (d dashboardDeps) collectPage(r *http.Request) dashPage { @@ -310,6 +319,11 @@ func (d dashboardDeps) collectPage(r *http.Request) dashPage { // Build alerts: any account in needs_auth state or sleeping with 0 // eligible drops gets a top banner pointing at the right CTA. alerts := buildDashAlerts(r.Context(), d.q, cards, lang) + // tv_discovery goes first — it's an install-wide notice, not tied to + // one account, so it leads the other per-account alerts. + if tv := tvDiscoveryAlert(r.Context(), d.q, d.sessions, lang); tv != nil { + alerts = append([]dashAlert{*tv}, alerts...) + } camps := activeCampsFromDiscovery(r.Context(), d.sch, d.channelCounters, d.q, lang) diff --git a/internal/api/handlers_drops.go b/internal/api/handlers_drops.go index 382a7c9..41c1d97 100644 --- a/internal/api/handlers_drops.go +++ b/internal/api/handlers_drops.go @@ -16,6 +16,7 @@ import ( "github.com/go-chi/chi/v5" "github.com/aalejandrofer/grubdrops/internal/gameslug" + "github.com/aalejandrofer/grubdrops/internal/i18n" "github.com/aalejandrofer/grubdrops/internal/platform" "github.com/aalejandrofer/grubdrops/internal/store" "github.com/aalejandrofer/grubdrops/internal/store/gen" @@ -286,6 +287,9 @@ type dropsPage struct { // a bootstrap CTA in this case instead of misleading "discovery populates // this" empty text. NoWhitelist bool + // Alerts is the same top-of-page banner component the dashboard renders + // (dashAlert) — reused here for the tv_discovery notice. + Alerts []dashAlert } type dropsAccount struct { @@ -437,6 +441,7 @@ func (d *dropsDeps) list(w http.ResponseWriter, r *http.Request) { tab = tabCurrent } + lang := i18n.DetectLang(r) allow, hasWhitelist := allowedGamesUnion(r.Context(), d.q) now := time.Now().Unix() const limit = 200 @@ -526,6 +531,9 @@ func (d *dropsDeps) list(w http.ResponseWriter, r *http.Request) { CSRFToken: csrfToken(r), NoWhitelist: !hasWhitelist, } + if tv := tvDiscoveryAlert(r.Context(), d.q, d.sessions, lang); tv != nil { + page.Alerts = append(page.Alerts, *tv) + } switch tab { case tabPast: page.Rows = pastRows diff --git a/internal/api/server.go b/internal/api/server.go index 0e553a2..33a6012 100644 --- a/internal/api/server.go +++ b/internal/api/server.go @@ -177,6 +177,7 @@ func NewRouter(d Deps) http.Handler { loc: d.Zone, channelCounters: channelCountersFromRegistry(d.Registry), kickPath: d.KickActivePath, + sessions: d.Sessions, } accs := accountsDeps{q: d.Q, db: d.DB, t: d.Templates, sm: d.Session, sch: d.Scheduler, reload: d.Reload, authCheck: d.AuthCheck, reloadAccount: d.ReloadAccount, rootCtx: d.RootCtx, loc: d.Zone} loginTwitch := newLoginTwitchDeps(d, d.RootCtx) diff --git a/internal/api/tv_discovery.go b/internal/api/tv_discovery.go new file mode 100644 index 0000000..7bb9f73 --- /dev/null +++ b/internal/api/tv_discovery.go @@ -0,0 +1,70 @@ +package api + +import ( + "context" + + "github.com/aalejandrofer/grubdrops/internal/i18n" + "github.com/aalejandrofer/grubdrops/internal/platform/twitch" + "github.com/aalejandrofer/grubdrops/internal/store" + "github.com/aalejandrofer/grubdrops/internal/store/gen" +) + +// tvDiscoveryStatus reports whether at least one ENABLED Twitch account's +// session was minted by the TV client (Session.ClientID == twitch.ClientTV — +// every login since Twitch blocked the Android device-code client, #48) and, +// when so, whether any such account's effective whitelist is empty. A TV +// session can't see Twitch's drops dashboard at all, so its campaign +// discovery depends entirely on the whitelist (own games, falling back to +// the global list — the same resolution gameNamesForAccount already uses); +// an empty one means the account finds nothing. +// +// Cheap by construction: scans ENABLED accounts only (few, per +// buildDashAlerts' own assumption) and decrypts each Twitch session once. +// Returns booleans only — never session/token material — so callers can +// pass the result straight to a template without risk. +func tvDiscoveryStatus(ctx context.Context, q *gen.Queries, sessions *store.SessionStore) (anyTV bool, whitelistEmpty bool) { + if q == nil || sessions == nil { + return false, false + } + accs, err := q.ListEnabledAccounts(ctx) + if err != nil { + return false, false + } + for _, a := range accs { + if a.Platform != "twitch" { + continue + } + sess, ok, err := sessions.Get(ctx, a.ID) + if err != nil || !ok { + continue + } + if sess.ClientID != twitch.ClientTV { + continue + } + anyTV = true + if len(gameNamesForAccount(ctx, q, a.ID)) == 0 { + whitelistEmpty = true + } + } + return anyTV, whitelistEmpty +} + +// tvDiscoveryAlert builds the top-of-page banner warning that Twitch's +// TV-client login can't see the drops dashboard, so GrubDrops only finds +// campaigns for whitelisted games. It reuses the existing dashAlert +// component (the same one handlers_dashboard.go renders for needs_auth, +// etc.) so the dashboard and /drops share one code path and one CSS +// component. Returns nil when no enabled Twitch account has a TV session. +func tvDiscoveryAlert(ctx context.Context, q *gen.Queries, sessions *store.SessionStore, lang string) *dashAlert { + anyTV, empty := tvDiscoveryStatus(ctx, q, sessions) + if !anyTV { + return nil + } + return &dashAlert{ + Kind: "tv_discovery", + Account: i18n.T(lang, "dashboard.tv_discovery_title"), + URL: "/priority", + Action: i18n.T(lang, "dashboard.action_whitelist_games"), + Warning: empty, + } +} diff --git a/internal/api/tv_discovery_test.go b/internal/api/tv_discovery_test.go new file mode 100644 index 0000000..b4f031d --- /dev/null +++ b/internal/api/tv_discovery_test.go @@ -0,0 +1,257 @@ +package api + +import ( + "bytes" + "context" + "net/http" + "net/http/httptest" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/stretchr/testify/require" + + "github.com/aalejandrofer/grubdrops/internal/platform" + "github.com/aalejandrofer/grubdrops/internal/platform/twitch" + "github.com/aalejandrofer/grubdrops/internal/store" + "github.com/aalejandrofer/grubdrops/internal/store/gen" + "github.com/aalejandrofer/grubdrops/internal/timeutil" + "github.com/aalejandrofer/grubdrops/internal/web" +) + +// tvDiscoveryTestKey is an age secret key used only in tests. +const tvDiscoveryTestKey = "AGE-SECRET-KEY-1DZCAXYWJM6M42NSX5GR4QWZZ2JXEYKJ9ZKWYFYSNU997775JJ6XSY85FK9" + +func tvDiscoverySetup(t *testing.T) (context.Context, *gen.Queries, *store.SessionStore) { + t.Helper() + ctx := context.Background() + db, err := store.Open(ctx, filepath.Join(t.TempDir(), "tv.db")) + require.NoError(t, err) + t.Cleanup(func() { _ = db.Close() }) + q := gen.New(db) + c, err := store.NewCryptor(tvDiscoveryTestKey) + require.NoError(t, err) + sessions := store.NewSessionStore(db, q, c) + return ctx, q, sessions +} + +func createTwitchAccount(t *testing.T, ctx context.Context, q *gen.Queries, id string, enabled bool) { + t.Helper() + now := time.Now().Unix() + en := int64(0) + if enabled { + en = 1 + } + _, err := q.CreateAccount(ctx, gen.CreateAccountParams{ + ID: id, Platform: "twitch", DisplayName: id, + Status: "idle", FingerprintJson: "{}", Enabled: en, + CreatedAt: now, UpdatedAt: now, + }) + require.NoError(t, err) +} + +// No enabled Twitch account at all → no alert (Android-only or empty install). +func TestTVDiscoveryStatus_NoAccounts(t *testing.T) { + ctx, q, sessions := tvDiscoverySetup(t) + anyTV, empty := tvDiscoveryStatus(ctx, q, sessions) + require.False(t, anyTV) + require.False(t, empty) +} + +// An Android-login account (ClientID "") must never trigger the banner. +func TestTVDiscoveryStatus_AndroidOnlyDoesNotTrigger(t *testing.T) { + ctx, q, sessions := tvDiscoverySetup(t) + createTwitchAccount(t, ctx, q, "acc_android", true) + require.NoError(t, sessions.Put(ctx, "acc_android", platform.Session{ClientID: ""})) + + anyTV, empty := tvDiscoveryStatus(ctx, q, sessions) + require.False(t, anyTV) + require.False(t, empty) +} + +// A TV-client session on an enabled Twitch account triggers the banner. With +// at least one whitelisted game (account-level), the whitelist is NOT empty. +func TestTVDiscoveryStatus_TVAccountWithWhitelist(t *testing.T) { + ctx, q, sessions := tvDiscoverySetup(t) + createTwitchAccount(t, ctx, q, "acc_tv", true) + require.NoError(t, sessions.Put(ctx, "acc_tv", platform.Session{ClientID: twitch.ClientTV})) + require.NoError(t, q.UpsertGame(ctx, gen.UpsertGameParams{ID: "g_test_tv", Name: "Test TV Game", Slug: "test-tv-game"})) + require.NoError(t, q.AddAccountGame(ctx, gen.AddAccountGameParams{AccountID: "acc_tv", GameID: "g_test_tv", Rank: 0})) + + anyTV, empty := tvDiscoveryStatus(ctx, q, sessions) + require.True(t, anyTV) + require.False(t, empty) +} + +// A TV-client session with NO account games and NO global games → empty +// effective whitelist, the warning-variant trigger. +func TestTVDiscoveryStatus_TVAccountEmptyWhitelist(t *testing.T) { + ctx, q, sessions := tvDiscoverySetup(t) + createTwitchAccount(t, ctx, q, "acc_tv", true) + require.NoError(t, sessions.Put(ctx, "acc_tv", platform.Session{ClientID: twitch.ClientTV})) + + anyTV, empty := tvDiscoveryStatus(ctx, q, sessions) + require.True(t, anyTV) + require.True(t, empty) +} + +// A TV account with no account-level games but a non-empty GLOBAL whitelist +// falls back to it (same resolution as gameNamesForAccount) — not empty. +func TestTVDiscoveryStatus_TVAccountFallsBackToGlobal(t *testing.T) { + ctx, q, sessions := tvDiscoverySetup(t) + createTwitchAccount(t, ctx, q, "acc_tv", true) + require.NoError(t, sessions.Put(ctx, "acc_tv", platform.Session{ClientID: twitch.ClientTV})) + require.NoError(t, q.UpsertGame(ctx, gen.UpsertGameParams{ID: "g_test_tv", Name: "Test TV Game", Slug: "test-tv-game"})) + require.NoError(t, q.AddGlobalGame(ctx, gen.AddGlobalGameParams{GameID: "g_test_tv", Rank: 0})) + + anyTV, empty := tvDiscoveryStatus(ctx, q, sessions) + require.True(t, anyTV) + require.False(t, empty) +} + +// A DISABLED Twitch account with a TV session must not trigger the banner — +// only ENABLED accounts count, matching buildDashAlerts' own rule. +func TestTVDiscoveryStatus_DisabledAccountIgnored(t *testing.T) { + ctx, q, sessions := tvDiscoverySetup(t) + createTwitchAccount(t, ctx, q, "acc_tv_off", false) + require.NoError(t, sessions.Put(ctx, "acc_tv_off", platform.Session{ClientID: twitch.ClientTV})) + + anyTV, empty := tvDiscoveryStatus(ctx, q, sessions) + require.False(t, anyTV) + require.False(t, empty) +} + +// tvDiscoveryAlert renders nil when there is nothing to warn about. +func TestTVDiscoveryAlert_NilWhenNoTVAccount(t *testing.T) { + ctx, q, sessions := tvDiscoverySetup(t) + require.Nil(t, tvDiscoveryAlert(ctx, q, sessions, "en")) +} + +// tvDiscoveryAlert renders the info variant (Warning=false) when a TV +// account has a non-empty whitelist, and points at /priority. +func TestTVDiscoveryAlert_InfoVariant(t *testing.T) { + ctx, q, sessions := tvDiscoverySetup(t) + createTwitchAccount(t, ctx, q, "acc_tv", true) + require.NoError(t, sessions.Put(ctx, "acc_tv", platform.Session{ClientID: twitch.ClientTV})) + require.NoError(t, q.UpsertGame(ctx, gen.UpsertGameParams{ID: "g_test_tv", Name: "Test TV Game", Slug: "test-tv-game"})) + require.NoError(t, q.AddAccountGame(ctx, gen.AddAccountGameParams{AccountID: "acc_tv", GameID: "g_test_tv", Rank: 0})) + + a := tvDiscoveryAlert(ctx, q, sessions, "en") + require.NotNil(t, a) + require.Equal(t, "tv_discovery", a.Kind) + require.False(t, a.Warning) + require.Equal(t, "/priority", a.URL) + require.NotEmpty(t, a.Account) + require.NotEmpty(t, a.Action) +} + +// tvDiscoveryAlert renders the warning variant when the TV account's +// effective whitelist is empty. +func TestTVDiscoveryAlert_WarningVariant(t *testing.T) { + ctx, q, sessions := tvDiscoverySetup(t) + createTwitchAccount(t, ctx, q, "acc_tv", true) + require.NoError(t, sessions.Put(ctx, "acc_tv", platform.Session{ClientID: twitch.ClientTV})) + + a := tvDiscoveryAlert(ctx, q, sessions, "en") + require.NotNil(t, a) + require.True(t, a.Warning) +} + +// renderDashAlerts renders the shared dash_alerts partial (the same +// component both dashboard.html and drops.html use) with an ad-hoc page +// value carrying only the Alerts field, matching the field-by-name lookup +// html/template uses. +func renderDashAlerts(t *testing.T, alerts []dashAlert) string { + t.Helper() + tmpl, err := web.Templates() + require.NoError(t, err) + var buf bytes.Buffer + page := struct{ Alerts []dashAlert }{Alerts: alerts} + require.NoError(t, tmpl.ExecuteTemplate(&buf, "dash_alerts", page)) + return buf.String() +} + +// The tv_discovery banner (info variant) renders the title, body, CTA link +// to /priority, and does NOT carry the warning class. +func TestDashAlerts_TVDiscoveryInfoVariant(t *testing.T) { + out := renderDashAlerts(t, []dashAlert{{ + Kind: "tv_discovery", + Account: "Drop discovery is limited.", + URL: "/priority", + Action: "Whitelist games →", + Warning: false, + }}) + require.Contains(t, out, `href="/priority"`) + require.Contains(t, out, "Drop discovery is limited.") + require.Contains(t, out, "Twitch no longer lists campaigns for new logins") + require.Contains(t, out, "Whitelist games") + require.Contains(t, out, `class="alert alert-tv_discovery"`) + require.NotContains(t, out, "warning") +} + +// The warning variant (empty effective whitelist) adds the warning class to +// the same markup/copy. +func TestDashAlerts_TVDiscoveryWarningVariant(t *testing.T) { + out := renderDashAlerts(t, []dashAlert{{ + Kind: "tv_discovery", + Account: "Drop discovery is limited.", + URL: "/priority", + Action: "Whitelist games →", + Warning: true, + }}) + require.Contains(t, out, `class="alert alert-tv_discovery warning"`) +} + +// No alerts at all → the section doesn't render. +func TestDashAlerts_EmptyWhenNoAlerts(t *testing.T) { + out := renderDashAlerts(t, nil) + require.False(t, strings.Contains(out, `class="alerts"`)) +} + +// End-to-end: an enabled Twitch account with a TV session and no whitelist +// produces a dashboard page whose Alerts include the warning-variant +// tv_discovery entry (collectPage is the real dashboard view-model builder). +func TestDashboardCollectPage_TVDiscoveryBanner(t *testing.T) { + ctx, q, sessions := tvDiscoverySetup(t) + createTwitchAccount(t, ctx, q, "acc_tv", true) + require.NoError(t, sessions.Put(ctx, "acc_tv", platform.Session{ClientID: twitch.ClientTV})) + + d := dashboardDeps{q: q, sessions: sessions, start: time.Now(), loc: timeutil.NewZone(time.UTC)} + req := httptest.NewRequest(http.MethodGet, "/", nil) + page := d.collectPage(req) + + require.Len(t, page.Alerts, 1) + require.Equal(t, "tv_discovery", page.Alerts[0].Kind) + require.True(t, page.Alerts[0].Warning) +} + +// An Android-only install (no TV session anywhere) must not raise the +// banner on the dashboard. +func TestDashboardCollectPage_NoBannerForAndroidOnly(t *testing.T) { + ctx, q, sessions := tvDiscoverySetup(t) + createTwitchAccount(t, ctx, q, "acc_android", true) + require.NoError(t, sessions.Put(ctx, "acc_android", platform.Session{ClientID: ""})) + + d := dashboardDeps{q: q, sessions: sessions, start: time.Now(), loc: timeutil.NewZone(time.UTC)} + req := httptest.NewRequest(http.MethodGet, "/", nil) + page := d.collectPage(req) + + for _, a := range page.Alerts { + require.NotEqual(t, "tv_discovery", a.Kind) + } +} + +// The /drops page shares the same tvDiscoveryAlert construction: a TV +// account with a non-empty whitelist gets the info variant, not warning. +func TestDropsPage_TVDiscoveryBannerInfoVariant(t *testing.T) { + ctx, q, sessions := tvDiscoverySetup(t) + createTwitchAccount(t, ctx, q, "acc_tv", true) + require.NoError(t, sessions.Put(ctx, "acc_tv", platform.Session{ClientID: twitch.ClientTV})) + require.NoError(t, q.UpsertGame(ctx, gen.UpsertGameParams{ID: "g_test_tv2", Name: "Test TV Game 2", Slug: "test-tv-game-2"})) + require.NoError(t, q.AddAccountGame(ctx, gen.AddAccountGameParams{AccountID: "acc_tv", GameID: "g_test_tv2", Rank: 0})) + + tv := tvDiscoveryAlert(ctx, q, sessions, "en") + require.NotNil(t, tv) + require.False(t, tv.Warning) +} diff --git a/internal/i18n/locales/en.json b/internal/i18n/locales/en.json index cc2279c..373fe05 100644 --- a/internal/i18n/locales/en.json +++ b/internal/i18n/locales/en.json @@ -54,6 +54,10 @@ "dashboard.action_device_code": "Switch to device-code login →", "dashboard.action_connect": "Connect account →", "dashboard.action_add_games": "Add games →", + "dashboard.tv_discovery": "discovery limited", + "dashboard.tv_discovery_title": "Drop discovery is limited.", + "dashboard.alert_tv_discovery": "Twitch no longer lists campaigns for new logins, so GrubDrops only finds drops for games on your whitelist.", + "dashboard.action_whitelist_games": "Whitelist games →", "dashboard_modal.account": "account", "dashboard_modal.close": "Close", diff --git a/internal/i18n/locales/es.json b/internal/i18n/locales/es.json index 5370d3e..a9b5f4d 100644 --- a/internal/i18n/locales/es.json +++ b/internal/i18n/locales/es.json @@ -54,6 +54,10 @@ "dashboard.action_device_code": "Cambiar a inicio con código de dispositivo →", "dashboard.action_connect": "Conectar cuenta →", "dashboard.action_add_games": "Añadir juegos →", + "dashboard.tv_discovery": "descubrimiento limitado", + "dashboard.tv_discovery_title": "El descubrimiento de drops está limitado.", + "dashboard.alert_tv_discovery": "Twitch ya no lista campañas para inicios de sesión nuevos, así que GrubDrops solo encuentra drops de los juegos en tu lista blanca.", + "dashboard.action_whitelist_games": "Añadir juegos a la lista blanca →", "dashboard_modal.account": "cuenta", "dashboard_modal.close": "Cerrar", diff --git a/internal/i18n/locales/zh-CN.json b/internal/i18n/locales/zh-CN.json index 99d1866..3893a2f 100644 --- a/internal/i18n/locales/zh-CN.json +++ b/internal/i18n/locales/zh-CN.json @@ -54,6 +54,10 @@ "dashboard.action_device_code": "切换到设备代码登录 →", "dashboard.action_connect": "关联账号 →", "dashboard.action_add_games": "添加游戏 →", + "dashboard.tv_discovery": "发现受限", + "dashboard.tv_discovery_title": "掉宝发现功能受限。", + "dashboard.alert_tv_discovery": "Twitch 不再为新登录列出活动，因此 GrubDrops 只能找到白名单游戏的掉宝。", + "dashboard.action_whitelist_games": "加入游戏白名单 →", "dashboard_modal.account": "账号", "dashboard_modal.close": "关闭", diff --git a/internal/web/static/css/app.css b/internal/web/static/css/app.css index c8a44ea..873c74c 100644 --- a/internal/web/static/css/app.css +++ b/internal/web/static/css/app.css @@ -340,6 +340,16 @@ color: var(--accent); } .alert-cta:hover { color: var(--text); background: none; border: none; } + /* Warning emphasis — same flat alert row, red accent instead of the + default info orange (matches the red-for-warning convention used + elsewhere: .login-flash, .dot.error). Used when the condition the + alert describes has no fallback (e.g. an empty whitelist). */ + .alert.warning { + box-shadow: inset 3px 0 0 var(--red); + background: rgba(198, 74, 60, 0.06); + } + .alert.warning .alert-tag, + .alert.warning .alert-cta { color: var(--red); } /* TELEMETRY STRIP — 6 stats */ /* Flat stat strip — no boxed grid; cards separated by space, the strip diff --git a/internal/web/templates/_dash_alerts.html b/internal/web/templates/_dash_alerts.html new file mode 100644 index 0000000..359bc6a --- /dev/null +++ b/internal/web/templates/_dash_alerts.html @@ -0,0 +1,19 @@ +{{/* Shared top-of-page alert/banner component (dashAlert). Reused by + dashboard.html and drops.html so both pages render CTA banners + (needs_auth, no_games, tv_discovery, ...) from the same markup + CSS. */}} +{{define "dash_alerts"}} +{{if .Alerts}} +
+ {{range .Alerts}} +
+ {{t (printf "dashboard.%s" .Kind)}} + + {{.Account}} + — {{t (printf "dashboard.alert_%s" .Kind)}} + + {{.Action}} +
+ {{end}} +
+{{end}} +{{end}} diff --git a/internal/web/templates/dashboard.html b/internal/web/templates/dashboard.html index 820ff8e..1d65845 100644 --- a/internal/web/templates/dashboard.html +++ b/internal/web/templates/dashboard.html @@ -18,20 +18,7 @@

{{t "dashboard.heading"}}

-{{if .Alerts}} -
- {{range .Alerts}} -
- {{t (printf "dashboard.%s" .Kind)}} - - {{.Account}} - — {{t (printf "dashboard.alert_%s" .Kind)}} - - {{.Action}} -
- {{end}} -
-{{end}} +{{template "dash_alerts" .}}
diff --git a/internal/web/templates/drops.html b/internal/web/templates/drops.html index 77bcbee..92749cf 100644 --- a/internal/web/templates/drops.html +++ b/internal/web/templates/drops.html @@ -15,6 +15,8 @@

{{t "drops.heading"}}

+ {{template "dash_alerts" .}} +
Date: Tue, 29 Sep 2026 21:23:00 -0400 Subject: [PATCH 38/56] docs: note limited Twitch discovery for new logins in README Co-Authored-By: Claude Opus 5.5 (1M context) --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index b6c6091..1813e26 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,5 @@ +> ❗ **Drop discovery is limited.** Twitch no longer lists campaigns for new logins, so GrubDrops only finds drops for games on your whitelist. +

GrubDrops

From 5b7dc1b421347a1396b5c3cef45040ef6caa6f10 Mon Sep 17 00:00:00 2001 From: Alejandro Date: Tue, 29 Sep 2026 22:12:19 -0400 Subject: [PATCH 39/56] fix(pipeline): wire drops-page channel whitelist and force-watch into v2 - Channel add/remove on /drops now writes both account_channels (v1) and account_streamer_priority (v2), since v2 reads the priority table exclusively and account_channels was only ever copied there once by the 0016 migration seed. New adds rank after the account's current max. - Add sqlc AddStreamerPriority/RemoveStreamerPriority queries. - The v2 branch in build() now only passes ListForceChannels rows when the account's force-watch toggle is enabled, via a shared forceWatchEnabled helper reused by v1's forceWatchStore.Next (no v1 behaviour change). Co-Authored-By: Claude Opus 5.5 (1M context) --- cmd/miner/main.go | 39 +++++-- cmd/miner/main_test.go | 50 +++++++++ docs/CHANGELOG.md | 2 + internal/api/handlers_drops.go | 30 +++++- internal/api/handlers_drops_channel_test.go | 104 +++++++++++++++++++ internal/store/gen/querier.go | 2 + internal/store/gen/streamer_priority.sql.go | 37 +++++++ internal/store/queries/streamer_priority.sql | 8 ++ 8 files changed, 263 insertions(+), 9 deletions(-) diff --git a/cmd/miner/main.go b/cmd/miner/main.go index f6fbb0a..df88e3b 100644 --- a/cmd/miner/main.go +++ b/cmd/miner/main.go @@ -483,12 +483,7 @@ func run() error { if err != nil { logger.Warn("pipeline v2: load streamer priority failed", "account", a.ID, "err", err) } - var force []string - if rows, err := q.ListForceChannels(ctx, a.ID); err == nil { - for _, r := range rows { - force = append(force, r.Channel) - } - } + force := v2ForceChannels(ctx, q, a.ID) l, err := loop.New(loop.Config{ AccountID: a.ID, AccountLabel: a.DisplayName, Platform: a.Platform, Backend: b, Session: sess, @@ -1091,8 +1086,7 @@ func loadAccountChannels(ctx context.Context, q *gen.Queries, accountID string) type forceWatchStore struct{ q *gen.Queries } func (f forceWatchStore) Next(ctx context.Context, accountID string) (watcher.ForceTask, bool) { - v, err := f.q.GetSettingString(ctx, api.ForceWatchEnabledKey(accountID)) - if err != nil || string(v) != "1" { + if !forceWatchEnabled(ctx, f.q, accountID) { return watcher.ForceTask{}, false } rows, err := f.q.ListForceChannels(ctx, accountID) @@ -1102,6 +1096,35 @@ func (f forceWatchStore) Next(ctx context.Context, accountID string) (watcher.Fo return watcher.ForceTask{Channel: rows[0].Channel}, true } +// forceWatchEnabled reports whether the per-account force-watch toggle +// (force_watch: KV flag, set via /accounts/:id/force-watch) is +// on. Shared by v1's forceWatchStore and the v2 wiring so both agree on +// when a configured force-watch channel actually applies. +func forceWatchEnabled(ctx context.Context, q *gen.Queries, accountID string) bool { + v, err := q.GetSettingString(ctx, api.ForceWatchEnabledKey(accountID)) + return err == nil && string(v) == "1" +} + +// v2ForceChannels returns the account's configured force-watch channel +// logins when the force-watch toggle is enabled, or nil otherwise. Used by +// the pipeline v2 branch in build() so a disabled toggle behaves like v1's +// forceWatchStore.Next (no forced channel), instead of always passing the +// configured list. +func v2ForceChannels(ctx context.Context, q *gen.Queries, accountID string) []string { + if !forceWatchEnabled(ctx, q, accountID) { + return nil + } + rows, err := q.ListForceChannels(ctx, accountID) + if err != nil { + return nil + } + var force []string + for _, r := range rows { + force = append(force, r.Channel) + } + return force +} + // parseDuration parses a Go duration string (e.g. "5m", "30s") with a // fallback when the env var is empty or malformed. Matches the pattern // used elsewhere in the codebase for config-by-env. diff --git a/cmd/miner/main_test.go b/cmd/miner/main_test.go index 84105ce..359c96f 100644 --- a/cmd/miner/main_test.go +++ b/cmd/miner/main_test.go @@ -8,6 +8,7 @@ import ( "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" + "github.com/aalejandrofer/grubdrops/internal/api" "github.com/aalejandrofer/grubdrops/internal/store" "github.com/aalejandrofer/grubdrops/internal/store/gen" ) @@ -88,3 +89,52 @@ func TestMatchAnyChannel(t *testing.T) { assert.True(t, m([]string{"x", "fav"})) assert.False(t, m([]string{"x"})) } + +// TestForceWatchEnabled proves the extracted helper reads the same +// force_watch: KV flag forceWatchStore.Next checks, so v1 and v2 +// agree on whether force-watch is on for an account. +func TestForceWatchEnabled(t *testing.T) { + ctx := context.Background() + db, err := store.Open(ctx, t.TempDir()+"/t.db") + require.NoError(t, err) + t.Cleanup(func() { _ = db.Close() }) + q := gen.New(db) + + assert.False(t, forceWatchEnabled(ctx, q, "acc-1"), "flag unset -> disabled") + + require.NoError(t, q.UpsertSettingString(ctx, gen.UpsertSettingStringParams{ + Key: api.ForceWatchEnabledKey("acc-1"), Value: []byte("1"), + })) + assert.True(t, forceWatchEnabled(ctx, q, "acc-1"), "flag=1 -> enabled") + + assert.False(t, forceWatchEnabled(ctx, q, "acc-2"), "different account unaffected") +} + +// TestV2ForceChannels proves the v2 wiring only surfaces ListForceChannels +// rows when the account's force-watch flag is enabled — matching v1's +// forceWatchStore.Next gate (previously v2 passed the list unconditionally). +func TestV2ForceChannels(t *testing.T) { + ctx := context.Background() + db, err := store.Open(ctx, t.TempDir()+"/t.db") + require.NoError(t, err) + t.Cleanup(func() { _ = db.Close() }) + q := gen.New(db) + + now := time.Now().Unix() + _, err = q.CreateAccount(ctx, gen.CreateAccountParams{ + ID: "acc-1", Platform: "kick", DisplayName: "k", + Status: "idle", FingerprintJson: "{}", Enabled: 1, + CreatedAt: now, UpdatedAt: now, + }) + require.NoError(t, err) + require.NoError(t, q.AddForceChannel(ctx, gen.AddForceChannelParams{ + AccountID: "acc-1", Channel: "somechannel", Rank: 0, CreatedAt: now, + })) + + assert.Empty(t, v2ForceChannels(ctx, q, "acc-1"), "flag off -> no force channels even though rows exist") + + require.NoError(t, q.UpsertSettingString(ctx, gen.UpsertSettingStringParams{ + Key: api.ForceWatchEnabledKey("acc-1"), Value: []byte("1"), + })) + assert.Equal(t, []string{"somechannel"}, v2ForceChannels(ctx, q, "acc-1"), "flag on -> rows surfaced") +} diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index d8a14e5..6bcbacf 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -43,6 +43,8 @@ All notable changes to GrubDrops. - The stall claim check only runs on a drop's second stall, and an empty Twitch claim status counts as a failure, so a blind check can never mark an unenrolled drop claimed. - Drops already claimed under v1 no longer look unclaimed to v2 when an account switches over; v2 reads the claim history after every sync, not only at startup. - v2 no longer watches drops from campaigns that need an unlinked account between syncs. +- Channel whitelist edits on the drops page now reach v2 accounts. +- The force-watch on/off switch now applies to v2 accounts. ## [1.4.2] — 2026-09-28 diff --git a/internal/api/handlers_drops.go b/internal/api/handlers_drops.go index 41c1d97..2292e28 100644 --- a/internal/api/handlers_drops.go +++ b/internal/api/handlers_drops.go @@ -1030,10 +1030,22 @@ func (d *dropsDeps) addChannelWhitelist(w http.ResponseWriter, r *http.Request) http.Redirect(w, r, "/drops", http.StatusSeeOther) return } - if _, err := d.q.GetAccount(r.Context(), accID); err != nil { + acct, err := d.q.GetAccount(r.Context(), accID) + if err != nil { http.NotFound(w, r) return } + // v2 reads account_streamer_priority exclusively (account_channels was + // only ever copied there once, by the 0016 migration seed), so every + // add here must land in both tables to reach v2 accounts too. + nextRank := int64(0) + if prio, err := d.q.ListStreamerPriority(r.Context(), accID); err == nil { + for _, p := range prio { + if p.Rank >= nextRank { + nextRank = p.Rank + 1 + } + } + } seen := map[string]struct{}{} added := 0 for _, raw := range r.Form["channel"] { @@ -1054,6 +1066,16 @@ func (d *dropsDeps) addChannelWhitelist(w http.ResponseWriter, r *http.Request) http.Redirect(w, r, "/drops", http.StatusSeeOther) return } + if err := d.q.AddStreamerPriority(r.Context(), gen.AddStreamerPriorityParams{ + AccountID: accID, Platform: acct.Platform, Login: ch, Rank: nextRank, + }); err != nil { + if d.sm != nil { + d.sm.Put(r.Context(), "flash", "flash.channel_whitelist_failed") + } + http.Redirect(w, r, "/drops", http.StatusSeeOther) + return + } + nextRank++ added++ } // Reload the scheduler so the watcher re-picks immediately and starts @@ -1098,6 +1120,12 @@ func (d *dropsDeps) removeChannelWhitelist(w http.ResponseWriter, r *http.Reques http.Error(w, err.Error(), http.StatusInternalServerError) return } + if err := d.q.RemoveStreamerPriority(r.Context(), gen.RemoveStreamerPriorityParams{ + AccountID: accID, Login: ch, + }); err != nil { + http.Error(w, err.Error(), http.StatusInternalServerError) + return + } removed++ } if removed > 0 && d.reload != nil { diff --git a/internal/api/handlers_drops_channel_test.go b/internal/api/handlers_drops_channel_test.go index 707e0d8..9a2581c 100644 --- a/internal/api/handlers_drops_channel_test.go +++ b/internal/api/handlers_drops_channel_test.go @@ -54,3 +54,107 @@ func TestAddChannelWhitelist_InsertsRows(t *testing.T) { require.Len(t, rows, 1) assert.Equal(t, "adrianozendejas32", rows[0].Channel) } + +// TestAddChannelWhitelist_WritesBothTables proves that whitelisting a +// channel writes account_channels (v1) AND account_streamer_priority (v2), +// with the platform pulled from the account and rank = next rank (max+1) +// for that account — v2's dropStore.StreamerPriority never saw edits after +// the one-time 0016 migration seed, since it reads account_streamer_priority +// exclusively. +func TestAddChannelWhitelist_WritesBothTables(t *testing.T) { + ctx := context.Background() + db, err := store.Open(ctx, filepath.Join(t.TempDir(), "test.db")) + require.NoError(t, err) + t.Cleanup(func() { _ = db.Close() }) + q := gen.New(db) + + now := time.Now().Unix() + _, err = q.CreateAccount(ctx, gen.CreateAccountParams{ + ID: "acc-1", Platform: "kick", DisplayName: "TTik3r", + Status: "idle", FingerprintJson: "{}", Enabled: 1, + CreatedAt: now, UpdatedAt: now, + }) + require.NoError(t, err) + + // Pre-seed one existing priority row so the new one must land at + // max+1, not a hardcoded rank. + require.NoError(t, q.AddStreamerPriority(ctx, gen.AddStreamerPriorityParams{ + AccountID: "acc-1", Platform: "kick", Login: "existingchan", Rank: 0, + })) + + d := &dropsDeps{q: q} + + form := url.Values{} + form.Set("account_id", "acc-1") + form.Add("channel", "adrianozendejas32") + + req := httptest.NewRequest(http.MethodPost, "/drops/whitelist/channel", strings.NewReader(form.Encode())) + req.Header.Set("Content-Type", "application/x-www-form-urlencoded") + rec := httptest.NewRecorder() + d.addChannelWhitelist(rec, req) + + require.Equal(t, http.StatusSeeOther, rec.Code) + + acRows, err := q.ListAccountChannels(ctx, "acc-1") + require.NoError(t, err) + require.Len(t, acRows, 1) + assert.Equal(t, "adrianozendejas32", acRows[0].Channel) + + prioRows, err := q.ListStreamerPriority(ctx, "acc-1") + require.NoError(t, err) + require.Len(t, prioRows, 2) + var found bool + for _, r := range prioRows { + if r.Login == "adrianozendejas32" { + found = true + assert.Equal(t, int64(1), r.Rank, "new entry ranks after the existing max (0)") + } + } + assert.True(t, found, "adrianozendejas32 must be written to account_streamer_priority") +} + +// TestRemoveChannelWhitelist_DeletesFromBothTables proves the ✕ un-whitelist +// action deletes from account_channels (v1) AND account_streamer_priority +// (v2), so a removed channel actually stops being mined under v2 too. +func TestRemoveChannelWhitelist_DeletesFromBothTables(t *testing.T) { + ctx := context.Background() + db, err := store.Open(ctx, filepath.Join(t.TempDir(), "test.db")) + require.NoError(t, err) + t.Cleanup(func() { _ = db.Close() }) + q := gen.New(db) + + now := time.Now().Unix() + _, err = q.CreateAccount(ctx, gen.CreateAccountParams{ + ID: "acc-1", Platform: "kick", DisplayName: "TTik3r", + Status: "idle", FingerprintJson: "{}", Enabled: 1, + CreatedAt: now, UpdatedAt: now, + }) + require.NoError(t, err) + require.NoError(t, q.AddAccountChannel(ctx, gen.AddAccountChannelParams{ + AccountID: "acc-1", Channel: "adrianozendejas32", Rank: 0, + })) + require.NoError(t, q.AddStreamerPriority(ctx, gen.AddStreamerPriorityParams{ + AccountID: "acc-1", Platform: "kick", Login: "adrianozendejas32", Rank: 0, + })) + + d := &dropsDeps{q: q} + + form := url.Values{} + form.Set("account_id", "acc-1") + form.Add("channel", "adrianozendejas32") + + req := httptest.NewRequest(http.MethodPost, "/drops/whitelist/channel/remove", strings.NewReader(form.Encode())) + req.Header.Set("Content-Type", "application/x-www-form-urlencoded") + rec := httptest.NewRecorder() + d.removeChannelWhitelist(rec, req) + + require.Equal(t, http.StatusSeeOther, rec.Code) + + acRows, err := q.ListAccountChannels(ctx, "acc-1") + require.NoError(t, err) + assert.Empty(t, acRows) + + prioRows, err := q.ListStreamerPriority(ctx, "acc-1") + require.NoError(t, err) + assert.Empty(t, prioRows) +} diff --git a/internal/store/gen/querier.go b/internal/store/gen/querier.go index 567380d..a2b9dff 100644 --- a/internal/store/gen/querier.go +++ b/internal/store/gen/querier.go @@ -14,6 +14,7 @@ type Querier interface { AddAccountGame(ctx context.Context, arg AddAccountGameParams) error AddForceChannel(ctx context.Context, arg AddForceChannelParams) error AddGlobalGame(ctx context.Context, arg AddGlobalGameParams) error + AddStreamerPriority(ctx context.Context, arg AddStreamerPriorityParams) error AdminExists(ctx context.Context) (bool, error) ClearAccountChannels(ctx context.Context, accountID string) error ClearAccountGames(ctx context.Context, accountID string) error @@ -92,6 +93,7 @@ type Querier interface { RemoveAccountChannel(ctx context.Context, arg RemoveAccountChannelParams) error RemoveAccountGame(ctx context.Context, arg RemoveAccountGameParams) error RemoveForceChannel(ctx context.Context, arg RemoveForceChannelParams) error + RemoveStreamerPriority(ctx context.Context, arg RemoveStreamerPriorityParams) error SetAccountEnabled(ctx context.Context, arg SetAccountEnabledParams) error // Lifetime watch minutes: sum of per-benefit progress. Persistent, so it // survives restarts (unlike the heartbeat log ring used for today's count). diff --git a/internal/store/gen/streamer_priority.sql.go b/internal/store/gen/streamer_priority.sql.go index 1e9abbd..de64a43 100644 --- a/internal/store/gen/streamer_priority.sql.go +++ b/internal/store/gen/streamer_priority.sql.go @@ -9,6 +9,29 @@ import ( "context" ) +const addStreamerPriority = `-- name: AddStreamerPriority :exec +INSERT INTO account_streamer_priority (account_id, platform, login, rank) +VALUES (?, ?, ?, ?) +ON CONFLICT(account_id, login) DO UPDATE SET rank = excluded.rank, platform = excluded.platform +` + +type AddStreamerPriorityParams struct { + AccountID string `json:"account_id"` + Platform string `json:"platform"` + Login string `json:"login"` + Rank int64 `json:"rank"` +} + +func (q *Queries) AddStreamerPriority(ctx context.Context, arg AddStreamerPriorityParams) error { + _, err := q.db.ExecContext(ctx, addStreamerPriority, + arg.AccountID, + arg.Platform, + arg.Login, + arg.Rank, + ) + return err +} + const listStreamerPriority = `-- name: ListStreamerPriority :many SELECT login, rank FROM account_streamer_priority WHERE account_id = ? @@ -42,3 +65,17 @@ func (q *Queries) ListStreamerPriority(ctx context.Context, accountID string) ([ } return items, nil } + +const removeStreamerPriority = `-- name: RemoveStreamerPriority :exec +DELETE FROM account_streamer_priority WHERE account_id = ? AND login = ? +` + +type RemoveStreamerPriorityParams struct { + AccountID string `json:"account_id"` + Login string `json:"login"` +} + +func (q *Queries) RemoveStreamerPriority(ctx context.Context, arg RemoveStreamerPriorityParams) error { + _, err := q.db.ExecContext(ctx, removeStreamerPriority, arg.AccountID, arg.Login) + return err +} diff --git a/internal/store/queries/streamer_priority.sql b/internal/store/queries/streamer_priority.sql index d34e63b..29d823e 100644 --- a/internal/store/queries/streamer_priority.sql +++ b/internal/store/queries/streamer_priority.sql @@ -2,3 +2,11 @@ SELECT login, rank FROM account_streamer_priority WHERE account_id = ? ORDER BY rank ASC, login ASC; + +-- name: AddStreamerPriority :exec +INSERT INTO account_streamer_priority (account_id, platform, login, rank) +VALUES (?, ?, ?, ?) +ON CONFLICT(account_id, login) DO UPDATE SET rank = excluded.rank, platform = excluded.platform; + +-- name: RemoveStreamerPriority :exec +DELETE FROM account_streamer_priority WHERE account_id = ? AND login = ?; From 492c3cce033944a88b666f2c949c9acf5e904e0f Mon Sep 17 00:00:00 2001 From: Alejandro Date: Tue, 29 Sep 2026 22:18:25 -0400 Subject: [PATCH 40/56] perf(api): compute TV discovery banner only on full page renders Move tvDiscoveryAlert computation out of collectPage to the full-page handler (page()). The TV discovery banner decrypts every enabled Twitch account's session to check for TV-client logins, which was running unnecessarily on every polled HTMX partial (cards/telemetry, every 10s). collectPage now skips the expensive session scan entirely; only the full dashboard render adds the alert. The /drops page handler still calls tvDiscoveryAlert directly and is unaffected. Test: collectPage no longer includes tv_discovery alerts; full-page banner computation is verified separately via tvDiscoveryAlert function. Co-Authored-By: Claude Haiku 4.5 --- internal/api/handlers_dashboard.go | 15 +++++++++------ internal/api/tv_discovery_test.go | 30 +++++++++++++++++++----------- 2 files changed, 28 insertions(+), 17 deletions(-) diff --git a/internal/api/handlers_dashboard.go b/internal/api/handlers_dashboard.go index 64835d6..ef2f9a0 100644 --- a/internal/api/handlers_dashboard.go +++ b/internal/api/handlers_dashboard.go @@ -319,11 +319,6 @@ func (d dashboardDeps) collectPage(r *http.Request) dashPage { // Build alerts: any account in needs_auth state or sleeping with 0 // eligible drops gets a top banner pointing at the right CTA. alerts := buildDashAlerts(r.Context(), d.q, cards, lang) - // tv_discovery goes first — it's an install-wide notice, not tied to - // one account, so it leads the other per-account alerts. - if tv := tvDiscoveryAlert(r.Context(), d.q, d.sessions, lang); tv != nil { - alerts = append([]dashAlert{*tv}, alerts...) - } camps := activeCampsFromDiscovery(r.Context(), d.sch, d.channelCounters, d.q, lang) @@ -966,12 +961,20 @@ func (d dashboardDeps) page(w http.ResponseWriter, r *http.Request) { if d.sm != nil { flash = d.sm.PopString(r.Context(), "flash") } + page := d.collectPage(r) + // tv_discovery alert goes first — it's an install-wide notice, not tied to + // one account, so it leads the other per-account alerts. Only computed on + // full-page renders, not on polled HTMX partials (cards/telemetry). + lang := i18n.DetectLang(r) + if tv := tvDiscoveryAlert(r.Context(), d.q, d.sessions, lang); tv != nil { + page.Alerts = append([]dashAlert{*tv}, page.Alerts...) + } render(w, r, d.t, "dashboard.html", templateData{ AuthedAdmin: true, CSRFToken: csrfToken(r), Active: "dashboard", Flash: flash, - Page: d.collectPage(r), + Page: page, }) } diff --git a/internal/api/tv_discovery_test.go b/internal/api/tv_discovery_test.go index b4f031d..2eb6450 100644 --- a/internal/api/tv_discovery_test.go +++ b/internal/api/tv_discovery_test.go @@ -209,26 +209,34 @@ func TestDashAlerts_EmptyWhenNoAlerts(t *testing.T) { require.False(t, strings.Contains(out, `class="alerts"`)) } -// End-to-end: an enabled Twitch account with a TV session and no whitelist -// produces a dashboard page whose Alerts include the warning-variant -// tv_discovery entry (collectPage is the real dashboard view-model builder). -func TestDashboardCollectPage_TVDiscoveryBanner(t *testing.T) { +// collectPage does NOT compute the tv_discovery alert — it's only added by +// the full-page handler. This guards against the alert decryption running on +// every polled HTMX partial (cards/telemetry, every 10s). +func TestDashboardCollectPage_NoTVDiscoveryComputed(t *testing.T) { ctx, q, sessions := tvDiscoverySetup(t) createTwitchAccount(t, ctx, q, "acc_tv", true) require.NoError(t, sessions.Put(ctx, "acc_tv", platform.Session{ClientID: twitch.ClientTV})) d := dashboardDeps{q: q, sessions: sessions, start: time.Now(), loc: timeutil.NewZone(time.UTC)} req := httptest.NewRequest(http.MethodGet, "/", nil) + + // collectPage should return alerts from buildDashAlerts only, not tvDiscoveryAlert. page := d.collectPage(req) + for _, a := range page.Alerts { + require.NotEqual(t, "tv_discovery", a.Kind, "collectPage must not compute TV discovery alert") + } - require.Len(t, page.Alerts, 1) - require.Equal(t, "tv_discovery", page.Alerts[0].Kind) - require.True(t, page.Alerts[0].Warning) + // Verify tvDiscoveryAlert still works when called directly (as the page handler does). + tv := tvDiscoveryAlert(ctx, q, sessions, "en") + require.NotNil(t, tv) + require.Equal(t, "tv_discovery", tv.Kind) + require.True(t, tv.Warning) } -// An Android-only install (no TV session anywhere) must not raise the -// banner on the dashboard. -func TestDashboardCollectPage_NoBannerForAndroidOnly(t *testing.T) { +// collectPage excludes tv_discovery alert even when called with sessions present, +// since the page handler is responsible for adding it (not collectPage). This +// ensures polled partials (cards/telemetry) never trigger session decryption. +func TestDashboardCollectPage_ExcludesTVDiscoveryAlertAlways(t *testing.T) { ctx, q, sessions := tvDiscoverySetup(t) createTwitchAccount(t, ctx, q, "acc_android", true) require.NoError(t, sessions.Put(ctx, "acc_android", platform.Session{ClientID: ""})) @@ -238,7 +246,7 @@ func TestDashboardCollectPage_NoBannerForAndroidOnly(t *testing.T) { page := d.collectPage(req) for _, a := range page.Alerts { - require.NotEqual(t, "tv_discovery", a.Kind) + require.NotEqual(t, "tv_discovery", a.Kind, "collectPage must never compute TV discovery alert") } } From 6119427559d7a12f41d58c3ee07593e20fe09712 Mon Sep 17 00:00:00 2001 From: Alejandro Date: Wed, 30 Sep 2026 01:18:39 -0400 Subject: [PATCH 41/56] Update README.md --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 1813e26..50decfd 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,5 @@ -> ❗ **Drop discovery is limited.** Twitch no longer lists campaigns for new logins, so GrubDrops only finds drops for games on your whitelist. +> [!WARNING] +> **Drop discovery is limited.** Twitch no longer lists campaigns for new logins, so GrubDrops only finds drops for games on your whitelist.

GrubDrops From dfd617b2d7f637c5f0571d023d60456db7cee6d1 Mon Sep 17 00:00:00 2001 From: Alejandro Date: Wed, 30 Sep 2026 01:19:01 -0400 Subject: [PATCH 42/56] Clarify drop discovery limitations in README Updated warning message for clarity. --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 50decfd..938306d 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,6 @@ > [!WARNING] -> **Drop discovery is limited.** Twitch no longer lists campaigns for new logins, so GrubDrops only finds drops for games on your whitelist. +> **Drop discovery is limited.** +Twitch no longer lists campaigns for new logins, so GrubDrops only finds drops for games on your whitelist.

GrubDrops From 91e30e8c24d7b6778a3f63da3063e9d545fb4b39 Mon Sep 17 00:00:00 2001 From: Alejandro Date: Wed, 30 Sep 2026 01:23:45 -0400 Subject: [PATCH 43/56] fix(pipeline): PubSub claimable no longer bypasses skip and link blocks The Required==0 branch set Claimable directly, ignoring a user skip or a campaign link block. Both branches now return early for those rows. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/CHANGELOG.md | 1 + internal/pipeline/loop/batch145_test.go | 66 +++++++++++++++++++++++++ internal/pipeline/loop/loop.go | 11 ++++- 3 files changed, 77 insertions(+), 1 deletion(-) create mode 100644 internal/pipeline/loop/batch145_test.go diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 6bcbacf..6dbf327 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -45,6 +45,7 @@ All notable changes to GrubDrops. - v2 no longer watches drops from campaigns that need an unlinked account between syncs. - Channel whitelist edits on the drops page now reach v2 accounts. - The force-watch on/off switch now applies to v2 accounts. +- A Twitch claim notification no longer overrides a drop you skipped or that needs a linked account. ## [1.4.2] — 2026-09-28 diff --git a/internal/pipeline/loop/batch145_test.go b/internal/pipeline/loop/batch145_test.go new file mode 100644 index 0000000..7fff42e --- /dev/null +++ b/internal/pipeline/loop/batch145_test.go @@ -0,0 +1,66 @@ +package loop + +import ( + "context" + "testing" + "time" + + "github.com/stretchr/testify/assert" + + "github.com/aalejandrofer/grubdrops/internal/pipeline/dropstate" + "github.com/aalejandrofer/grubdrops/internal/platform" +) + +// Batch 1.4.5 item 2: a PubSub "claimable" event must not override a user +// skip or a campaign link block, even when Required is still unknown (the +// Required==0 branch used to set Claimable directly and bypass every block). +// Revert-proof: drop the block guard in onPubSub "claimable" and the +// Required==0 cases flip to Claimable. +func TestOnPubSub_ClaimableRespectsUserSkipAndUnlinked(t *testing.T) { + for _, tc := range []struct { + name string + reason dropstate.Reason + required int + }{ + {"user_skip unknown required", dropstate.UserSkip, 0}, + {"unlinked unknown required", dropstate.Unlinked, 0}, + {"user_skip known required", dropstate.UserSkip, 60}, + {"unlinked known required", dropstate.Unlinked, 60}, + } { + t.Run(tc.name, func(t *testing.T) { + st := &memStore{rows: map[string]dropstate.Row{}} + prev := dropstate.Row{AccountID: "acc", DropID: "d1", CampaignID: "c1", Status: dropstate.Blocked, Reason: tc.reason, Required: tc.required, Minutes: 5, Source: dropstate.FromUser} + l := &Loop{ + cfg: Config{AccountID: "acc", Store: st, Now: time.Now}, + rows: map[string]dropstate.Row{"d1": prev}, + progress: map[string]platform.DropProgress{}, + } + l.onPubSub(context.Background(), pubsubEvent{kind: "claimable", drop: "d1", instance: "inst1"}) + assert.Equal(t, prev, l.rows["d1"], "blocked row must be untouched") + assert.Equal(t, dropstate.Row{}, st.get("d1"), "nothing persisted") + assert.Empty(t, l.progress, "no instance recorded for a blocked drop") + }) + } +} + +// Control: a claimable event for a platform-confirmed claim does nothing, +// and one for an ordinary Accruing row with a known Required makes it +// Claimable through Apply. +func TestOnPubSub_ClaimableNormalRowBecomesClaimable(t *testing.T) { + st := &memStore{rows: map[string]dropstate.Row{}} + claimed := dropstate.Row{AccountID: "acc", DropID: "d2", CampaignID: "c1", Status: dropstate.Claimed, Source: dropstate.FromPlatform, Required: 60, Minutes: 60} + l := &Loop{ + cfg: Config{AccountID: "acc", Store: st, Now: time.Now}, + rows: map[string]dropstate.Row{ + "d1": {AccountID: "acc", DropID: "d1", CampaignID: "c1", Status: dropstate.Accruing, Required: 60, Minutes: 30, Source: dropstate.FromPlatform}, + "d2": claimed, + }, + progress: map[string]platform.DropProgress{}, + } + l.onPubSub(context.Background(), pubsubEvent{kind: "claimable", drop: "d1", instance: "inst1"}) + l.onPubSub(context.Background(), pubsubEvent{kind: "claimable", drop: "d2", instance: "inst2"}) + assert.Equal(t, dropstate.Claimable, l.rows["d1"].Status) + assert.Equal(t, 60, l.rows["d1"].Minutes) + assert.Equal(t, "inst1", l.progress["d1"].InstanceID) + assert.Equal(t, claimed, l.rows["d2"]) +} diff --git a/internal/pipeline/loop/loop.go b/internal/pipeline/loop/loop.go index 3b57f4c..1ed7bee 100644 --- a/internal/pipeline/loop/loop.go +++ b/internal/pipeline/loop/loop.go @@ -640,7 +640,10 @@ func (l *Loop) onPubSub(ctx context.Context, e pubsubEvent) { l.applyProgress(ctx, []platform.Progress{{BenefitID: e.drop, MinutesWatched: e.minutes}}) case "claimable": r, ok := l.rows[e.drop] - if !ok || r.Status == dropstate.Claimed { + if !ok || r.Status == dropstate.Claimed || userOrLinkBlocked(r) { + // A skip or a campaign link block is lifted only by the user or + // the reconciler, never by a claim notification: the Required==0 + // branch below would otherwise bypass it entirely. return } dp := l.progress[e.drop] @@ -671,6 +674,12 @@ func (l *Loop) onPubSub(ctx context.Context, e pubsubEvent) { } } +// userOrLinkBlocked reports a row blocked by a user skip or a campaign link +// block, the two blocks only the user or the reconciler may lift. +func userOrLinkBlocked(r dropstate.Row) bool { + return r.Status == dropstate.Blocked && (r.Reason == dropstate.UserSkip || r.Reason == dropstate.Unlinked) +} + func (l *Loop) maybeNotifyProgress(ctx context.Context, r dropstate.Row) { step := l.cfg.ProgressNotifyStepPct if step <= 0 || r.Required <= 0 || l.cfg.Notifier == nil { From 397ce07f8406fe503150ef07de244dabf840b799 Mon Sep 17 00:00:00 2001 From: Alejandro Date: Wed, 30 Sep 2026 01:24:57 -0400 Subject: [PATCH 44/56] fix(pipeline): re-check current channel before swapping off it Directory pages are top-N and churn; a refresh that omitted the channel being mined made v2 swap away and back every LiveEvery. refreshLive now re-probes just that channel for the campaigns it serves and keeps it when still live. No prober: unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/CHANGELOG.md | 1 + internal/pipeline/loop/batch145_test.go | 110 ++++++++++++++++++++++++ internal/pipeline/loop/loop.go | 60 +++++++++++++ 3 files changed, 171 insertions(+) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 6dbf327..537729a 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -46,6 +46,7 @@ All notable changes to GrubDrops. - Channel whitelist edits on the drops page now reach v2 accounts. - The force-watch on/off switch now applies to v2 accounts. - A Twitch claim notification no longer overrides a drop you skipped or that needs a linked account. +- v2 no longer bounces between two channels every few minutes when a channel drops out of Twitch's directory list. ## [1.4.2] — 2026-09-28 diff --git a/internal/pipeline/loop/batch145_test.go b/internal/pipeline/loop/batch145_test.go index 7fff42e..a39ed08 100644 --- a/internal/pipeline/loop/batch145_test.go +++ b/internal/pipeline/loop/batch145_test.go @@ -2,10 +2,13 @@ package loop import ( "context" + "strings" + "sync" "testing" "time" "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" "github.com/aalejandrofer/grubdrops/internal/pipeline/dropstate" "github.com/aalejandrofer/grubdrops/internal/platform" @@ -64,3 +67,110 @@ func TestOnPubSub_ClaimableNormalRowBecomesClaimable(t *testing.T) { assert.Equal(t, "inst1", l.progress["d1"].InstanceID) assert.Equal(t, claimed, l.rows["d2"]) } + +// liveProber is a ChannelProber whose answer is scripted per login. +type liveProber struct { + *fakeBackend + pmu sync.Mutex + isLive map[string]bool + calls [][]string // logins per call +} + +func (p *liveProber) ProbeChannels(_ context.Context, _ platform.Session, _ platform.Campaign, logins []string) ([]platform.Stream, error) { + p.pmu.Lock() + defer p.pmu.Unlock() + p.calls = append(p.calls, append([]string(nil), logins...)) + var out []platform.Stream + for _, lg := range logins { + if p.isLive[strings.ToLower(lg)] { + out = append(out, platform.Stream{Channel: lg, ViewerCount: 3}) + } + } + return out, nil +} + +func (f *fakeBackend) setLive(s ...platform.Stream) { + f.mu.Lock() + defer f.mu.Unlock() + f.live = s +} + +// flipSetup lands the loop on ch1 (c1), then makes the next directory page +// omit ch1 (top-N churn) while ch2 stays listed. +func flipSetup(t *testing.T, backend platform.Backend, f *fakeBackend, cfg Config) (*Loop, context.Context) { + t.Helper() + cfg.Backend = backend + l, err := New(cfg) + require.NoError(t, err) + ctx := context.Background() + require.NoError(t, l.load(ctx)) + l.reconcile(ctx) + l.refreshLive(ctx) + l.replan(ctx) + t.Cleanup(l.stopSession) + require.Equal(t, "ch1", l.current.Channel.Channel) + require.Eventually(t, func() bool { return len(f.watchedChannels()) == 1 }, 2*time.Second, 5*time.Millisecond) + f.setLive(platform.Stream{Channel: "ch2", ViewerCount: 5}) + return l, ctx +} + +// Batch 1.4.5 item 3: the current channel dropping out of a fresh +// ListEligibleChannels page is re-checked with the prober; still live means +// no swap. Revert-proof: remove the refreshLive re-check and v2 swaps to ch2. +func TestLoop_RefreshKeepsCurrentChannelWhenProberSaysLive(t *testing.T) { + f, _, _, cfg := setup(t) + p := &liveProber{fakeBackend: f, isLive: map[string]bool{"ch1": true}} + l, ctx := flipSetup(t, p, f, cfg) + + l.refreshLive(ctx) + l.replan(ctx) + time.Sleep(30 * time.Millisecond) + + assert.Equal(t, "ch1", l.current.Channel.Channel, "no swap while the current channel is still live") + assert.Equal(t, "ch1", l.Snapshot().Channel) + assert.Equal(t, []string{"ch1"}, f.watchedChannels(), "StartWatch count unchanged") + p.pmu.Lock() + assert.Equal(t, [][]string{{"ch1"}}, p.calls, "exactly one re-check, for the current channel only") + p.pmu.Unlock() +} + +// The prober reporting the current channel offline lets the swap happen. +func TestLoop_RefreshSwapsWhenProberSaysCurrentOffline(t *testing.T) { + f, _, _, cfg := setup(t) + p := &liveProber{fakeBackend: f, isLive: map[string]bool{}} + l, ctx := flipSetup(t, p, f, cfg) + + l.refreshLive(ctx) + l.replan(ctx) + + assert.Equal(t, "ch2", l.current.Channel.Channel) + require.Eventually(t, func() bool { return len(f.watchedChannels()) == 2 }, 2*time.Second, 5*time.Millisecond) + assert.Equal(t, []string{"ch1", "ch2"}, f.watchedChannels()) +} + +// No prober: behaviour unchanged, the refresh that omits ch1 swaps to ch2. +func TestLoop_RefreshWithoutProberSwapsAsBefore(t *testing.T) { + f, _, _, cfg := setup(t) + l, ctx := flipSetup(t, f, f, cfg) + + l.refreshLive(ctx) + l.replan(ctx) + + assert.Equal(t, "ch2", l.current.Channel.Channel) +} + +// A current channel the fresh page still lists is not re-probed. +func TestLoop_RefreshDoesNotReprobeListedCurrentChannel(t *testing.T) { + f, _, _, cfg := setup(t) + p := &liveProber{fakeBackend: f, isLive: map[string]bool{"ch1": true}} + l, ctx := flipSetup(t, p, f, cfg) + f.setLive(platform.Stream{Channel: "ch1", ViewerCount: 10}, platform.Stream{Channel: "ch2", ViewerCount: 5}) + + l.refreshLive(ctx) + l.replan(ctx) + + assert.Equal(t, "ch1", l.current.Channel.Channel) + p.pmu.Lock() + assert.Empty(t, p.calls) + p.pmu.Unlock() +} diff --git a/internal/pipeline/loop/loop.go b/internal/pipeline/loop/loop.go index 1ed7bee..d1d4797 100644 --- a/internal/pipeline/loop/loop.go +++ b/internal/pipeline/loop/loop.go @@ -527,6 +527,7 @@ func (l *Loop) refreshLive(ctx context.Context) { } live[c.ID] = streams } + l.keepCurrentLive(ctx, live) l.live = live l.forceLive = nil if l.prober != nil && len(l.cfg.ForceWatch) > 0 { @@ -536,6 +537,65 @@ func (l *Loop) refreshLive(ctx context.Context) { } } +// keepCurrentLive re-checks the channel being mined when a fresh directory +// page omits it for every campaign it serves. Directory pages are top-N and +// churn, so a still-live channel can drop out of one refresh; without this +// the planner reads it as gone, swaps away, and swaps back on the next +// refresh that lists it again (a flip-flop every LiveEvery). One prober call +// per served campaign, only in that case; without a prober nothing changes. +func (l *Loop) keepCurrentLive(ctx context.Context, live map[string][]platform.Stream) { + d := l.current + if l.prober == nil || d == nil || d.Kind != planner.Mine || d.Channel.Channel == "" { + return + } + ch := d.Channel.Channel + var campIDs []string + seen := map[string]bool{} + add := func(id string) { + if id != "" && !seen[id] { + seen[id] = true + campIDs = append(campIDs, id) + } + } + add(d.CampaignID) + for _, id := range d.Serves { + add(l.rows[id].CampaignID) + } + for _, id := range campIDs { + for _, s := range live[id] { + if strings.EqualFold(s.Channel, ch) { + return // still listed for a campaign it serves + } + } + } + for _, id := range campIDs { + c, ok := l.campaign(id) + if !ok { + continue + } + ps, err := l.prober.ProbeChannels(ctx, l.cfg.Session, c, []string{ch}) + if err != nil { + slog.Debug("pipeline: current channel re-check failed", "account", l.cfg.AccountID, "campaign", id, "channel", ch, "err", err) + continue + } + for _, s := range ps { + if strings.EqualFold(s.Channel, ch) { + live[id] = mergeStreams(live[id], []platform.Stream{s}) + break + } + } + } +} + +func (l *Loop) campaign(id string) (platform.Campaign, bool) { + for _, c := range l.camps { + if c.ID == id { + return c, true + } + } + return platform.Campaign{}, false +} + func (l *Loop) isCurrent(channel string) bool { return l.current != nil && l.current.Kind != planner.Idle && strings.EqualFold(l.current.Channel.Channel, channel) } From 1a515a7ec722a8449fcafc7af5d496f7431e9032 Mon Sep 17 00:00:00 2001 From: Alejandro Date: Wed, 30 Sep 2026 01:28:25 -0400 Subject: [PATCH 45/56] feat(pipeline): catch-up claim probe for drops Twitch can't vouch for An account added after finishing its drops has every row Eligible with no platform record, and v2 only learned they were claimed via two stalls per drop. After each reconcile, queue up to 5 such drops (active in-scope campaign, Required>0, Known=false, unprobed this run) and send one claim each, one every 3s off a loop timer so the loop stays responsive. OK/Already -> claimed, NeedsLink -> needs_link, Failed -> row unchanged. Rename StallClaimProbe to ClaimProbe (gates both; still Twitch-only in main.go). Co-Authored-By: Claude Opus 5.5 (1M context) --- cmd/miner/main.go | 2 +- docs/CHANGELOG.md | 1 + internal/pipeline/loop/batch145_test.go | 151 ++++++++++++++++++++++++ internal/pipeline/loop/loop.go | 131 ++++++++++++++++++-- internal/pipeline/loop/loop_test.go | 19 ++- 5 files changed, 293 insertions(+), 11 deletions(-) diff --git a/cmd/miner/main.go b/cmd/miner/main.go index df88e3b..7401f66 100644 --- a/cmd/miner/main.go +++ b/cmd/miner/main.go @@ -498,7 +498,7 @@ func run() error { ProgressNotifyStepPct: progressStep, // Twitch hides drops claimed outside the app once their // campaign leaves the Inventory; Kick keeps them listed. - StallClaimProbe: a.Platform == "twitch", + ClaimProbe: a.Platform == "twitch", }) if err == nil { logger.Info("pipeline v2 enabled for account", "account", a.ID, "platform", a.Platform) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 537729a..88057eb 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -45,6 +45,7 @@ All notable changes to GrubDrops. - v2 no longer watches drops from campaigns that need an unlinked account between syncs. - Channel whitelist edits on the drops page now reach v2 accounts. - The force-watch on/off switch now applies to v2 accounts. +- A newly added Twitch account that already finished its drops is recognised within a few syncs instead of being watched drop by drop. - A Twitch claim notification no longer overrides a drop you skipped or that needs a linked account. - v2 no longer bounces between two channels every few minutes when a channel drops out of Twitch's directory list. diff --git a/internal/pipeline/loop/batch145_test.go b/internal/pipeline/loop/batch145_test.go index a39ed08..d04756c 100644 --- a/internal/pipeline/loop/batch145_test.go +++ b/internal/pipeline/loop/batch145_test.go @@ -2,6 +2,7 @@ package loop import ( "context" + "fmt" "strings" "sync" "testing" @@ -174,3 +175,153 @@ func TestLoop_RefreshDoesNotReprobeListedCurrentChannel(t *testing.T) { assert.Empty(t, p.calls) p.pmu.Unlock() } + +// catchUpSetup scripts a Twitch account added after it finished its drops: +// campaign c1 is active and in scope with n drops, and the platform reports +// nothing for any of them (campaign left the Inventory, details self null), +// so reconcile derives every row Eligible from a Known=false observation. +func catchUpSetup(t *testing.T, n int) (*fakeBackend, *memStore, Config) { + f, st, _, cfg := setup(t) + var bs []platform.DropBenefit + for i := 1; i <= n; i++ { + id := fmt.Sprintf("d%d", i) + bs = append(bs, platform.DropBenefit{ID: id, CampaignID: "c1", Name: id, RequiredMinutes: 60 * i}) + } + f.camps[0].Benefits = bs + f.progress = nil + f.inventory = nil + f.live = nil // nothing live: any claim below comes from catch-up, never a watch + f.claimRes = platform.ClaimResult{Outcome: platform.ClaimFailed} + cfg.ClaimProbe = true + cfg.CatchUpEvery = 2 * time.Millisecond + return f, st, cfg +} + +func (f *fakeBackend) claimedIDsCopy() []string { + f.mu.Lock() + defer f.mu.Unlock() + return append([]string(nil), f.claimedIDs...) +} + +func (f *fakeBackend) listCount() int { + f.mu.Lock() + defer f.mu.Unlock() + return len(f.listGames) +} + +// Batch 1.4.5 item 1: a new account whose drops are already done. Already +// → Claimed/FromPlatform without any watch; Failed leaves the row Eligible +// and is not re-probed on the next reconcile. Revert-proof: remove the +// catch-up enqueue in reconcile and no claim is ever sent (nothing is live, +// so the stall probe never runs either). +func TestLoop_CatchUpClaimProbe_NewAccountAlreadyClaimed(t *testing.T) { + f, st, cfg := catchUpSetup(t, 3) + f.claimByDrop = map[string]platform.ClaimResult{ + "d1": {Outcome: platform.ClaimAlready}, + "d2": {Outcome: platform.ClaimAlready}, + "d3": {Outcome: platform.ClaimFailed}, + } + l, _ := run(t, cfg) + require.Eventually(t, func() bool { + return st.get("d1").Status == dropstate.Claimed && st.get("d2").Status == dropstate.Claimed && f.claimCount() == 3 + }, 2*time.Second, 2*time.Millisecond) + for _, id := range []string{"d1", "d2"} { + assert.Equal(t, dropstate.FromPlatform, st.get(id).Source, id) + } + r3 := st.get("d3") + assert.Equal(t, dropstate.Eligible, r3.Status, "a failed catch-up probe leaves the row unchanged") + assert.Equal(t, dropstate.NoReason, r3.Reason) + assert.Equal(t, 0, r3.FailCount, "a catch-up probe is not a claim failure") + assert.Empty(t, f.watchedChannels(), "claimed without any watch") + + before := f.listCount() + l.Nudge() + require.Eventually(t, func() bool { return f.listCount() > before }, 2*time.Second, 2*time.Millisecond) + time.Sleep(40 * time.Millisecond) // many CatchUpEvery periods + assert.ElementsMatch(t, []string{"d1", "d2", "d3"}, f.claimedIDsCopy(), "each drop probed once per loop run") + assert.Equal(t, dropstate.Eligible, st.get("d3").Status) +} + +// NeedsLink maps to the needs_link block. +func TestLoop_CatchUpClaimProbe_NeedsLinkBlocks(t *testing.T) { + f, st, cfg := catchUpSetup(t, 1) + f.claimRes = platform.ClaimResult{Outcome: platform.ClaimNeedsLink} + run(t, cfg) + require.Eventually(t, func() bool { return st.get("d1").Reason == dropstate.NeedsLink }, 2*time.Second, 2*time.Millisecond) + assert.Equal(t, dropstate.Blocked, st.get("d1").Status) +} + +// Cap: with 8 candidates one reconcile queues only catchUpPerReconcile; the +// rest wait for the next reconcile, and nothing is probed twice. Driven by +// hand so the drain is deterministic. +func TestLoop_CatchUpClaimProbe_CapPerReconcile(t *testing.T) { + f, _, cfg := catchUpSetup(t, 8) + l, err := New(cfg) + require.NoError(t, err) + ctx := context.Background() + require.NoError(t, l.load(ctx)) + drain := func() { + for i := 0; i < 20; i++ { + l.catchUpOne(ctx) + } + } + + l.reconcile(ctx) + assert.Equal(t, 0, f.claimCount(), "reconcile only queues; probes are spaced by the loop timer") + drain() + assert.Equal(t, catchUpPerReconcile, f.claimCount()) + + l.reconcile(ctx) + drain() + assert.Equal(t, 8, f.claimCount(), "the remaining candidates go on the next reconcile") + + l.reconcile(ctx) + drain() + assert.Equal(t, 8, f.claimCount(), "no drop is probed twice") + assert.ElementsMatch(t, []string{"d1", "d2", "d3", "d4", "d5", "d6", "d7", "d8"}, f.claimedIDsCopy()) +} + +// Only Eligible rows with a Known=false observation, Required>0 and an +// active campaign qualify. +func TestLoop_CatchUpClaimProbe_SkipsNonCandidates(t *testing.T) { + f, _, cfg := catchUpSetup(t, 3) + f.progress = []platform.DropProgress{{DropID: "d1", CampaignID: "c1", Minutes: 0, Required: 60, Known: true}} + f.camps = append(f.camps, platform.Campaign{ID: "c2", Platform: "twitch", Game: "G", Name: "Old", Status: "expired", + AccountLinked: true, AccountLinkChecked: true, + Benefits: []platform.DropBenefit{{ID: "e1", CampaignID: "c2", RequiredMinutes: 60}}}) + l, err := New(cfg) + require.NoError(t, err) + ctx := context.Background() + require.NoError(t, l.load(ctx)) + r := dropstate.Row{AccountID: "acc", DropID: "d2", CampaignID: "c1", Platform: "twitch", Status: dropstate.Accruing, Minutes: 5, Required: 120, Source: dropstate.FromPlatform} + l.rows["d2"] = r + l.reconcile(ctx) + for i := 0; i < 10; i++ { + l.catchUpOne(ctx) + } + assert.Equal(t, []string{"d3"}, f.claimedIDsCopy(), "d1 is Known, d2 is not Eligible (Accruing then not_enrolled), e1 is in an expired campaign") +} + +// Probes are not sent inline: with an hour between probes, none is sent. +func TestLoop_CatchUpClaimProbe_SpacedNotInline(t *testing.T) { + f, _, cfg := catchUpSetup(t, 3) + cfg.CatchUpEvery = time.Hour + run(t, cfg) + require.Eventually(t, func() bool { return f.listCount() >= 1 }, 2*time.Second, 2*time.Millisecond) + time.Sleep(40 * time.Millisecond) + assert.Equal(t, 0, f.claimCount()) +} + +// Kick keeps claimed rewards listed, so main.go leaves ClaimProbe off: no +// catch-up probes. +func TestLoop_CatchUpClaimProbe_KickOff(t *testing.T) { + f, st, cfg := catchUpSetup(t, 3) + cfg.Platform = "kick" + cfg.ClaimProbe = false + f.claimRes = platform.ClaimResult{Outcome: platform.ClaimAlready} + run(t, cfg) + require.Eventually(t, func() bool { return f.listCount() >= 1 }, 2*time.Second, 2*time.Millisecond) + time.Sleep(40 * time.Millisecond) + assert.Equal(t, 0, f.claimCount()) + assert.Equal(t, dropstate.Eligible, st.get("d1").Status) +} diff --git a/internal/pipeline/loop/loop.go b/internal/pipeline/loop/loop.go index d1d4797..94df8df 100644 --- a/internal/pipeline/loop/loop.go +++ b/internal/pipeline/loop/loop.go @@ -59,12 +59,20 @@ type Config struct { StreamerPriority []string ForceWatch []string ProgressNotifyStepPct int - // StallClaimProbe (Twitch): on a stall, send one claim for each served - // Eligible drop that was absent from the last inventory read. Twitch - // drops claimed outside the app whose campaign left the Inventory are - // otherwise invisible (DropCampaignDetails self: null) and would be - // watched forever. Kick keeps claimed rewards listed, so it leaves this off. - StallClaimProbe bool + // ClaimProbe (Twitch) turns on both blind claim checks for drops the + // platform says nothing about. Twitch drops claimed outside the app whose + // campaign left the Inventory are otherwise invisible (DropCampaignDetails + // self: null) and would be watched forever. Kick keeps claimed rewards + // listed, so it leaves this off. + // - catch-up: after each reconcile, one claim per Eligible drop of an + // active campaign that the reconcile read had no record of, once per + // loop run, spaced CatchUpEvery apart (catchUpPerReconcile queued per + // reconcile). Recognises an account added after it finished its drops. + // - stall: on a stall, one claim for each served Eligible drop absent + // from the last inventory read, on that drop's second stall. + ClaimProbe bool + // CatchUpEvery spaces catch-up claim probes (default 3s). + CatchUpEvery time.Duration Now func() time.Time ReconcileEvery time.Duration // default 15m @@ -78,6 +86,14 @@ type Config struct { // maxLiveCampaigns bounds channel lookups per refresh. const maxLiveCampaigns = 8 +// catchUpPerReconcile bounds the catch-up claim-probe queue: a reconcile +// tops it up to this many, and the loop drains one every CatchUpEvery. +// Probes never run inline in reconcile, so the loop stays responsive (at +// most one claim call per timer fire) and the rate stays at or below one +// claim per CatchUpEvery (3s) and catchUpPerReconcile (5, well under 20) +// per reconcile. Candidates past the cap wait for the next reconcile. +const catchUpPerReconcile = 5 + type pubsubEvent struct { kind string // "progress" | "claimable" | "down" drop string @@ -126,6 +142,13 @@ type Loop struct { // (Eligible, absent from lastInv). The probe fires on the second stall; // an entry is dropped once probed or once the row leaves Eligible. stallCount map[string]int + // Catch-up claim probes (ClaimProbe). probed holds every drop id probed + // this loop run (a restart may probe once more); catchUpQ is the pending + // queue, drained one per catchUpC fire (nil while no timer is armed). + probed map[string]bool + catchUpQ []string + catchUpT *time.Timer + catchUpC <-chan time.Time } // New assembles the loop, checking the backend's pipeline capabilities once. @@ -159,6 +182,9 @@ func New(cfg Config) (*Loop, error) { if cfg.DownCooldown <= 0 { cfg.DownCooldown = 2 * time.Minute } + if cfg.CatchUpEvery <= 0 { + cfg.CatchUpEvery = 3 * time.Second + } cfg.Session.AccountID = cfg.AccountID if cfg.Session.Games == nil { cfg.Session.Games = cfg.Games @@ -178,6 +204,7 @@ func New(cfg Config) (*Loop, error) { milestones: map[string]int{}, pendingHistory: map[string]bool{}, stallCount: map[string]int{}, + probed: map[string]bool{}, } l.prober, _ = cfg.Backend.(platform.ChannelProber) l.subs, _ = cfg.Backend.(platform.ChannelSubscriber) @@ -250,6 +277,11 @@ func (l *Loop) Run(ctx context.Context) error { liveT := time.NewTicker(l.cfg.LiveEvery) defer recT.Stop() defer liveT.Stop() + defer func() { + if l.catchUpT != nil { + l.catchUpT.Stop() + } + }() for { select { case <-ctx.Done(): @@ -265,6 +297,10 @@ func (l *Loop) Run(ctx context.Context) error { l.onSession(ctx, ev) case pe := <-l.pubsub: l.onPubSub(ctx, pe) + case <-l.catchUpC: + l.catchUpC, l.catchUpT = nil, nil + l.catchUpOne(ctx) + l.armCatchUp() } if l.authBlocked { return nil @@ -363,6 +399,87 @@ func (l *Loop) reconcile(ctx context.Context) { // already-existing row is left to dropstate.Apply's own rules so a // definite platform answer can overwrite a stale user mark and stick. l.bridgeClaimHistory(ctx, func(id string) bool { return !existed[id] }) + if l.cfg.ClaimProbe { + l.queueCatchUp(res) + } +} + +// queueCatchUp tops the catch-up queue up to catchUpPerReconcile with drops +// this reconcile left Eligible although the platform had no record of them +// (Known=false: absent from Inventory and campaign details), in an active +// in-scope campaign, with a known requirement, not yet probed this run. On +// Twitch that is what an account added after finishing its drops looks +// like; without this each one needs two stalls to be recognised. +func (l *Loop) queueCatchUp(res reconcile.Result) { + queued := make(map[string]bool, len(l.catchUpQ)) + for _, id := range l.catchUpQ { + queued[id] = true + } + for _, c := range res.Campaigns { + for _, b := range c.Benefits { + if len(l.catchUpQ) >= catchUpPerReconcile { + l.armCatchUp() + return + } + id := b.ID + if id == "" || queued[id] || l.probed[id] || res.Progress[id].Known { + continue + } + r, ok := l.rows[id] + if !ok || r.CampaignID != c.ID || r.Status != dropstate.Eligible || r.Required <= 0 { + continue + } + queued[id] = true + l.catchUpQ = append(l.catchUpQ, id) + } + } + l.armCatchUp() +} + +// armCatchUp starts the catch-up timer when work is queued and none runs. +func (l *Loop) armCatchUp() { + if len(l.catchUpQ) == 0 || l.catchUpT != nil { + return + } + l.catchUpT = time.NewTimer(l.cfg.CatchUpEvery) + l.catchUpC = l.catchUpT.C +} + +// catchUpOne sends one catch-up claim probe: the first queued drop still +// Eligible and unprobed. OK/ALREADY_CLAIMED marks it claimed; NeedsLink +// blocks it needs_link; a failure leaves the row untouched (it may simply +// be unstarted) and counts nothing. Either way it is not probed again this +// loop run. +func (l *Loop) catchUpOne(ctx context.Context) { + for len(l.catchUpQ) > 0 { + id := l.catchUpQ[0] + l.catchUpQ = l.catchUpQ[1:] + r, ok := l.rows[id] + if !ok || r.Status != dropstate.Eligible || l.probed[id] { + continue + } + l.probed[id] = true + dp, ok := l.progress[id] + if !ok || dp.DropID == "" { + dp = platform.DropProgress{DropID: id, CampaignID: r.CampaignID} + } + res := l.claimer.ClaimDrop(ctx, l.cfg.Session, dp) + now := l.cfg.Now() + kind := "probe" + var next dropstate.Row + switch res.Outcome { + case platform.ClaimOK, platform.ClaimAlready: + next, kind = dropstate.ClaimOK(r, now), "claim" + case platform.ClaimNeedsLink: + next = dropstate.ClaimNeedsLink(r, now) + } + slog.Info("pipeline catch-up claim-probe", "kind", kind, "account", l.cfg.AccountID, "drop", id, + "outcome", res.Outcome.String(), "detail", res.Detail) + if !next.IsZero() { + l.commit(ctx, next) + } + return + } } // onIntegrityBlocked mirrors v1: stop watching, surface auth_required so the @@ -619,7 +736,7 @@ func (l *Loop) onSession(ctx context.Context, ev session.Event) { } case session.Stalled: if l.isCurrent(ev.Channel) { - if l.cfg.StallClaimProbe { + if l.cfg.ClaimProbe { l.stallClaimProbe(ctx, now) } l.cooldowns[strings.ToLower(ev.Channel)] = now.Add(l.cfg.StallCooldown) diff --git a/internal/pipeline/loop/loop_test.go b/internal/pipeline/loop/loop_test.go index e72a659..83a9f6c 100644 --- a/internal/pipeline/loop/loop_test.go +++ b/internal/pipeline/loop/loop_test.go @@ -41,6 +41,10 @@ type fakeBackend struct { listGames [][]string // claimWatchLen is len(watching) at the first ClaimDrop call. claimWatchLen int + // claimByDrop overrides claimRes per drop id; claimedIDs records every + // ClaimDrop call's drop id in order. + claimByDrop map[string]platform.ClaimResult + claimedIDs []string } func (f *fakeBackend) ListActiveCampaigns(_ context.Context, s platform.Session) ([]platform.Campaign, error) { @@ -84,10 +88,14 @@ func (f *fakeBackend) StopWatch(context.Context, platform.WatchHandle) error { f.stops++ return nil } -func (f *fakeBackend) ClaimDrop(context.Context, platform.Session, platform.DropProgress) platform.ClaimResult { +func (f *fakeBackend) ClaimDrop(_ context.Context, _ platform.Session, dp platform.DropProgress) platform.ClaimResult { f.mu.Lock() defer f.mu.Unlock() f.claims++ + f.claimedIDs = append(f.claimedIDs, dp.DropID) + if r, ok := f.claimByDrop[dp.DropID]; ok { + return r + } if f.claims == 1 { f.claimWatchLen = len(f.watching) } @@ -776,7 +784,12 @@ func probeSetup(t *testing.T) (*fakeBackend, *memStore, *memHistory, Config) { f.progress = nil f.inventory = nil cfg.StallPolls = 3 - cfg.StallClaimProbe = true + cfg.ClaimProbe = true + // These tests isolate the stall probe: ClaimProbe also enables the + // catch-up pass (batch145_test.go), which would claim d1 one + // CatchUpEvery (default 3s) after the first reconcile. An hour keeps it + // out explicitly instead of relying on the tests finishing within 3s. + cfg.CatchUpEvery = time.Hour return f, st, h, cfg } @@ -913,7 +926,7 @@ func TestLoop_StallClaimProbe_SkipsDropInInventory(t *testing.T) { // (d) Probe disabled (Kick): stall behaviour unchanged, no claim. func TestLoop_StallClaimProbe_DisabledNoClaim(t *testing.T) { f, st, _, cfg := probeSetup(t) - cfg.StallClaimProbe = false + cfg.ClaimProbe = false l, _ := run(t, cfg) waitBothStalled(t, f, l) assert.Equal(t, 0, f.claimCount()) From d4822899762b62858c301a6bb27f26a3c3b843e1 Mon Sep 17 00:00:00 2001 From: Alejandro Date: Wed, 30 Sep 2026 01:33:00 -0400 Subject: [PATCH 46/56] fix(pipeline): no claim notification for probe-found historical claims Co-Authored-By: Claude Opus 5.5 (1M context) --- internal/pipeline/loop/batch145_test.go | 25 ++++++++++++++++ internal/pipeline/loop/loop.go | 38 +++++++++++++++++++------ internal/pipeline/loop/loop_test.go | 29 +++++++++++++++++-- 3 files changed, 81 insertions(+), 11 deletions(-) diff --git a/internal/pipeline/loop/batch145_test.go b/internal/pipeline/loop/batch145_test.go index d04756c..deaf98b 100644 --- a/internal/pipeline/loop/batch145_test.go +++ b/internal/pipeline/loop/batch145_test.go @@ -242,6 +242,31 @@ func TestLoop_CatchUpClaimProbe_NewAccountAlreadyClaimed(t *testing.T) { assert.Equal(t, dropstate.Eligible, st.get("d3").Status) } +// Fix round 1 ruling: an ALREADY_CLAIMED catch-up answer is a historical +// claim (recorded in history, no notification); only a claim that succeeded +// just now (OK) notifies. Revert-proof: make commitProbeResult always call +// commit and the Already count becomes 1. +func TestLoop_CatchUpClaimProbe_NotifiesOnlyOK(t *testing.T) { + f, st, cfg := catchUpSetup(t, 2) + f.claimByDrop = map[string]platform.ClaimResult{ + "d1": {Outcome: platform.ClaimAlready}, + "d2": {Outcome: platform.ClaimOK}, + } + h := &memHistory{} + cfg.History = h + n := &recNotifier{} + cfg.Notifier = n + run(t, cfg) + require.Eventually(t, func() bool { + return st.get("d1").Status == dropstate.Claimed && st.get("d2").Status == dropstate.Claimed + }, 2*time.Second, 2*time.Millisecond) + time.Sleep(20 * time.Millisecond) + assert.Equal(t, 1, n.count("claim"), "only the OK claim notifies") + h.mu.Lock() + assert.ElementsMatch(t, []string{"d1", "d2"}, h.recorded, "both claims still reach the claim history") + h.mu.Unlock() +} + // NeedsLink maps to the needs_link block. func TestLoop_CatchUpClaimProbe_NeedsLinkBlocks(t *testing.T) { f, st, cfg := catchUpSetup(t, 1) diff --git a/internal/pipeline/loop/loop.go b/internal/pipeline/loop/loop.go index 94df8df..26aeec7 100644 --- a/internal/pipeline/loop/loop.go +++ b/internal/pipeline/loop/loop.go @@ -262,6 +262,12 @@ func (l *Loop) Run(ctx context.Context) error { return err } defer l.stopSession() + // Registered before the first reconcile, which may arm the timer. + defer func() { + if l.catchUpT != nil { + l.catchUpT.Stop() + } + }() l.reconcile(ctx) if l.authBlocked { return nil @@ -277,11 +283,6 @@ func (l *Loop) Run(ctx context.Context) error { liveT := time.NewTicker(l.cfg.LiveEvery) defer recT.Stop() defer liveT.Stop() - defer func() { - if l.catchUpT != nil { - l.catchUpT.Stop() - } - }() for { select { case <-ctx.Done(): @@ -476,7 +477,7 @@ func (l *Loop) catchUpOne(ctx context.Context) { slog.Info("pipeline catch-up claim-probe", "kind", kind, "account", l.cfg.AccountID, "drop", id, "outcome", res.Outcome.String(), "detail", res.Detail) if !next.IsZero() { - l.commit(ctx, next) + l.commitProbeResult(ctx, next, res.Outcome) } return } @@ -510,7 +511,23 @@ func (l *Loop) benefit(dropID string) (platform.DropBenefit, platform.Campaign) } // commit stores a row and handles the transition into claimed. -func (l *Loop) commit(ctx context.Context, next dropstate.Row) { +func (l *Loop) commit(ctx context.Context, next dropstate.Row) { l.commitRow(ctx, next, true) } + +// commitQuiet is commit without the claim notification, for historical +// claims a blind probe uncovers (ClaimAlready): the claim history is still +// recorded, but nothing was claimed just now, so nothing is announced. +func (l *Loop) commitQuiet(ctx context.Context, next dropstate.Row) { l.commitRow(ctx, next, false) } + +// commitProbeResult commits a probe outcome row, quietly for ClaimAlready. +func (l *Loop) commitProbeResult(ctx context.Context, next dropstate.Row, o platform.ClaimOutcome) { + if o == platform.ClaimAlready { + l.commitQuiet(ctx, next) + return + } + l.commit(ctx, next) +} + +func (l *Loop) commitRow(ctx context.Context, next dropstate.Row, notify bool) { prev := l.rows[next.DropID] l.rows[next.DropID] = next if next.Status != dropstate.Eligible { @@ -531,7 +548,7 @@ func (l *Loop) commit(ctx context.Context, next dropstate.Row) { } slog.Info("pipeline drop claimed", "kind", "claim", "account", l.cfg.AccountID, "drop", next.DropID, "source", string(next.Source)) // Only notify live transitions, not historical claims seen on first sync. - if !prev.IsZero() && next.Source == dropstate.FromPlatform { + if notify && !prev.IsZero() && next.Source == dropstate.FromPlatform { l.notify(ctx, "claim", c, b, nil) } } @@ -685,6 +702,9 @@ func (l *Loop) keepCurrentLive(ctx context.Context, live map[string][]platform.S } } } + // A restricted campaign with an unknown allow-list is re-probed too (the + // probe filters by game only); the exposure is bounded by the stall + // detector, which cools a channel that stops crediting. for _, id := range campIDs { c, ok := l.campaign(id) if !ok { @@ -783,7 +803,7 @@ func (l *Loop) stallClaimProbe(ctx context.Context, now time.Time) { } slog.Info("pipeline stall claim-probe", "kind", kind, "account", l.cfg.AccountID, "drop", id, "outcome", res.Outcome.String(), "detail", res.Detail) - l.commit(ctx, next) + l.commitProbeResult(ctx, next, res.Outcome) } } diff --git a/internal/pipeline/loop/loop_test.go b/internal/pipeline/loop/loop_test.go index 83a9f6c..c261fa0 100644 --- a/internal/pipeline/loop/loop_test.go +++ b/internal/pipeline/loop/loop_test.go @@ -546,6 +546,18 @@ func (n *recNotifier) Notify(_ context.Context, event string, _ map[string]any) return nil } +func (n *recNotifier) count(event string) int { + n.mu.Lock() + defer n.mu.Unlock() + c := 0 + for _, e := range n.events { + if e == event { + c++ + } + } + return c +} + // TestMaybeNotifyProgress_MilestoneNeverLowers covers item 5: a later, // lower reading of Minutes (e.g. a stale re-sync) must not lower the // recorded milestone nor re-fire a notification once real progress passes it. @@ -809,7 +821,8 @@ func (f *fakeBackend) watchedChannels() []string { // (a) An Eligible served drop absent from the inventory is probed on its // SECOND stall (ch1 stalls: plain cooldown; ch2 stalls: one claim). // ALREADY_CLAIMED marks it claimed from the platform, records history, -// notifies once, and the planner moves on instead of re-watching it. +// does not notify (historical claim), and the planner moves on instead of +// re-watching it. func TestLoop_StallClaimProbe_AlreadyClaimedMarksClaimed(t *testing.T) { f, st, h, cfg := probeSetup(t) f.claimRes = platform.ClaimResult{Outcome: platform.ClaimAlready} @@ -838,7 +851,19 @@ func TestLoop_StallClaimProbe_AlreadyClaimedMarksClaimed(t *testing.T) { } } n.mu.Unlock() - assert.Equal(t, 1, claimEvents, "exactly one claim notification") + assert.Equal(t, 0, claimEvents, "ALREADY_CLAIMED is a historical claim: no claim notification") +} + +// A stall probe answered OK did claim the drop just now, so it notifies once. +func TestLoop_StallClaimProbe_OKNotifiesOnce(t *testing.T) { + f, st, _, cfg := probeSetup(t) + f.claimRes = platform.ClaimResult{Outcome: platform.ClaimOK} + n := &recNotifier{} + cfg.Notifier = n + run(t, cfg) + require.Eventually(t, func() bool { return st.get("d1").Status == dropstate.Claimed }, 3*time.Second, 5*time.Millisecond) + time.Sleep(50 * time.Millisecond) + assert.Equal(t, 1, n.count("claim")) } // A single stall is a plain cooldown: no claim is sent. From fa4dbc54d3f52b4152eb13582bfe72ba4e86294d Mon Sep 17 00:00:00 2001 From: Alejandro Date: Wed, 30 Sep 2026 01:41:50 -0400 Subject: [PATCH 47/56] fix(twitch): match AvailableDrops game by ID, evict stale catalog sources - listByChannels now matches vc.Game.ID against the directory stream's game ID first, falling back to the display-name slug compare only when either ID is empty. A campaign whose game display name diverges from the directory's was wrongly dropped even though both carry the same game ID. - Catalog.Publish/snapshot now evict sources older than catalogTTL instead of merely skipping them, so a Reload-recycled backend (keyed by pointer when AccountID is unset) doesn't grow the sources map without bound across the process lifetime. - "tv discovery: merged shared catalog" now logs at INFO only when it actually merged something; a no-op merge logs at DEBUG. - Added a concurrent Publish/snapshot test (run under -race). Co-Authored-By: Claude Opus 5.5 (1M context) --- internal/platform/twitch/catalog.go | 26 ++++- internal/platform/twitch/catalog_test.go | 132 ++++++++++++++++++++++ internal/platform/twitch/chandisc.go | 18 ++- internal/platform/twitch/chandisc_test.go | 31 +++++ 4 files changed, 205 insertions(+), 2 deletions(-) diff --git a/internal/platform/twitch/catalog.go b/internal/platform/twitch/catalog.go index 55e61df..ba3d0e6 100644 --- a/internal/platform/twitch/catalog.go +++ b/internal/platform/twitch/catalog.go @@ -61,9 +61,25 @@ func (c *Catalog) Publish(source string, camps []platform.Campaign, allowed map[ } c.mu.Lock() c.sources[source] = catalogEntry{campaigns: cp, allowed: al, at: time.Now()} + c.evictStale() c.mu.Unlock() } +// evictStale drops sources whose last Publish is older than catalogTTL. +// Called by Publish and snapshot, both already holding mu. Without this a +// source keyed by backend pointer (catalogSource's no-AccountID fallback) +// accumulates forever across scheduler Reloads: the old *Backend becomes +// garbage, but nothing ever deletes its map entry — snapshot merely +// skipped it, so c.sources grew without bound over the process lifetime. +func (c *Catalog) evictStale() { + now := time.Now() + for k, e := range c.sources { + if now.Sub(e.at) >= catalogTTL { + delete(c.sources, k) + } + } +} + // snapshot unions every fresh source. For a campaign more than one source // carries, the entry with the most benefits wins (a publisher whose // whitelist lacks the game emits a benefit-less shell), newest breaking @@ -72,6 +88,7 @@ func (c *Catalog) Publish(source string, camps []platform.Campaign, allowed map[ func (c *Catalog) snapshot() (camps []platform.Campaign, allowed map[string][]string, age time.Duration, ok bool) { c.mu.Lock() defer c.mu.Unlock() + c.evictStale() keys := make([]string, 0, len(c.sources)) for k := range c.sources { keys = append(keys, k) @@ -180,6 +197,13 @@ func mergeCatalog(cat *Catalog, s platform.Session, camps []platform.Campaign, a } merged++ } - slog.Info("tv discovery: merged shared catalog", "merged", merged, "catalog_age", age.Round(time.Second)) + // Only interesting at INFO when it actually added a campaign; an + // every-tick no-op merge (the common case once the catalog has + // converged) would otherwise flood INFO logs for every account. + if merged > 0 { + slog.Info("tv discovery: merged shared catalog", "merged", merged, "catalog_age", age.Round(time.Second)) + } else { + slog.Debug("tv discovery: merged shared catalog", "merged", merged, "catalog_age", age.Round(time.Second)) + } return camps } diff --git a/internal/platform/twitch/catalog_test.go b/internal/platform/twitch/catalog_test.go index cb9138a..1818622 100644 --- a/internal/platform/twitch/catalog_test.go +++ b/internal/platform/twitch/catalog_test.go @@ -2,6 +2,9 @@ package twitch import ( "context" + "fmt" + "log/slog" + "sync" "testing" "time" @@ -230,3 +233,132 @@ func TestCatalog_PrefersEntryWithBenefits(t *testing.T) { assert.Len(t, camps[0].Benefits, 1) assert.Equal(t, []string{"x"}, allowed["c1"]) } + +// TestCatalog_PublishEvictsStaleSources: Publish must sweep any source +// entry older than catalogTTL out of the map, not just skip it in +// snapshot. Without this, a source keyed by backend pointer (the +// no-AccountID fallback in catalogSource) accumulates forever across +// Reloads, since the old *Backend is garbage but its map entry never is. +func TestCatalog_PublishEvictsStaleSources(t *testing.T) { + cat := NewCatalog() + cat.Publish("stale-src", []platform.Campaign{{ID: "c1", Game: "Rust"}}, nil) + cat.mu.Lock() + e := cat.sources["stale-src"] + e.at = e.at.Add(-catalogTTL - time.Minute) + cat.sources["stale-src"] = e + cat.mu.Unlock() + + cat.Publish("fresh-src", []platform.Campaign{{ID: "c2", Game: "Rust"}}, nil) + + cat.mu.Lock() + _, stillThere := cat.sources["stale-src"] + n := len(cat.sources) + cat.mu.Unlock() + assert.False(t, stillThere, "Publish must evict sources older than catalogTTL") + assert.Equal(t, 1, n, "only the fresh source should remain") +} + +// TestCatalog_SnapshotEvictsStaleSources: snapshot must also sweep stale +// entries out of the map (not merely skip them while computing the +// result), so a catalog that's never Published to again still shrinks. +func TestCatalog_SnapshotEvictsStaleSources(t *testing.T) { + cat := NewCatalog() + cat.Publish("stale-src", []platform.Campaign{{ID: "c1", Game: "Rust"}}, nil) + cat.mu.Lock() + e := cat.sources["stale-src"] + e.at = e.at.Add(-catalogTTL - time.Minute) + cat.sources["stale-src"] = e + cat.mu.Unlock() + + _, _, _, ok := cat.snapshot() + assert.False(t, ok, "no fresh source: snapshot reports not-ok") + + cat.mu.Lock() + n := len(cat.sources) + cat.mu.Unlock() + assert.Equal(t, 0, n, "snapshot must evict the stale source from the map") +} + +// levelCapture is a minimal slog.Handler that records the level each +// message was logged at, keyed by message text. +type levelCapture struct { + mu sync.Mutex + levels map[string]slog.Level +} + +func (h *levelCapture) Enabled(context.Context, slog.Level) bool { return true } +func (h *levelCapture) Handle(_ context.Context, r slog.Record) error { + h.mu.Lock() + defer h.mu.Unlock() + if h.levels == nil { + h.levels = map[string]slog.Level{} + } + h.levels[r.Message] = r.Level + return nil +} +func (h *levelCapture) WithAttrs([]slog.Attr) slog.Handler { return h } +func (h *levelCapture) WithGroup(string) slog.Handler { return h } + +func (h *levelCapture) levelOf(t *testing.T, msg string) slog.Level { + t.Helper() + h.mu.Lock() + defer h.mu.Unlock() + lvl, ok := h.levels[msg] + require.True(t, ok, "message %q was never logged", msg) + return lvl +} + +// TestMergeCatalog_LogsInfoOnlyWhenSomethingMerged: the "merged shared +// catalog" line is only interesting at INFO when it actually added a +// campaign (merged > 0); an every-tick no-op merge must log at DEBUG so it +// doesn't flood INFO logs for accounts whose catalog never contributes. +func TestMergeCatalog_LogsInfoOnlyWhenSomethingMerged(t *testing.T) { + prev := slog.Default() + t.Cleanup(func() { slog.SetDefault(prev) }) + + const msg = "tv discovery: merged shared catalog" + + t.Run("merged=0 logs at DEBUG", func(t *testing.T) { + cap := &levelCapture{} + slog.SetDefault(slog.New(cap)) + cat := NewCatalog() + cat.Publish("src", []platform.Campaign{{ID: "c1", Game: "Rust", Status: "active", EndsAt: time.Now().Add(time.Hour)}}, nil) + sess := platform.Session{Games: []string{"OtherGame"}} + mergeCatalog(cat, sess, nil, map[string][]string{}) + assert.Equal(t, slog.LevelDebug, cap.levelOf(t, msg)) + }) + + t.Run("merged>0 logs at INFO", func(t *testing.T) { + cap := &levelCapture{} + slog.SetDefault(slog.New(cap)) + cat := NewCatalog() + cat.Publish("src", []platform.Campaign{{ID: "c1", Game: "Rust", Status: "active", EndsAt: time.Now().Add(time.Hour)}}, nil) + sess := platform.Session{Games: []string{"Rust"}} + mergeCatalog(cat, sess, nil, map[string][]string{}) + assert.Equal(t, slog.LevelInfo, cap.levelOf(t, msg)) + }) +} + +// TestCatalog_ConcurrentPublishSnapshot exercises Publish and snapshot +// concurrently from many goroutines (run with -race) to guard the shared +// sources map + the new eviction sweep against data races. +func TestCatalog_ConcurrentPublishSnapshot(t *testing.T) { + cat := NewCatalog() + var wg sync.WaitGroup + for i := 0; i < 8; i++ { + i := i + wg.Add(1) + go func() { + defer wg.Done() + src := fmt.Sprintf("src-%d", i) + for j := 0; j < 25; j++ { + cat.Publish(src, []platform.Campaign{{ + ID: fmt.Sprintf("c-%d-%d", i, j), Game: "Rust", + Status: "active", EndsAt: time.Now().Add(time.Hour), + }}, map[string][]string{fmt.Sprintf("c-%d-%d", i, j): {"chan"}}) + _, _, _, _ = cat.snapshot() + } + }() + } + wg.Wait() +} diff --git a/internal/platform/twitch/chandisc.go b/internal/platform/twitch/chandisc.go index c431f3a..3a45893 100644 --- a/internal/platform/twitch/chandisc.go +++ b/internal/platform/twitch/chandisc.go @@ -37,6 +37,7 @@ type availableDropsFull struct { StartAt string `json:"startAt"` EndAt string `json:"endAt"` Game struct { + ID string `json:"id"` Name string `json:"name"` } `json:"game"` TimeBasedDrops []tvDrop `json:"timeBasedDrops"` @@ -162,7 +163,7 @@ func (d *discovery) listByChannels(ctx context.Context, sess platform.Session, c // AvailableDrops also returns the streamer's own channel // campaigns (no game) and campaigns for other games. Keep // only the game being walked; they flooded /drops otherwise. - if vc.Game.Name == "" || gameslug.Slug(vc.Game.Name) != slug { + if !vcMatchesGame(vc.Game.ID, s.GameID, vc.Game.Name, slug) { continue } start, end := parseISO(vc.StartAt), parseISO(vc.EndAt) @@ -246,6 +247,21 @@ func (d *discovery) listByChannels(ctx context.Context, sess platform.Session, c return out, allowed, nil } +// vcMatchesGame reports whether an AvailableDrops campaign belongs to the +// game being walked. The campaign's game ID and the directory stream's +// game ID -- both already read off Twitch's own payloads (the directory +// query resolves streamGameID; see listForGameDirectory) -- are the +// authoritative signal: a display name can vary between the two payloads +// (localization, punctuation) even for the identical game. The +// slug-vs-directory-slug compare is only a fallback for when either side's +// ID is empty (older/incomplete payload shapes). +func vcMatchesGame(vcGameID, streamGameID, vcGameName, slug string) bool { + if vcGameID != "" && streamGameID != "" { + return vcGameID == streamGameID + } + return vcGameName != "" && gameslug.Slug(vcGameName) == slug +} + func appendUnique(xs []string, x string) []string { for _, e := range xs { if e == x { diff --git a/internal/platform/twitch/chandisc_test.go b/internal/platform/twitch/chandisc_test.go index e5f5079..283ffc0 100644 --- a/internal/platform/twitch/chandisc_test.go +++ b/internal/platform/twitch/chandisc_test.go @@ -507,3 +507,34 @@ func TestListByChannels_SkipsAvailableDropsForOtherGames(t *testing.T) { assert.False(t, ids["campOther"], "campaign for a different game skipped") assert.Len(t, camps, 2) } + +// TestListByChannels_GameIDMatchOverridesNameSlugMismatch: Twitch's +// AvailableDrops game display name can diverge from the directory's own +// display name for the same game (e.g. a differently punctuated/localized +// variant) even though both payloads carry the same game ID. The ID -- not +// the display-name slug -- is the authoritative match signal; a campaign +// whose name wouldn't slug-match the directory's game must still be kept +// when its game ID matches the directory stream's game ID. +func TestListByChannels_GameIDMatchOverridesNameSlugMismatch(t *testing.T) { + const availIDMatchNameMismatch = `{"data":{"channel":{"viewerDropCampaigns":[ + {"id":"campIDMatch","name":"Renamed Drop","game":{"id":"263490","name":"RUST: Legacy Display"},"endAt":"2030-01-01T00:00:00Z", + "timeBasedDrops":[{"id":"dI","name":"I","requiredMinutesWatched":30,"requiredSubs":0,"benefitEdges":[{"benefit":{"id":"bI","name":"I","imageAssetURL":""}}]}]}]}}}` + srv := fakeGQL(t, map[string]func(map[string]any) string{ + "DirectoryPage_Game": func(v map[string]any) string { return dirRust }, + "DropsHighlightService_AvailableDrops": func(v map[string]any) string { + return availIDMatchNameMismatch + }, + "Inventory": func(map[string]any) string { return inventoryEmpty }, + }) + defer srv.Close() + + b := newForTest(srv.URL) + sess := platform.Session{AccessToken: "tv", ClientID: ClientTV, Games: []string{"Rust"}} + camps, err := b.ListActiveCampaigns(context.Background(), sess) + require.NoError(t, err) + ids := map[string]bool{} + for _, c := range camps { + ids[c.ID] = true + } + assert.True(t, ids["campIDMatch"], "game ID match must keep the campaign despite a display-name/slug mismatch") +} From 27b0be2db3ebff4b857e63fca495f26c4568ee61 Mon Sep 17 00:00:00 2001 From: Alejandro Date: Wed, 30 Sep 2026 01:43:14 -0400 Subject: [PATCH 48/56] perf(scheduler): build Reload's account entries concurrently Reload built every EntryBuilder serially, so one slow account (e.g. a browser-sidecar cold start) held up every other account's rebuild. buildEntriesConcurrently now runs builders with a cap of 4 workers while preserving the original entry order (indexed by builder position, not completion order). Co-Authored-By: Claude Opus 5.5 (1M context) --- internal/scheduler/control.go | 31 +++++++++++++++++++++++-- internal/scheduler/control_test.go | 36 ++++++++++++++++++++++++++++++ 2 files changed, 65 insertions(+), 2 deletions(-) diff --git a/internal/scheduler/control.go b/internal/scheduler/control.go index 8fad37d..6753eab 100644 --- a/internal/scheduler/control.go +++ b/internal/scheduler/control.go @@ -107,12 +107,39 @@ func (s *Scheduler) Reload(parent context.Context, builders []EntryBuilder) erro s.mu.Lock() s.entries = nil s.mu.Unlock() - for _, b := range builders { - s.AddEntry(b()) + for _, e := range buildEntriesConcurrently(builders) { + s.AddEntry(e) } return s.Start(parent) } +// reloadBuildConcurrency caps how many EntryBuilders run at once. A single +// slow builder (e.g. a browser-sidecar cold start) must not serialize every +// other account's rebuild, but an unbounded fan-out on a large roster would +// hammer the backend/session-store all at once; 4 balances the two. +const reloadBuildConcurrency = 4 + +// buildEntriesConcurrently runs builders with a cap of reloadBuildConcurrency +// concurrent workers and returns their entries in the SAME order as +// builders, regardless of completion order. +func buildEntriesConcurrently(builders []EntryBuilder) []entry { + entries := make([]entry, len(builders)) + sem := make(chan struct{}, reloadBuildConcurrency) + var wg sync.WaitGroup + for i, b := range builders { + i, b := i, b + wg.Add(1) + sem <- struct{}{} + go func() { + defer wg.Done() + defer func() { <-sem }() + entries[i] = b() + }() + } + wg.Wait() + return entries +} + // ReloadAccount restarts a SINGLE account's entry (build a fresh one) while // every other account keeps running untouched. Used for targeted account // edits so we don't tear down the whole roster. If the scheduler isn't diff --git a/internal/scheduler/control_test.go b/internal/scheduler/control_test.go index 7de9c01..539fa6e 100644 --- a/internal/scheduler/control_test.go +++ b/internal/scheduler/control_test.go @@ -150,3 +150,39 @@ func TestScheduler_ReloadAccountSurvivesTriggerContextCancel(t *testing.T) { require.Eventually(t, func() bool { return !fresh.running(20 * time.Millisecond) }, time.Second, 2*time.Millisecond, "watcher should stop when the root context is cancelled") } + +// TestScheduler_ReloadBuildsBuildersConcurrently: Reload must build the +// EntryBuilders concurrently (capped) rather than one at a time — a single +// slow account (e.g. a browser-sidecar cold start) must not hold up every +// other account's rebuild. 4 builders that each block 100ms must complete +// well under their serial total (400ms); the resulting entry order must +// still match the builders slice order, not completion order. +func TestScheduler_ReloadBuildsBuildersConcurrently(t *testing.T) { + s := New(Options{Notifier: &counterNotifier{}}) + + mkBuilder := func(id string) EntryBuilder { + return func() Entry { + time.Sleep(100 * time.Millisecond) + return NewEntry(id, &liveRunner{}) + } + } + + start := time.Now() + require.NoError(t, s.Reload(context.Background(), []EntryBuilder{ + mkBuilder("a"), mkBuilder("b"), mkBuilder("c"), mkBuilder("d"), + })) + elapsed := time.Since(start) + require.Less(t, elapsed, 250*time.Millisecond, + "4 concurrent 100ms builders should overlap, not serialize to ~400ms+") + + s.mu.Lock() + ids := make([]string, len(s.entries)) + for i, e := range s.entries { + ids[i] = e.id + } + s.mu.Unlock() + require.Equal(t, []string{"a", "b", "c", "d"}, ids, + "entry order must match builder order regardless of completion timing") + + s.Stop(context.Background()) +} From 56311c817eda86628a24ce1b71c1357ef7cd774d Mon Sep 17 00:00:00 2001 From: Alejandro Date: Wed, 30 Sep 2026 01:44:21 -0400 Subject: [PATCH 49/56] fix(watcher): back off Kick's idle re-discovery instead of a flat cadence MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A Kick account with no live channel for any campaign exhausts every campaign, idles, and immediately re-arms on the next tick, cycling pick_stream→pick_campaign→sleeping roughly every TickInterval and spamming state-change logs. nextKickIdleWait ramps the idle re-discovery wait 30s -> 60s -> 120s, then holds at the same recheckInterval (5m) cap Twitch already uses, and resets to the 30s floor the moment a pick succeeds. Twitch's fixed recheckInterval cadence is unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) --- internal/watcher/watcher.go | 45 ++++++++++++++++++++++++++++++-- internal/watcher/watcher_test.go | 35 +++++++++++++++++++++++++ 2 files changed, 78 insertions(+), 2 deletions(-) diff --git a/internal/watcher/watcher.go b/internal/watcher/watcher.go index e65648b..a61bf21 100644 --- a/internal/watcher/watcher.go +++ b/internal/watcher/watcher.go @@ -700,10 +700,18 @@ func (w *Watcher) Run(ctx context.Context) error { backoff := time.Duration(0) const maxBackoff = 5 * time.Minute + // Kick idle-sleep ramp (see nextKickIdleWait). Reset to 0 whenever a + // step makes real forward progress, so it restarts at the 30s floor the + // next time the account goes idle instead of resuming a stale ramp. + idleWait := time.Duration(0) + for { err := w.step(ctx) if err == nil { backoff = 0 + if s := w.State(); s != StateSleeping && s != StateAwaitingConnect { + idleWait = 0 + } } else if errors.Is(err, errIdle) { // Idle (sleeping / awaiting connect): nothing to mine right now. // As the LOWEST-priority fallback, run a queued force-watch task @@ -716,7 +724,12 @@ func (w *Watcher) Run(ctx context.Context) error { continue } w.setState(ctx, StatePickCampaign) - timer := time.NewTimer(recheckInterval) + wait := recheckInterval + if w.cfg.Platform == "kick" { + idleWait = nextKickIdleWait(idleWait) + wait = idleWait + } + timer := time.NewTimer(wait) select { case <-ctx.Done(): timer.Stop() @@ -798,9 +811,37 @@ var errIdle = errors.New("idle; recheck later") // recheckInterval is how long a sleeping/awaiting-connect watcher waits // before re-discovering. Matches the discovery scraper cadence (~5m) so -// the watcher's view and the persisted /drops view converge. +// the watcher's view and the persisted /drops view converge. Twitch +// accounts always wait exactly this long; see nextKickIdleWait for Kick's +// ramped variant. const recheckInterval = 5 * time.Minute +// kickIdleWaitSteps is the ramp nextKickIdleWait climbs through before +// holding at recheckInterval. +var kickIdleWaitSteps = []time.Duration{30 * time.Second, 60 * time.Second, 120 * time.Second, recheckInterval} + +// nextKickIdleWait computes how long a Kick watcher should wait before its +// next re-discovery after landing on StateSleeping/StateAwaitingConnect, +// given the PREVIOUS wait it used (0 the first time since going idle). A +// Kick account with no live channel for any of its campaigns otherwise +// re-picks and exhausts every campaign, then idles, then immediately +// re-arms on the very next tick — cycling +// pick_stream→pick_campaign→sleeping roughly every TickInterval and +// spamming state-change logs. Ramping 30s→60s→120s before holding at the +// same recheckInterval cap Twitch uses cuts that churn while still +// re-checking quickly the first couple of times (a channel may go live +// any moment). The caller resets prev to 0 the moment a pick succeeds, so +// the ramp always restarts at its 30s floor rather than resuming from +// wherever it had climbed to. +func nextKickIdleWait(prev time.Duration) time.Duration { + for _, step := range kickIdleWaitSteps { + if prev < step { + return step + } + } + return kickIdleWaitSteps[len(kickIdleWaitSteps)-1] +} + // stepErrBackoff is the first retry delay after a failed step (doubling up // to 5 min). A var only so tests can shrink it. var stepErrBackoff = 5 * time.Second diff --git a/internal/watcher/watcher_test.go b/internal/watcher/watcher_test.go index 198b130..ca32da9 100644 --- a/internal/watcher/watcher_test.go +++ b/internal/watcher/watcher_test.go @@ -1764,3 +1764,38 @@ func TestWatcher_GhostSkip_SelfHealsWhenBenefitReappears(t *testing.T) { w.mu.Unlock() assert.False(t, stillSkipped, "healme must be removed from skippedBenefits after self-heal") } + +// TestNextKickIdleWait_RampsThenCaps: a Kick account with no live channel +// for any campaign otherwise cycles pick_stream→pick_campaign→sleeping on +// every tick, spamming state-change logs. nextKickIdleWait ramps the idle +// re-discovery wait 30s→60s→120s, then holds at recheckInterval (5m) — +// never exceeding it. +func TestNextKickIdleWait_RampsThenCaps(t *testing.T) { + seq := make([]time.Duration, 0, 6) + wait := time.Duration(0) + for i := 0; i < 6; i++ { + wait = nextKickIdleWait(wait) + seq = append(seq, wait) + } + assert.Equal(t, []time.Duration{ + 30 * time.Second, + 60 * time.Second, + 120 * time.Second, + recheckInterval, + recheckInterval, + recheckInterval, + }, seq) +} + +// TestNextKickIdleWait_ResetStartsOverAtFloor: resetting to 0 (a +// successful pick) must restart the ramp from its 30s floor, not resume +// from wherever it had climbed to. +func TestNextKickIdleWait_ResetStartsOverAtFloor(t *testing.T) { + wait := nextKickIdleWait(0) + wait = nextKickIdleWait(wait) // 60s + wait = nextKickIdleWait(wait) // 120s + require.Equal(t, 120*time.Second, wait) + + wait = 0 // simulated reset on successful pick + assert.Equal(t, 30*time.Second, nextKickIdleWait(wait)) +} From 8e72b9e5121a36a58266ce20f30434e73fa3dc42 Mon Sep 17 00:00:00 2001 From: Alejandro Date: Wed, 30 Sep 2026 01:45:05 -0400 Subject: [PATCH 50/56] feat(pipeline): make v2 the default pipeline pipelineModeFor now returns v2 unless GRUB_PIPELINE=v1 or a per-account kv override says v1. build()'s existing automatic fallback to v1 when loop.New fails (e.g. no BrowserBackend for Kick) is unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) --- cmd/miner/main.go | 11 +++++++---- cmd/miner/main_test.go | 12 ++++++------ 2 files changed, 13 insertions(+), 10 deletions(-) diff --git a/cmd/miner/main.go b/cmd/miner/main.go index 7401f66..87e7087 100644 --- a/cmd/miner/main.go +++ b/cmd/miner/main.go @@ -955,17 +955,20 @@ func decodeKickChannels(s platform.Session) []string { } // pipelineModeFor picks v1 or v2 for an account: kv override first, then -// GRUB_PIPELINE, default v1. +// GRUB_PIPELINE, default v2. Set GRUB_PIPELINE=v1 (or a per-account +// override) to fall back to the legacy watcher; build() also falls back +// automatically per-account when loop.New itself fails (e.g. no +// BrowserBackend wired for Kick). func pipelineModeFor(ctx context.Context, q *gen.Queries, accountID string) string { if v, err := q.GetSettingString(ctx, store.PipelineOverridePrefix+accountID); err == nil { if s := string(v); s == "v1" || s == "v2" { return s } } - if os.Getenv("GRUB_PIPELINE") == "v2" { - return "v2" + if os.Getenv("GRUB_PIPELINE") == "v1" { + return "v1" } - return "v1" + return "v2" } // matchAnyChannel reports whether any campaign channel is one of logins. diff --git a/cmd/miner/main_test.go b/cmd/miner/main_test.go index 359c96f..44347f1 100644 --- a/cmd/miner/main_test.go +++ b/cmd/miner/main_test.go @@ -74,13 +74,13 @@ func TestPipelineModeFor(t *testing.T) { q := gen.New(db) t.Setenv("GRUB_PIPELINE", "") - assert.Equal(t, "v1", pipelineModeFor(ctx, q, "acc")) - t.Setenv("GRUB_PIPELINE", "v2") - assert.Equal(t, "v2", pipelineModeFor(ctx, q, "acc")) - require.NoError(t, q.UpsertSettingString(ctx, gen.UpsertSettingStringParams{Key: store.PipelineOverridePrefix + "acc", Value: []byte("v1")})) - assert.Equal(t, "v1", pipelineModeFor(ctx, q, "acc"), "per-account override beats env") + assert.Equal(t, "v2", pipelineModeFor(ctx, q, "acc"), "v2 is the default") + t.Setenv("GRUB_PIPELINE", "v1") + assert.Equal(t, "v1", pipelineModeFor(ctx, q, "acc"), "GRUB_PIPELINE=v1 falls back to the legacy watcher") + require.NoError(t, q.UpsertSettingString(ctx, gen.UpsertSettingStringParams{Key: store.PipelineOverridePrefix + "acc", Value: []byte("v2")})) + assert.Equal(t, "v2", pipelineModeFor(ctx, q, "acc"), "per-account override beats env") require.NoError(t, q.UpsertSettingString(ctx, gen.UpsertSettingStringParams{Key: store.PipelineOverridePrefix + "acc", Value: []byte("junk")})) - assert.Equal(t, "v2", pipelineModeFor(ctx, q, "acc"), "invalid override falls through") + assert.Equal(t, "v1", pipelineModeFor(ctx, q, "acc"), "invalid override falls through to env (still v1 here)") } func TestMatchAnyChannel(t *testing.T) { From 75ee5f717eca7389b40bab94c36a7343721c6b58 Mon Sep 17 00:00:00 2001 From: Alejandro Date: Wed, 30 Sep 2026 01:45:51 -0400 Subject: [PATCH 51/56] docs(pipeline): document v2-as-default + batch B changes README config table gains GRUB_PIPELINE (default v2, v1 = legacy fallback); spec section 6.5 records the default flip. CHANGELOG [Unreleased] logs the batch B changes: v2 default, parallel Reload builds, Kick idle-sleep backoff, and the catalog game-ID-match + eviction fixes. Co-Authored-By: Claude Opus 5.5 (1M context) --- README.md | 1 + docs/CHANGELOG.md | 5 +++++ .../specs/2026-09-27-pipeline-v2-design.md | 11 ++++++----- 3 files changed, 12 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 1813e26..c5c857e 100644 --- a/README.md +++ b/README.md @@ -223,6 +223,7 @@ default shown. | `GRUB_AUTHBYPASS` | `false` | **Disables all auth** when truthy (`1`/`true`). | | `GRUB_TWITCH_BROWSER` | `0` | `1` routes Twitch through the browser sidecar instead of direct HTTP. Experimental; the default direct-HTTP path is recommended. | | `GRUB_CANARY_INTERVAL` | Health-tab value | Overrides the accrual-canary run cadence (e.g. `6h`); falls back to the Settings ▸ Health value. | +| `GRUB_PIPELINE` | `v2` | `v1` falls back to the legacy watcher process-wide; also overridable per account (kv `pipeline_override:`). | > **"Invalid CSRF token"?** `GRUB_SECURE_COOKIES` must match your scheme: `0` > over plain HTTP, `1` over HTTPS (proxy must forward `X-Forwarded-Proto: https`). diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 88057eb..78b89fd 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -16,10 +16,15 @@ All notable changes to GrubDrops. - Twitch claim code split into status fetch + classification (behaviour unchanged). - Scheduler reads dashboard snapshots and discoveries through interfaces, so v1 and v2 runners both report state. +- Pipeline v2 is now the default. Set `GRUB_PIPELINE=v1` (or a per-account override) to fall back to the legacy watcher. +- Reloading accounts builds them in parallel, so a slow account no longer holds up the others. ### Fixed - Streamers' own channel campaigns with no game no longer flood the drops page, and campaigns past their end time show as ended. +- Twitch TV-channel discovery now matches a campaign's game by ID first (falling back to the display-name slug only when either ID is missing), so a campaign whose display name doesn't match the live directory's is no longer wrongly skipped. +- The shared Twitch discovery catalog no longer accumulates stale entries from recycled account backends; sources older than the TTL are evicted, not just ignored. +- Idle Kick accounts no longer flood the log with state changes every few seconds. ### Fixed (pipeline v2 only) diff --git a/docs/superpowers/specs/2026-09-27-pipeline-v2-design.md b/docs/superpowers/specs/2026-09-27-pipeline-v2-design.md index 3f24aab..dbf307c 100644 --- a/docs/superpowers/specs/2026-09-27-pipeline-v2-design.md +++ b/docs/superpowers/specs/2026-09-27-pipeline-v2-design.md @@ -344,12 +344,13 @@ Twitch placeholder-ID handling, Kick channel preferences. ### 6.5 Rollout -- `GRUB_PIPELINE=v1|v2` (default `v1`), overridable per account (account - settings toggle) so v2 can run on one account first. +- `GRUB_PIPELINE=v1|v2` (default `v2` as of 1.4.5), overridable per account + (account settings toggle / kv `pipeline_override:`) so a single + account can still fall back to v1 without affecting the rest. - `cmd/miner/main.go` `build()` branches on the selection; v1 wiring - unchanged. -- Default flips to `v2` only after live verification on a Twitch drop and a - Kick drop (CLAUDE.md accrual/claim gate). v1 deletion (incl. + unchanged. `build()` also falls back to v1 automatically, per account, + when `loop.New` itself fails (e.g. no `BrowserBackend` wired for Kick). +- Default flipped from `v1` to `v2` in 1.4.5 batch B. v1 deletion (incl. `watcher.go`, `skip_override`/`collect_override` loaders, `account_channels`) is a separate follow-up release. From 4c848b566d26bba461f4b4e32bee182ad5b8bcd5 Mon Sep 17 00:00:00 2001 From: Alejandro Date: Wed, 30 Sep 2026 02:00:24 -0400 Subject: [PATCH 52/56] feat(pipeline): watch-to-enroll discovery for TV Twitch accounts TV-minted Twitch tokens can't read the campaign list, but Inventory lists every campaign the viewer is enrolled in, and watching a counting channel enrolls them. When Plan() says Idle, the v2 loop now watches the top live DROPS_ENABLED channel of the next whitelisted game (round-robin over Games, via ListEligibleChannels on a synthetic open campaign -> Twitch game directory) for EnrollWatch (10m), then marks the game probed (6h cooldown), reconciles and refreshes channels. Any non-Idle decision preempts the enroll watch at once; a down channel is cooled and the game retried on its next channel. Planner stays pure: enrolling is loop-side only. Snapshot state "discovering" with channel + game. Wired in main.go for twitch sessions with ClientID == twitch.ClientTV; Kick and Android off. Co-Authored-By: Claude Opus 5.5 (1M context) --- cmd/miner/main.go | 12 +- cmd/miner/main_test.go | 11 ++ docs/CHANGELOG.md | 1 + internal/pipeline/loop/enroll.go | 147 +++++++++++++++ internal/pipeline/loop/enroll_test.go | 229 +++++++++++++++++++++++ internal/pipeline/loop/loop.go | 108 ++++++++++- internal/pipeline/loop/loop_test.go | 19 +- internal/platform/twitch/backend_test.go | 30 +++ 8 files changed, 545 insertions(+), 12 deletions(-) create mode 100644 internal/pipeline/loop/enroll.go create mode 100644 internal/pipeline/loop/enroll_test.go diff --git a/cmd/miner/main.go b/cmd/miner/main.go index 87e7087..b6f3eff 100644 --- a/cmd/miner/main.go +++ b/cmd/miner/main.go @@ -498,7 +498,8 @@ func run() error { ProgressNotifyStepPct: progressStep, // Twitch hides drops claimed outside the app once their // campaign leaves the Inventory; Kick keeps them listed. - ClaimProbe: a.Platform == "twitch", + ClaimProbe: a.Platform == "twitch", + EnrollDiscovery: enrollDiscoveryFor(a.Platform, sess), }) if err == nil { logger.Info("pipeline v2 enabled for account", "account", a.ID, "platform", a.Platform) @@ -971,6 +972,15 @@ func pipelineModeFor(ctx context.Context, q *gen.Queries, accountID string) stri return "v2" } +// enrollDiscoveryFor turns on pipeline v2 watch-to-enroll discovery for +// TV-client Twitch sessions: they can't read the campaign list, so they +// find campaigns by watching a drops-enabled channel per whitelisted game +// until Inventory lists what Twitch enrolled them in. Android sessions see +// the list directly and Kick has no such gap, so both stay off. +func enrollDiscoveryFor(platformName string, sess platform.Session) bool { + return platformName == "twitch" && sess.ClientID == twitch.ClientTV +} + // matchAnyChannel reports whether any campaign channel is one of logins. // Nil when logins is empty, so the null-game gate stays off. func matchAnyChannel(logins []string) func([]string) bool { diff --git a/cmd/miner/main_test.go b/cmd/miner/main_test.go index 44347f1..3c841bf 100644 --- a/cmd/miner/main_test.go +++ b/cmd/miner/main_test.go @@ -9,10 +9,21 @@ import ( "github.com/stretchr/testify/require" "github.com/aalejandrofer/grubdrops/internal/api" + "github.com/aalejandrofer/grubdrops/internal/platform" + "github.com/aalejandrofer/grubdrops/internal/platform/twitch" "github.com/aalejandrofer/grubdrops/internal/store" "github.com/aalejandrofer/grubdrops/internal/store/gen" ) +// TestEnrollDiscoveryFor: watch-to-enroll discovery is for TV-client Twitch +// sessions only; Android (legacy empty ClientID) and Kick stay off. +func TestEnrollDiscoveryFor(t *testing.T) { + assert.True(t, enrollDiscoveryFor("twitch", platform.Session{ClientID: twitch.ClientTV})) + assert.False(t, enrollDiscoveryFor("twitch", platform.Session{}), "Android / legacy session") + assert.False(t, enrollDiscoveryFor("twitch", platform.Session{ClientID: "android"})) + assert.False(t, enrollDiscoveryFor("kick", platform.Session{ClientID: twitch.ClientTV})) +} + func TestLoadAccountChannels(t *testing.T) { ctx := context.Background() db, err := store.Open(ctx, t.TempDir()+"/t.db") diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 78b89fd..76564c5 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -11,6 +11,7 @@ All notable changes to GrubDrops. - `drop_state` and `account_streamer_priority` tables (migration 0016), with a one-time backfill from claim history, manual marks and ghost-skips. - Accounts on the Twitch TV login share the full campaign list found by any Android-login account in the same GrubDrops, so campaigns hidden from the TV client (for example behind a global badge campaign) are still found. - A banner explains that Twitch limits campaign discovery for new (TV) logins and points to the game whitelist. +- Accounts on the Twitch TV login find campaigns by briefly watching a top drops-enabled channel for each whitelisted game when there is nothing else to mine, so Twitch enrolls them and the campaigns appear. (Pipeline v2: 10 minutes per game, each game at most once every 6 hours; dashboard state `discovering`.) ### Changed diff --git a/internal/pipeline/loop/enroll.go b/internal/pipeline/loop/enroll.go new file mode 100644 index 0000000..a911257 --- /dev/null +++ b/internal/pipeline/loop/enroll.go @@ -0,0 +1,147 @@ +package loop + +import ( + "context" + "log/slog" + "sort" + "strings" + "time" + + "github.com/aalejandrofer/grubdrops/internal/platform" +) + +// Watch-to-enroll discovery (Config.EnrollDiscovery). A TV-client Twitch +// token can't read the campaign list, but Inventory lists every campaign +// the viewer is enrolled in, and Twitch enrolls a viewer who watches a +// channel that counts toward a campaign. When the planner says Idle, the +// loop watches the top drops-enabled channel of the next whitelisted game +// for EnrollWatch, then reconciles so the enrolled campaigns become rows. + +// maybeEnroll starts an enroll watch on the next whitelisted game (round- +// robin from enrollNext) that is off cooldown and has a usable channel. +// Called only when the planner said Idle and no enroll watch runs. Games +// probed within EnrollGameCooldown are skipped without I/O; a game whose +// lookup found nothing is skipped for LiveEvery, so an all-dark whitelist +// costs at most one directory call per game per LiveEvery. +func (l *Loop) maybeEnroll(ctx context.Context) { + games := l.cfg.Games + if !l.cfg.EnrollDiscovery || l.enroll != nil || l.authBlocked || len(games) == 0 { + return + } + now := l.cfg.Now() + n := len(games) + for i := 0; i < n; i++ { + idx := (l.enrollNext + i) % n + g := games[idx] + if g == "" { + continue + } + k := strings.ToLower(g) + if t, ok := l.enrollProbed[k]; ok && now.Sub(t) < l.cfg.EnrollGameCooldown { + continue + } + if t, ok := l.enrollEmpty[k]; ok && now.Sub(t) < l.cfg.LiveEvery { + continue + } + s, ok := l.enrollChannel(ctx, g, now) + if !ok { + l.enrollEmpty[k] = now + continue + } + l.enrollNext = (idx + 1) % n + l.startEnroll(ctx, g, idx, s) + return + } +} + +// enrollChannel returns the top live drops-enabled channel for game that +// is not cooling down. It lists the channels of a synthetic open campaign +// (no ID, so no allow-list), which the Twitch backend answers from the +// DROPS_ENABLED game directory sorted by viewers. +func (l *Loop) enrollChannel(ctx context.Context, game string, now time.Time) (platform.Stream, bool) { + streams, err := l.cfg.Backend.ListEligibleChannels(ctx, l.cfg.Session, platform.Campaign{Platform: l.cfg.Platform, Game: game}) + if err != nil { + slog.Debug("pipeline enroll: list channels failed", "account", l.cfg.AccountID, "game", game, "err", err) + return platform.Stream{}, false + } + sort.SliceStable(streams, func(i, j int) bool { return streams[i].ViewerCount > streams[j].ViewerCount }) + for _, s := range streams { + if s.Channel == "" || !s.DropsEnabled { + continue + } + if until, ok := l.cooldowns[strings.ToLower(s.Channel)]; ok && now.Before(until) { + continue + } + return s, true + } + return platform.Stream{}, false +} + +// startEnroll runs a stall-free session on s and arms the EnrollWatch +// timer. No PubSub subscription: the watch serves no known drop. +func (l *Loop) startEnroll(ctx context.Context, game string, idx int, s platform.Stream) { + l.enroll = &enrollRun{game: game, idx: idx, stream: s, started: l.cfg.Now()} + l.runSession(ctx, s, nil, 0) + l.stopEnrollTimer() + l.enrollT = time.NewTimer(l.cfg.EnrollWatch) + l.enrollC = l.enrollT.C + slog.Info("pipeline enroll discovery", "kind", "discovery", "account", l.cfg.AccountID, + "phase", "start", "game", game, "channel", s.Channel, "viewers", s.ViewerCount, "watch", l.cfg.EnrollWatch.String()) +} + +func (l *Loop) stopEnrollTimer() { + if l.enrollT != nil { + l.enrollT.Stop() + } + l.enrollT, l.enrollC = nil, nil +} + +// endEnroll stops the enroll watch without marking its game probed (stream +// down, preempted, auth wall); the cursor rewinds so the game is retried. +func (l *Loop) endEnroll(reason string) { + e := l.enroll + if e == nil { + return + } + l.stopEnrollTimer() + l.haltSession() + l.enroll = nil + l.enrollNext = e.idx + slog.Info("pipeline enroll discovery", "kind", "discovery", "account", l.cfg.AccountID, + "phase", "stop", "game", e.game, "channel", e.stream.Channel, "reason", reason) +} + +// finishEnroll ends an enroll watch whose EnrollWatch is up: stop the +// session, mark the game probed, then reconcile (Inventory now lists any +// campaign the watch enrolled the account in) and refresh channels so the +// replan that follows can mine what it found. +func (l *Loop) finishEnroll(ctx context.Context) { + e := l.enroll + l.enrollT, l.enrollC = nil, nil // already fired + if e == nil { + return + } + l.haltSession() + l.enroll = nil + l.enrollProbed[strings.ToLower(e.game)] = l.cfg.Now() + before := make(map[string]bool, len(l.rows)) + for id := range l.rows { + before[id] = true + } + l.reconcile(ctx) + if l.authBlocked { + return + } + l.refreshLive(ctx) + newDrops := 0 + camps := map[string]bool{} + for id, r := range l.rows { + if !before[id] { + newDrops++ + camps[r.CampaignID] = true + } + } + slog.Info("pipeline enroll discovery", "kind", "discovery", "account", l.cfg.AccountID, + "phase", "stop", "game", e.game, "channel", e.stream.Channel, "reason", "done", + "campaigns_found", len(camps), "new_drops", newDrops) +} diff --git a/internal/pipeline/loop/enroll_test.go b/internal/pipeline/loop/enroll_test.go new file mode 100644 index 0000000..741acbc --- /dev/null +++ b/internal/pipeline/loop/enroll_test.go @@ -0,0 +1,229 @@ +package loop + +import ( + "sync" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/aalejandrofer/grubdrops/internal/platform" +) + +// fakeClock is a settable Config.Now. +type fakeClock struct { + mu sync.Mutex + t time.Time +} + +func (c *fakeClock) Now() time.Time { + c.mu.Lock() + defer c.mu.Unlock() + return c.t +} + +func (c *fakeClock) Advance(d time.Duration) { + c.mu.Lock() + c.t = c.t.Add(d) + c.mu.Unlock() +} + +// enrollSetup is an idle TV account: no campaigns visible, whitelist +// [A, B], each game's directory listing one drops-enabled channel. +func enrollSetup(t *testing.T) (*fakeBackend, Config) { + f, _, _, cfg := setup(t) + f.camps, f.progress, f.inventory = nil, nil, nil + f.gameLive = map[string][]platform.Stream{ + // A higher-viewer stream without drops must never be picked. + "A": {{Channel: "nodrops", ViewerCount: 500}, {Channel: "a1", ViewerCount: 100, DropsEnabled: true}}, + "B": {{Channel: "b1", ViewerCount: 50, DropsEnabled: true}}, + } + cfg.Games = []string{"A", "B"} + cfg.AllowGame = func(g string) bool { return g == "A" || g == "B" } + cfg.EnrollDiscovery = true + cfg.EnrollWatch = time.Hour // tests shorten it where they need a finish + return f, cfg +} + +func (f *fakeBackend) watched() []string { + f.mu.Lock() + defer f.mu.Unlock() + return append([]string(nil), f.watching...) +} + +func (f *fakeBackend) reconciles() int { + f.mu.Lock() + defer f.mu.Unlock() + return len(f.listGames) +} + +func TestEnroll_RoundRobinWithCooldown(t *testing.T) { + f, cfg := enrollSetup(t) + cfg.EnrollWatch = 80 * time.Millisecond + l, _ := run(t, cfg) + + require.Eventually(t, func() bool { + s := l.Snapshot() + return s.State == "discovering" && s.Channel == "a1" + }, 2*time.Second, 2*time.Millisecond, "enroll starts on A's top drops-enabled channel") + assert.Equal(t, "A", l.Snapshot().CampaignGame) + recBefore := f.reconciles() + + // After EnrollWatch the A session stops, a reconcile runs, and the next + // idle cycle moves on to B (A is on cooldown). + require.Eventually(t, func() bool { + w := f.watched() + return len(w) == 2 && w[1] == "b1" + }, 2*time.Second, 2*time.Millisecond) + assert.Greater(t, f.reconciles(), recBefore, "post-enroll reconcile ran") + f.mu.Lock() + assert.GreaterOrEqual(t, f.stops, 1, "A session stopped before B started") + f.mu.Unlock() + require.Eventually(t, func() bool { return l.Snapshot().Channel == "b1" }, time.Second, 2*time.Millisecond) + + // Both games probed: back to plain idle, no further StartWatch. + require.Eventually(t, func() bool { return l.Snapshot().State == "idle" }, 2*time.Second, 2*time.Millisecond) + l.Nudge() + time.Sleep(150 * time.Millisecond) + assert.Equal(t, []string{"a1", "b1"}, f.watched(), "every game on cooldown: stay idle") + assert.Equal(t, "idle", l.Snapshot().State) +} + +func TestEnroll_ReconcileFindsCampaignThenMines(t *testing.T) { + f, cfg := enrollSetup(t) + cfg.EnrollWatch = 60 * time.Millisecond + // Watching a1 enrolls the account: the next reconcile lists c1 (game A). + f.onStartWatch = func(ch string) { + if ch != "a1" { + return + } + f.camps = []platform.Campaign{{ID: "c1", Platform: "twitch", Game: "A", Name: "Camp", Status: "active", + AccountLinked: true, AccountLinkChecked: true, + Benefits: []platform.DropBenefit{{ID: "d1", CampaignID: "c1", Name: "Drop", RequiredMinutes: 60}}}} + f.progress = []platform.DropProgress{{DropID: "d1", CampaignID: "c1", Minutes: 0, Required: 60, Known: true}} + f.live = []platform.Stream{{Channel: "ch1", ViewerCount: 10}} + } + l, _ := run(t, cfg) + + require.Eventually(t, func() bool { + s := l.Snapshot() + return s.State == "watching" && s.Channel == "ch1" + }, 2*time.Second, 2*time.Millisecond, "post-enroll reconcile surfaced a mineable drop: mine it") + time.Sleep(150 * time.Millisecond) + assert.Equal(t, []string{"a1", "ch1"}, f.watched(), "mining preempts further enrolling (no b1)") + assert.Equal(t, "watching", l.Snapshot().State) +} + +func TestEnroll_MineMidEnrollPreempts(t *testing.T) { + f, cfg := enrollSetup(t) // EnrollWatch 1h: only preemption can end it + l, _ := run(t, cfg) + require.Eventually(t, func() bool { return l.Snapshot().State == "discovering" }, 2*time.Second, 2*time.Millisecond) + + f.mu.Lock() + f.camps = []platform.Campaign{{ID: "c1", Platform: "twitch", Game: "B", Name: "Camp", Status: "active", + AccountLinked: true, AccountLinkChecked: true, + Benefits: []platform.DropBenefit{{ID: "d1", CampaignID: "c1", Name: "Drop", RequiredMinutes: 60}}}} + f.progress = []platform.DropProgress{{DropID: "d1", CampaignID: "c1", Minutes: 0, Required: 60, Known: true}} + f.live = []platform.Stream{{Channel: "ch1", ViewerCount: 10}} + f.mu.Unlock() + l.Nudge() + + require.Eventually(t, func() bool { + s := l.Snapshot() + return s.State == "watching" && s.Channel == "ch1" + }, 2*time.Second, 2*time.Millisecond) + assert.Equal(t, []string{"a1", "ch1"}, f.watched()) + f.mu.Lock() + assert.GreaterOrEqual(t, f.stops, 1, "enroll session stopped") + f.mu.Unlock() +} + +func TestEnroll_DisabledNeverEnrolls(t *testing.T) { + f, cfg := enrollSetup(t) + cfg.EnrollDiscovery = false // Android / Kick + l, _ := run(t, cfg) + require.Eventually(t, func() bool { return l.Snapshot().State == "idle" }, 2*time.Second, 2*time.Millisecond) + l.Nudge() + time.Sleep(150 * time.Millisecond) + assert.Empty(t, f.watched()) + f.mu.Lock() + assert.Empty(t, f.gameLookups, "no directory lookups when enroll discovery is off") + f.mu.Unlock() +} + +func TestEnroll_CooldownFollowsConfigNow(t *testing.T) { + f, cfg := enrollSetup(t) + cfg.Games = []string{"A"} + cfg.EnrollWatch = 40 * time.Millisecond + clk := &fakeClock{t: time.Date(2026, 9, 30, 12, 0, 0, 0, time.UTC)} + cfg.Now = clk.Now + l, _ := run(t, cfg) + + require.Eventually(t, func() bool { return len(f.watched()) == 1 }, 2*time.Second, 2*time.Millisecond) + require.Eventually(t, func() bool { return l.Snapshot().State == "idle" }, 2*time.Second, 2*time.Millisecond) + clk.Advance(EnrollGameCooldown - time.Minute) + l.Nudge() + time.Sleep(150 * time.Millisecond) + assert.Len(t, f.watched(), 1, "A still on cooldown: stay idle, no StartWatch") + + clk.Advance(2 * time.Minute) // past the 6h cooldown + l.Nudge() + require.Eventually(t, func() bool { return len(f.watched()) == 2 }, 2*time.Second, 2*time.Millisecond) + assert.Equal(t, []string{"a1", "a1"}, f.watched()) +} + +func TestEnroll_SkipsGameWithoutChannelsAndBacksOff(t *testing.T) { + f, cfg := enrollSetup(t) + f.gameLive["A"] = nil // nothing live for A + l, _ := run(t, cfg) + require.Eventually(t, func() bool { return l.Snapshot().Channel == "b1" }, 2*time.Second, 2*time.Millisecond) + for i := 0; i < 5; i++ { + l.Nudge() + time.Sleep(10 * time.Millisecond) + } + f.mu.Lock() + defer f.mu.Unlock() + assert.Equal(t, 1, f.gameLookups["A"], "an empty game is not re-queried while B enrolls") + assert.Equal(t, []string{"b1"}, f.watching) +} + +func TestEnroll_AllDarkWhitelistBacksOffLookups(t *testing.T) { + f, cfg := enrollSetup(t) + f.gameLive = map[string][]platform.Stream{} // nothing live anywhere + clk := &fakeClock{t: time.Date(2026, 9, 30, 12, 0, 0, 0, time.UTC)} + cfg.Now = clk.Now + cfg.LiveEvery = time.Hour + l, _ := run(t, cfg) + require.Eventually(t, func() bool { return l.Snapshot().State == "idle" }, 2*time.Second, 2*time.Millisecond) + for i := 0; i < 5; i++ { + l.Nudge() + time.Sleep(10 * time.Millisecond) + } + f.mu.Lock() + assert.Equal(t, 1, f.gameLookups["A"], "a dark game is looked up once per LiveEvery, not every iteration") + assert.Equal(t, 1, f.gameLookups["B"]) + assert.Empty(t, f.watching) + f.gameLive["B"] = []platform.Stream{{Channel: "b1", ViewerCount: 5, DropsEnabled: true}} + f.mu.Unlock() + + clk.Advance(time.Hour + time.Minute) + l.Nudge() + require.Eventually(t, func() bool { return l.Snapshot().Channel == "b1" }, 2*time.Second, 2*time.Millisecond, + "after LiveEvery the dark games are retried") +} + +func TestEnroll_ChannelDownCoolsItAndPicksNext(t *testing.T) { + f, cfg := enrollSetup(t) + f.gameLive["A"] = []platform.Stream{ + {Channel: "a1", ViewerCount: 100, DropsEnabled: true}, + {Channel: "a2", ViewerCount: 50, DropsEnabled: true}, + } + f.beatFailOn = "a1" + l, _ := run(t, cfg) + require.Eventually(t, func() bool { + s := l.Snapshot() + return s.State == "discovering" && s.Channel == "a2" + }, 2*time.Second, 2*time.Millisecond, "a1 went down: cooled, A retried on a2") + assert.Equal(t, []string{"a1", "a2"}, f.watched()) +} diff --git a/internal/pipeline/loop/loop.go b/internal/pipeline/loop/loop.go index 26aeec7..81d46bd 100644 --- a/internal/pipeline/loop/loop.go +++ b/internal/pipeline/loop/loop.go @@ -73,6 +73,17 @@ type Config struct { ClaimProbe bool // CatchUpEvery spaces catch-up claim probes (default 3s). CatchUpEvery time.Duration + // EnrollDiscovery (TV-client Twitch sessions) turns on watch-to-enroll + // discovery. A TV token can't read the campaign list, but Inventory + // lists every campaign the viewer is enrolled in, and Twitch enrolls a + // viewer who watches a channel that counts toward a campaign. So when + // the planner has nothing to do, the loop watches the top live + // drops-enabled channel of the next whitelisted game (round-robin over + // Games) for EnrollWatch, then reconciles. A game is probed at most + // once per EnrollGameCooldown. Kick and Android sessions leave it off. + EnrollDiscovery bool + EnrollWatch time.Duration // default EnrollWatch (10m) + EnrollGameCooldown time.Duration // default EnrollGameCooldown (6h) Now func() time.Time ReconcileEvery time.Duration // default 15m @@ -83,6 +94,12 @@ type Config struct { DownCooldown time.Duration // default 2m } +// Enroll discovery defaults (see Config.EnrollDiscovery). +const ( + EnrollWatch = 10 * time.Minute + EnrollGameCooldown = 6 * time.Hour +) + // maxLiveCampaigns bounds channel lookups per refresh. const maxLiveCampaigns = 8 @@ -149,6 +166,26 @@ type Loop struct { catchUpQ []string catchUpT *time.Timer catchUpC <-chan time.Time + // Enroll discovery (EnrollDiscovery). enroll is the running enroll + // watch (nil when none); enrollC fires once its EnrollWatch is up. + // enrollProbed holds each game's last completed probe (lower-cased + // name), enrollEmpty the last lookup that found no usable channel + // (retried after LiveEvery), enrollNext the round-robin cursor into + // Games. In memory only: a restart may probe each game once more. + enroll *enrollRun + enrollT *time.Timer + enrollC <-chan time.Time + enrollProbed map[string]time.Time + enrollEmpty map[string]time.Time + enrollNext int +} + +// enrollRun is one enroll-discovery watch: game Games[idx] on stream. +type enrollRun struct { + game string + idx int + stream platform.Stream + started time.Time } // New assembles the loop, checking the backend's pipeline capabilities once. @@ -185,6 +222,12 @@ func New(cfg Config) (*Loop, error) { if cfg.CatchUpEvery <= 0 { cfg.CatchUpEvery = 3 * time.Second } + if cfg.EnrollWatch <= 0 { + cfg.EnrollWatch = EnrollWatch + } + if cfg.EnrollGameCooldown <= 0 { + cfg.EnrollGameCooldown = EnrollGameCooldown + } cfg.Session.AccountID = cfg.AccountID if cfg.Session.Games == nil { cfg.Session.Games = cfg.Games @@ -205,6 +248,8 @@ func New(cfg Config) (*Loop, error) { pendingHistory: map[string]bool{}, stallCount: map[string]int{}, probed: map[string]bool{}, + enrollProbed: map[string]time.Time{}, + enrollEmpty: map[string]time.Time{}, } l.prober, _ = cfg.Backend.(platform.ChannelProber) l.subs, _ = cfg.Backend.(platform.ChannelSubscriber) @@ -267,6 +312,7 @@ func (l *Loop) Run(ctx context.Context) error { if l.catchUpT != nil { l.catchUpT.Stop() } + l.stopEnrollTimer() }() l.reconcile(ctx) if l.authBlocked { @@ -302,6 +348,8 @@ func (l *Loop) Run(ctx context.Context) error { l.catchUpC, l.catchUpT = nil, nil l.catchUpOne(ctx) l.armCatchUp() + case <-l.enrollC: + l.finishEnroll(ctx) } if l.authBlocked { return nil @@ -489,6 +537,9 @@ func (l *Loop) catchUpOne(ctx context.Context) { func (l *Loop) onIntegrityBlocked(ctx context.Context) { slog.Warn("pipeline: integrity blocked, marking account needs_auth", "kind", "auth", "account", l.cfg.AccountID) l.authBlocked = true + if l.enroll != nil { + l.endEnroll("auth_blocked") + } l.stopSession() l.current = nil l.mu.Lock() @@ -754,6 +805,12 @@ func (l *Loop) onSession(ctx context.Context, ev session.Event) { if l.isCurrent(ev.Channel) { l.cooldowns[strings.ToLower(ev.Channel)] = now.Add(l.cfg.DownCooldown) } + if e := l.enroll; e != nil && strings.EqualFold(e.stream.Channel, ev.Channel) { + // Cool the channel and end the watch unprobed: the replan that + // follows retries the same game on its next channel. + l.cooldowns[strings.ToLower(ev.Channel)] = now.Add(l.cfg.DownCooldown) + l.endEnroll("stream_down") + } case session.Stalled: if l.isCurrent(ev.Channel) { if l.cfg.ClaimProbe { @@ -942,8 +999,29 @@ func sameServeSet(a, b []string) bool { return true } +// replan runs the planner and applies its decision. Enroll discovery sits +// on top of it, loop-side, so the planner stays pure: an Idle decision +// keeps (or starts) an enroll watch; anything else preempts it at once. func (l *Loop) replan(ctx context.Context) { d := planner.Plan(l.plannerInput()) + if l.enroll != nil { + if d.Kind == planner.Idle { + if l.current != nil { + l.current.Serves, l.current.Reason = d.Serves, d.Reason + } + l.updateSnapshot() + return + } + l.endEnroll("preempted") + } + l.apply(ctx, d) + if l.current != nil && l.current.Kind == planner.Idle && l.enroll == nil { + l.maybeEnroll(ctx) + l.updateSnapshot() + } +} + +func (l *Loop) apply(ctx context.Context, d planner.Decision) { if l.current != nil && l.current.Same(d) { servesChanged := !sameServeSet(l.current.Serves, d.Serves) l.current.Serves, l.current.Reason = d.Serves, d.Reason @@ -979,10 +1057,6 @@ func (l *Loop) replan(ctx context.Context) { } func (l *Loop) startSession(ctx context.Context, d planner.Decision) { - sctx, cancel := context.WithCancel(ctx) - l.sessCancel = cancel - l.gen++ - l.lastInv = nil if want := d.Channel.ChannelID; l.subs != nil && want != l.subscribed { l.unsubscribe() if want != "" { @@ -990,15 +1064,25 @@ func (l *Loop) startSession(ctx context.Context, d planner.Decision) { l.subscribed = want } } + stall := l.cfg.StallPolls + if d.Kind == planner.ForceWatch { + stall = 0 // channel-points farming has no drop progress to stall on + } + l.runSession(ctx, d.Channel, d.Serves, stall) +} + +// runSession starts one session goroutine under a fresh generation stamp. +func (l *Loop) runSession(ctx context.Context, stream platform.Stream, serves []string, stallPolls int) { + sctx, cancel := context.WithCancel(ctx) + l.sessCancel = cancel + l.gen++ + l.lastInv = nil ticker := time.NewTicker(l.cfg.BeatEvery) cfg := session.Config{ - Backend: l.cfg.Backend, Session: l.cfg.Session, Stream: d.Channel, - Serves: d.Serves, Ticks: ticker.C, StallPolls: l.cfg.StallPolls, Gen: l.gen, + Backend: l.cfg.Backend, Session: l.cfg.Session, Stream: stream, + Serves: serves, Ticks: ticker.C, StallPolls: stallPolls, Gen: l.gen, AccountID: l.cfg.AccountID, } - if d.Kind == planner.ForceWatch { - cfg.StallPolls = 0 // channel-points farming has no drop progress to stall on - } done := make(chan struct{}) l.sessDone = done go func() { @@ -1062,7 +1146,11 @@ func stateString(d *planner.Decision) string { func (l *Loop) updateSnapshot() { s := watcher.Snapshot{AccountID: l.cfg.AccountID, State: stateString(l.current)} - if d := l.current; d != nil && d.Kind != planner.Idle { + if e := l.enroll; e != nil { + s.State = "discovering" + s.Channel, s.ViewerCount, s.StartedAt = e.stream.Channel, e.stream.ViewerCount, e.started + s.CampaignGame = e.game + } else if d := l.current; d != nil && d.Kind != planner.Idle { s.Channel, s.ViewerCount, s.StartedAt = d.Channel.Channel, d.Channel.ViewerCount, l.lastSwap if len(d.Serves) > 0 { b, c := l.benefit(d.Serves[0]) diff --git a/internal/pipeline/loop/loop_test.go b/internal/pipeline/loop/loop_test.go index c261fa0..ad2df06 100644 --- a/internal/pipeline/loop/loop_test.go +++ b/internal/pipeline/loop/loop_test.go @@ -45,6 +45,13 @@ type fakeBackend struct { // ClaimDrop call's drop id in order. claimByDrop map[string]platform.ClaimResult claimedIDs []string + // gameLive answers ListEligibleChannels for a synthetic open campaign + // (ID "", Game set): the Twitch directory fallback enroll discovery + // uses. gameLookups counts those calls per game. + gameLive map[string][]platform.Stream + gameLookups map[string]int + // onStartWatch, when set, runs (under mu) on every StartWatch. + onStartWatch func(channel string) } func (f *fakeBackend) ListActiveCampaigns(_ context.Context, s platform.Session) ([]platform.Campaign, error) { @@ -58,9 +65,16 @@ func (f *fakeBackend) DropProgress(context.Context, platform.Session, []platform defer f.mu.Unlock() return append([]platform.DropProgress(nil), f.progress...), nil } -func (f *fakeBackend) ListEligibleChannels(context.Context, platform.Session, platform.Campaign) ([]platform.Stream, error) { +func (f *fakeBackend) ListEligibleChannels(_ context.Context, _ platform.Session, c platform.Campaign) ([]platform.Stream, error) { f.mu.Lock() defer f.mu.Unlock() + if c.ID == "" && c.Game != "" { + if f.gameLookups == nil { + f.gameLookups = map[string]int{} + } + f.gameLookups[c.Game]++ + return append([]platform.Stream(nil), f.gameLive[c.Game]...), nil + } return append([]platform.Stream(nil), f.live...), nil } func (f *fakeBackend) InventoryProgress(context.Context, platform.Session) ([]platform.Progress, error) { @@ -72,6 +86,9 @@ func (f *fakeBackend) StartWatch(_ context.Context, _ platform.Session, s platfo f.mu.Lock() defer f.mu.Unlock() f.watching = append(f.watching, s.Channel) + if f.onStartWatch != nil { + f.onStartWatch(s.Channel) + } return platform.WatchHandle{Channel: s.Channel}, nil } func (f *fakeBackend) Heartbeat(_ context.Context, h platform.WatchHandle) error { diff --git a/internal/platform/twitch/backend_test.go b/internal/platform/twitch/backend_test.go index e3a5c69..41816a0 100644 --- a/internal/platform/twitch/backend_test.go +++ b/internal/platform/twitch/backend_test.go @@ -136,3 +136,33 @@ func TestBackend_FetchAvatar_NullUser(t *testing.T) { _, err := b.FetchAvatar(context.Background(), platform.Session{AccessToken: "tok"}) require.Error(t, err) } + +// TestBackend_ListEligibleChannels_SyntheticGameCampaign pins the path +// pipeline v2 enroll discovery relies on: a synthetic open campaign (no +// ID, only a game name) has no allow-list, so the backend answers from the +// DROPS_ENABLED game directory with streams flagged DropsEnabled. +func TestBackend_ListEligibleChannels_SyntheticGameCampaign(t *testing.T) { + var slug, filters any + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + var req gqlRequest + require.NoError(t, json.NewDecoder(r.Body).Decode(&req)) + require.Equal(t, OpGameDirectory.Name, req.OperationName) + slug = req.Variables["slug"] + if opts, ok := req.Variables["options"].(map[string]any); ok { + filters = opts["systemFilters"] + } + _, _ = w.Write([]byte(dirRust)) + })) + defer srv.Close() + + b := newForTest(srv.URL) + out, err := b.ListEligibleChannels(context.Background(), platform.Session{AccessToken: "tok", ClientID: ClientTV}, + platform.Campaign{Platform: "twitch", Game: "Rust"}) + require.NoError(t, err) + assert.Equal(t, "rust", slug) + assert.Equal(t, []any{"DROPS_ENABLED"}, filters) + require.Len(t, out, 2) + assert.Equal(t, "alpha", out[0].Channel) + assert.True(t, out[0].DropsEnabled) + assert.Equal(t, "c1", out[0].ChannelID) +} From 15ecb611c88b6a749375783109ef3e19b773a083 Mon Sep 17 00:00:00 2001 From: Alejandro Date: Wed, 30 Sep 2026 02:08:17 -0400 Subject: [PATCH 53/56] fix(watcher): Kick idle backoff resets only when a watch starts Critical review finding on batch B item 3: idleWait was reset whenever a step returned nil and the state wasn't Sleeping/AwaitingConnect. pickCampaign->StatePickStream and pickStream's no-live-channel path->StatePickCampaign both return nil too, so for a Kick account with matched-but-offline campaigns idleWait was zeroed every round and the ramp never climbed past its first step -- re-running discovery every 30s instead of the old flat 5m, worse than before. idleWait now resets only when the watcher actually reaches StateWatching (a successful stream pick), not on intermediate discovery states. Twitch's fixed recheckInterval cadence is unchanged. Added a Run()-level regression test (TestKickIdleWait_RampsWhileOfflineAndResetsOnlyAfterWatching) that drives a real Watcher through repeated pick_campaign->pick_stream(no live)->sleeping rounds and asserts the idle wait climbs step to step and caps, then resets to the floor only after a round reaches StateWatching. Verified revert-proof: restoring the old reset condition fails the ramp assertions (every gap collapses to ~the floor step). Co-Authored-By: Claude Opus 5.5 (1M context) --- internal/watcher/kick_idle_backoff_test.go | 246 +++++++++++++++++++++ internal/watcher/watcher.go | 16 +- 2 files changed, 258 insertions(+), 4 deletions(-) create mode 100644 internal/watcher/kick_idle_backoff_test.go diff --git a/internal/watcher/kick_idle_backoff_test.go b/internal/watcher/kick_idle_backoff_test.go new file mode 100644 index 0000000..15a8537 --- /dev/null +++ b/internal/watcher/kick_idle_backoff_test.go @@ -0,0 +1,246 @@ +package watcher + +import ( + "context" + "sync" + "sync/atomic" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/aalejandrofer/grubdrops/internal/platform" +) + +// kickIdleRampBackend is a minimal platform.Backend double for driving +// Watcher.Run through repeated no-live-channel discovery rounds and then +// exactly one round that finds a live channel. One campaign, one +// 1-minute benefit (so a single Heartbeat completes it), and a +// toggleable "live" flag controlling ListEligibleChannels. +type kickIdleRampBackend struct { + live atomic.Bool + + mu sync.Mutex + minutes int + claimed bool +} + +var _ platform.Backend = (*kickIdleRampBackend)(nil) + +func (b *kickIdleRampBackend) Name() string { return "kick-idle-ramp-test" } + +func (b *kickIdleRampBackend) StartDeviceLogin(context.Context) (platform.DeviceChallenge, error) { + return platform.DeviceChallenge{}, nil +} + +func (b *kickIdleRampBackend) PollDeviceLogin(context.Context, platform.DeviceChallenge) (platform.Session, error) { + return platform.Session{}, nil +} + +func (b *kickIdleRampBackend) LoginViaBrowser(context.Context, platform.BrowserRPC) (platform.Session, error) { + return platform.Session{}, nil +} + +func (b *kickIdleRampBackend) RefreshSession(_ context.Context, s platform.Session) (platform.Session, error) { + return s, nil +} + +func (b *kickIdleRampBackend) ListActiveCampaigns(context.Context, platform.Session) ([]platform.Campaign, error) { + return []platform.Campaign{{ + ID: "camp1", Game: "Mock", Name: "Mock Campaign", + Benefits: []platform.DropBenefit{{ID: "drop1", CampaignID: "camp1", Name: "Mock Drop", RequiredMinutes: 1}}, + }}, nil +} + +func (b *kickIdleRampBackend) ListEligibleChannels(context.Context, platform.Session, platform.Campaign) ([]platform.Stream, error) { + if b.live.Load() { + return []platform.Stream{{Channel: "mockstreamer", DropsEnabled: true}}, nil + } + return nil, nil +} + +func (b *kickIdleRampBackend) InventoryProgress(context.Context, platform.Session) ([]platform.Progress, error) { + b.mu.Lock() + defer b.mu.Unlock() + return []platform.Progress{{BenefitID: "drop1", MinutesWatched: b.minutes, Claimed: b.claimed}}, nil +} + +func (b *kickIdleRampBackend) StartWatch(context.Context, platform.Session, platform.Stream) (platform.WatchHandle, error) { + return platform.WatchHandle{Channel: "mockstreamer"}, nil +} + +func (b *kickIdleRampBackend) Heartbeat(context.Context, platform.WatchHandle) error { + b.mu.Lock() + b.minutes++ + b.mu.Unlock() + return nil +} + +func (b *kickIdleRampBackend) StopWatch(context.Context, platform.WatchHandle) error { return nil } + +func (b *kickIdleRampBackend) Claim(context.Context, platform.Session, platform.DropBenefit) error { + b.mu.Lock() + b.claimed = true + b.mu.Unlock() + return nil +} + +// stateHit is one "state" notification with the wall-clock time it fired. +type stateHit struct { + at time.Time + state string +} + +// stateTimeline records every "state" Notify call with its arrival time, +// so a test can measure the real wall-clock gap between state changes — +// in particular, the gap between consecutive StateSleeping entries, which +// is exactly one idle-sleep wait (plus a few negligible tick-cadence +// steps). +type stateTimeline struct { + mu sync.Mutex + events []stateHit +} + +func (r *stateTimeline) Notify(_ context.Context, ev string, fields map[string]any) error { + if ev != "state" { + return nil + } + s, _ := fields["state"].(string) + r.mu.Lock() + r.events = append(r.events, stateHit{at: time.Now(), state: s}) + r.mu.Unlock() + return nil +} + +func (r *stateTimeline) snapshot() []stateHit { + r.mu.Lock() + defer r.mu.Unlock() + return append([]stateHit(nil), r.events...) +} + +func (r *stateTimeline) count(state string) int { + n := 0 + for _, e := range r.snapshot() { + if e.state == state { + n++ + } + } + return n +} + +// sleepingGaps returns the wall-clock deltas between consecutive +// StateSleeping hits, in order. +func sleepingGaps(hits []stateHit) []time.Duration { + var at []time.Time + for _, h := range hits { + if h.state == "sleeping" { + at = append(at, h.at) + } + } + gaps := make([]time.Duration, 0, len(at)-1) + for i := 1; i < len(at); i++ { + gaps = append(gaps, at[i].Sub(at[i-1])) + } + return gaps +} + +func sleepingCount(hits []stateHit) int { + n := 0 + for _, h := range hits { + if h.state == "sleeping" { + n++ + } + } + return n +} + +// TestKickIdleWait_RampsWhileOfflineAndResetsOnlyAfterWatching is the +// Run()-level regression test for the critical review finding on batch B +// item 3: idleWait must climb (30s->60s->120s->cap in production; scaled +// down here for test speed) across repeated +// pick_campaign->pick_stream(no live)->sleeping rounds, and must be reset +// ONLY by a round that reaches StateWatching — not by the intermediate +// pick_campaign/pick_stream states every offline round also passes +// through. Revert-proof: restoring the old condition ("reset whenever a +// step returns nil and the state isn't Sleeping/AwaitingConnect") zeroes +// idleWait on every pick_campaign/pick_stream transition, so the ramp +// assertions below (gap 2 clearly bigger than gap 1, etc.) fail. +func TestKickIdleWait_RampsWhileOfflineAndResetsOnlyAfterWatching(t *testing.T) { + prevSteps := kickIdleWaitSteps + kickIdleWaitSteps = []time.Duration{50 * time.Millisecond, 120 * time.Millisecond, 220 * time.Millisecond, 350 * time.Millisecond} + t.Cleanup(func() { kickIdleWaitSteps = prevSteps }) + + backend := &kickIdleRampBackend{} + timeline := &stateTimeline{} + + w := New(Config{ + AccountID: "acc-kick", + Platform: "kick", + Backend: backend, + Session: platform.Session{AccessToken: "x"}, + Notifier: timeline, + TickInterval: 2 * time.Millisecond, + HeartbeatInterval: 2 * time.Millisecond, + }) + + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + done := make(chan error, 1) + go func() { done <- w.Run(ctx) }() + + // Channel stays offline: let the ramp climb through all 4 steps (5 + // "sleeping" hits give 4 gaps: steps 0->1->2 climbing, then the cap + // held into gap 4). + require.Eventually(t, func() bool { return timeline.count("sleeping") >= 5 }, + 5*time.Second, time.Millisecond, "watcher should idle-sleep repeatedly while offline") + + preWatch := timeline.snapshot() + gaps := sleepingGaps(preWatch) + require.GreaterOrEqual(t, len(gaps), 4, "need at least 4 gaps to see the ramp climb + cap") + + // The regression itself: with the old (reverted) reset condition, + // EVERY round zeroes idleWait via the pick_campaign/pick_stream nil + // returns, so every gap collapses to ~the 50ms floor and never climbs. + assert.Greater(t, gaps[1], gaps[0]*3/2, + "gap 2 must be clearly bigger than gap 1 -- proves the ramp climbs instead of resetting every round") + assert.Greater(t, gaps[2], gaps[1]*3/2, + "gap 3 must be clearly bigger than gap 2") + // Cap: once at the top step, the wait must hold, not keep doubling. + assert.Less(t, gaps[3], gaps[2]*2, + "wait must cap at the top step, not keep growing unbounded") + + lastCappedGap := gaps[len(gaps)-1] + preWatchSleepingCount := sleepingCount(preWatch) + + // Flip the channel live: the next pick_campaign->pick_stream round + // must reach StateWatching, mine the 1-minute benefit in a single + // heartbeat, and claim it. + backend.live.Store(true) + require.Eventually(t, func() bool { return timeline.count("watching") >= 1 }, + 2*time.Second, time.Millisecond, "watcher should start watching once the channel is live") + + // After claiming, nothing is left to mine (the benefit is claimed + // forever regardless of live state), so the watcher idles again. Wait + // for a SECOND fresh sleeping hit past the one still using the + // pre-reset (capped) wait, so the gap we measure is fully governed by + // the POST-reset idleWait. + require.Eventually(t, func() bool { return sleepingCount(timeline.snapshot()) >= preWatchSleepingCount+2 }, + 5*time.Second, time.Millisecond, "watcher should idle-sleep again after claiming, twice over") + + allGaps := sleepingGaps(timeline.snapshot()) + postResetGap := allGaps[len(allGaps)-1] + + assert.Less(t, postResetGap, lastCappedGap/2, + "idle wait must reset to the ramp floor once StateWatching is reached, not resume from the capped value") + assert.Less(t, postResetGap, 100*time.Millisecond, + "post-reset gap should be close to the 50ms floor step, not mid-ramp") + + cancel() + select { + case err := <-done: + require.ErrorIs(t, err, context.Canceled) + case <-time.After(2 * time.Second): + t.Fatal("Run did not return after cancel") + } +} diff --git a/internal/watcher/watcher.go b/internal/watcher/watcher.go index a61bf21..6804bed 100644 --- a/internal/watcher/watcher.go +++ b/internal/watcher/watcher.go @@ -700,16 +700,24 @@ func (w *Watcher) Run(ctx context.Context) error { backoff := time.Duration(0) const maxBackoff = 5 * time.Minute - // Kick idle-sleep ramp (see nextKickIdleWait). Reset to 0 whenever a - // step makes real forward progress, so it restarts at the 30s floor the - // next time the account goes idle instead of resuming a stale ramp. + // Kick idle-sleep ramp (see nextKickIdleWait). Reset to 0 only when the + // watcher actually starts watching a stream — NOT on every non-idle + // step. pickCampaign->StatePickStream and pickStream's no-live-channel + // path->StatePickCampaign both return nil too, so resetting on "any + // nil, non-idle state" (the original version of this fix) zeroed + // idleWait every round for an account with matched-but-offline + // campaigns: the ramp never climbed past its first step, which is + // worse than the flat recheckInterval it replaced. Gating the reset on + // StateWatching specifically means it only fires once a live channel + // was actually found, restarting the ramp at its 30s floor the next + // time the account goes idle. idleWait := time.Duration(0) for { err := w.step(ctx) if err == nil { backoff = 0 - if s := w.State(); s != StateSleeping && s != StateAwaitingConnect { + if w.State() == StateWatching { idleWait = 0 } } else if errors.Is(err, errIdle) { From 27e45d71dea5487e778db7ce3b0fd1252161f89d Mon Sep 17 00:00:00 2001 From: Alejandro Date: Wed, 30 Sep 2026 02:20:37 -0400 Subject: [PATCH 54/56] =?UTF-8?q?fix(pipeline):=20enroll=20polish=20?= =?UTF-8?q?=E2=80=94=20dashboard=20label,=20warn=20on=20lookup=20errors,?= =?UTF-8?q?=20reset=20state=20on=20exit?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/CHANGELOG.md | 2 +- internal/api/handlers_dashboard.go | 7 + internal/api/handlers_dashboard_test.go | 26 +++ internal/i18n/locales/en.json | 1 + internal/i18n/locales/es.json | 1 + internal/i18n/locales/zh-CN.json | 1 + internal/pipeline/loop/enroll.go | 21 ++- internal/pipeline/loop/enroll_test.go | 166 ++++++++++++++++++ internal/pipeline/loop/loop.go | 5 + internal/pipeline/loop/loop_test.go | 6 + internal/web/static/css/app.css | 3 +- .../templates/dashboard_mining_columns.html | 16 ++ 12 files changed, 248 insertions(+), 7 deletions(-) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 76564c5..9588529 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -11,7 +11,7 @@ All notable changes to GrubDrops. - `drop_state` and `account_streamer_priority` tables (migration 0016), with a one-time backfill from claim history, manual marks and ghost-skips. - Accounts on the Twitch TV login share the full campaign list found by any Android-login account in the same GrubDrops, so campaigns hidden from the TV client (for example behind a global badge campaign) are still found. - A banner explains that Twitch limits campaign discovery for new (TV) logins and points to the game whitelist. -- Accounts on the Twitch TV login find campaigns by briefly watching a top drops-enabled channel for each whitelisted game when there is nothing else to mine, so Twitch enrolls them and the campaigns appear. (Pipeline v2: 10 minutes per game, each game at most once every 6 hours; dashboard state `discovering`.) +- Accounts on the Twitch TV login find campaigns by briefly watching a top drops-enabled channel for each whitelisted game when there is nothing else to mine, so Twitch enrolls them and the campaigns appear. (Pipeline v2: 10 minutes per game, each game at most once every 6 hours; dashboard state `discovering`.) (the dashboard shows these accounts as Discovering) ### Changed diff --git a/internal/api/handlers_dashboard.go b/internal/api/handlers_dashboard.go index ef2f9a0..a663c57 100644 --- a/internal/api/handlers_dashboard.go +++ b/internal/api/handlers_dashboard.go @@ -767,6 +767,13 @@ func mineCardFromSnap(a gen.Account, s watcher.Snapshot, lang string) dashMineCa c.Channel = s.Channel c.ChannelInitial = initial(s.Channel) c.ChannelURL = channelURL(a.Platform, s.Channel) + case "discovering": + c.StateSub = "mining.discovering" + c.Uptime = formatShort(time.Since(s.StartedAt), lang) + c.Channel = s.Channel + c.ChannelInitial = initial(s.Channel) + c.ChannelGame = s.CampaignGame + c.ChannelURL = channelURL(a.Platform, s.Channel) case "pick_stream": c.StateSub = "mining.scanning_channels" c.DropName = s.BenefitName diff --git a/internal/api/handlers_dashboard_test.go b/internal/api/handlers_dashboard_test.go index c5b7f57..ddeb432 100644 --- a/internal/api/handlers_dashboard_test.go +++ b/internal/api/handlers_dashboard_test.go @@ -239,3 +239,29 @@ func TestCompletedByAllConnected_NilQueriesCountsTotalOnly(t *testing.T) { assert.Equal(t, 2, total) assert.Equal(t, 0, done) } + +// TestMineCardFromSnap_Discovering mirrors the "force_watch" case: the v2 +// enroll loop reports Snapshot State "discovering" while it briefly watches +// a channel to enroll the account in a campaign (see +// internal/pipeline/loop/enroll.go). The dashboard mining row must show it +// like a live activity (channel + game), not the raw state word, so the +// i18n key resolves and the pill/label mirror force_watch's handling. +func TestMineCardFromSnap_Discovering(t *testing.T) { + acc := gen.Account{ID: "acc1", Platform: "twitch", DisplayName: "Foo", Enabled: 1} + started := time.Now().Add(-2 * time.Minute) + snap := watcher.Snapshot{ + AccountID: "acc1", + State: "discovering", + Channel: "some_channel", + ViewerCount: 42, + CampaignGame: "Some Game", + StartedAt: started, + } + card := mineCardFromSnap(acc, snap, "en") + assert.Equal(t, "discovering", card.State) + assert.Equal(t, "mining.discovering", card.StateSub) + assert.Equal(t, "some_channel", card.Channel) + assert.Equal(t, "Some Game", card.ChannelGame) + assert.Equal(t, channelURL("twitch", "some_channel"), card.ChannelURL) + assert.NotEmpty(t, card.Uptime) +} diff --git a/internal/i18n/locales/en.json b/internal/i18n/locales/en.json index 373fe05..40ea417 100644 --- a/internal/i18n/locales/en.json +++ b/internal/i18n/locales/en.json @@ -226,6 +226,7 @@ "mining.scanning": "scanning", "mining.scanning_channels": "scanning channels", "mining.force_watching": "watching", + "mining.discovering": "Discovering", "mining.picking": "picking", "mining.idle": "idle", "mining.reload_account": "Reload this account", diff --git a/internal/i18n/locales/es.json b/internal/i18n/locales/es.json index a9b5f4d..6702ad8 100644 --- a/internal/i18n/locales/es.json +++ b/internal/i18n/locales/es.json @@ -226,6 +226,7 @@ "mining.scanning": "escaneando", "mining.scanning_channels": "escaneando canales", "mining.force_watching": "viendo", + "mining.discovering": "descubriendo", "mining.picking": "eligiendo", "mining.idle": "inactivo", "mining.reload_account": "Recargar esta cuenta", diff --git a/internal/i18n/locales/zh-CN.json b/internal/i18n/locales/zh-CN.json index 3893a2f..76dfbfe 100644 --- a/internal/i18n/locales/zh-CN.json +++ b/internal/i18n/locales/zh-CN.json @@ -226,6 +226,7 @@ "mining.scanning": "扫描中", "mining.scanning_channels": "正在扫描频道", "mining.force_watching": "观看中", + "mining.discovering": "探索中", "mining.picking": "选择中", "mining.idle": "空闲", "mining.reload_account": "重新加载此账号", diff --git a/internal/pipeline/loop/enroll.go b/internal/pipeline/loop/enroll.go index a911257..a399f34 100644 --- a/internal/pipeline/loop/enroll.go +++ b/internal/pipeline/loop/enroll.go @@ -61,7 +61,12 @@ func (l *Loop) maybeEnroll(ctx context.Context) { func (l *Loop) enrollChannel(ctx context.Context, game string, now time.Time) (platform.Stream, bool) { streams, err := l.cfg.Backend.ListEligibleChannels(ctx, l.cfg.Session, platform.Campaign{Platform: l.cfg.Platform, Game: game}) if err != nil { - slog.Debug("pipeline enroll: list channels failed", "account", l.cfg.AccountID, "game", game, "err", err) + // WARN, not Debug: this is the only signal an operator gets that + // enroll discovery can't see a game's directory. The caller + // (maybeEnroll) backs the game off for LiveEvery on any failed + // lookup (enrollEmpty), so this fires at most once per game per + // backoff window, not on every idle nudge. + slog.Warn("pipeline enroll: list channels failed", "account", l.cfg.AccountID, "game", game, "err", err) return platform.Stream{}, false } sort.SliceStable(streams, func(i, j int) bool { return streams[i].ViewerCount > streams[j].ViewerCount }) @@ -129,10 +134,12 @@ func (l *Loop) finishEnroll(ctx context.Context) { before[id] = true } l.reconcile(ctx) - if l.authBlocked { - return - } - l.refreshLive(ctx) + // campaigns_found/new_drops reflect reconcile alone (refreshLive only + // probes channels for candidates already in l.rows, it adds no rows), + // so this INFO line fires even when the reconcile itself trips + // authBlocked (integrity wall) and refreshLive never runs. Without it, + // an enroll watch that ends in an auth block would leave no + // phase=stop reason=done line at all. newDrops := 0 camps := map[string]bool{} for id, r := range l.rows { @@ -144,4 +151,8 @@ func (l *Loop) finishEnroll(ctx context.Context) { slog.Info("pipeline enroll discovery", "kind", "discovery", "account", l.cfg.AccountID, "phase", "stop", "game", e.game, "channel", e.stream.Channel, "reason", "done", "campaigns_found", len(camps), "new_drops", newDrops) + if l.authBlocked { + return + } + l.refreshLive(ctx) } diff --git a/internal/pipeline/loop/enroll_test.go b/internal/pipeline/loop/enroll_test.go index 741acbc..7b641ca 100644 --- a/internal/pipeline/loop/enroll_test.go +++ b/internal/pipeline/loop/enroll_test.go @@ -1,6 +1,9 @@ package loop import ( + "context" + "errors" + "log/slog" "sync" "testing" "time" @@ -11,6 +14,63 @@ import ( "github.com/aalejandrofer/grubdrops/internal/platform" ) +// recordCapture is a slog.Handler that stores every record it sees, so +// tests can assert on a message's level and attrs together (a plain +// message->level map can't tell two records with the same message but +// different attrs apart, e.g. two "pipeline enroll discovery" lines with +// different "reason" values). +type recordCapture struct { + mu sync.Mutex + records []slog.Record +} + +func (h *recordCapture) Enabled(context.Context, slog.Level) bool { return true } +func (h *recordCapture) Handle(_ context.Context, r slog.Record) error { + h.mu.Lock() + defer h.mu.Unlock() + h.records = append(h.records, r.Clone()) + return nil +} +func (h *recordCapture) WithAttrs([]slog.Attr) slog.Handler { return h } +func (h *recordCapture) WithGroup(string) slog.Handler { return h } + +func recordAttr(r slog.Record, key string) (string, bool) { + var val string + var ok bool + r.Attrs(func(a slog.Attr) bool { + if a.Key == key { + val, ok = a.Value.String(), true + return false + } + return true + }) + return val, ok +} + +// has reports whether some captured record matches msg, level, and every +// key/value in attrs (extra attrs on the record are ignored). +func (h *recordCapture) has(msg string, level slog.Level, attrs map[string]string) bool { + h.mu.Lock() + defer h.mu.Unlock() + for _, r := range h.records { + if r.Message != msg || r.Level != level { + continue + } + ok := true + for k, want := range attrs { + got, present := recordAttr(r, k) + if !present || got != want { + ok = false + break + } + } + if ok { + return true + } + } + return false +} + // fakeClock is a settable Config.Now. type fakeClock struct { mu sync.Mutex @@ -227,3 +287,109 @@ func TestEnroll_ChannelDownCoolsItAndPicksNext(t *testing.T) { }, 2*time.Second, 2*time.Millisecond, "a1 went down: cooled, A retried on a2") assert.Equal(t, []string{"a1", "a2"}, f.watched()) } + +// TestEnroll_ChannelLookupErrorLogsWarn: a directory lookup failure is the +// only signal an operator gets that enroll discovery can't see a game's +// channels. It must log at WARN, not DEBUG, and (since maybeEnroll backs +// the game off via enrollEmpty on any failed lookup, the same as a +// legitimately-empty result) it must not fire again on a nudge that lands +// inside that backoff window. +func TestEnroll_ChannelLookupErrorLogsWarn(t *testing.T) { + f, cfg := enrollSetup(t) + cfg.Games = []string{"A"} + f.gameErr = map[string]error{"A": errors.New("directory unavailable")} + + prev := slog.Default() + t.Cleanup(func() { slog.SetDefault(prev) }) + cap := &recordCapture{} + slog.SetDefault(slog.New(cap)) + + l, _ := run(t, cfg) + require.Eventually(t, func() bool { return l.Snapshot().State == "idle" }, 2*time.Second, 2*time.Millisecond) + for i := 0; i < 5; i++ { + l.Nudge() + time.Sleep(10 * time.Millisecond) + } + + assert.True(t, cap.has("pipeline enroll: list channels failed", slog.LevelWarn, map[string]string{"game": "A"}), + "a directory lookup error must log at WARN") + f.mu.Lock() + defer f.mu.Unlock() + assert.Equal(t, 1, f.gameLookups["A"], "the failed lookup backs the game off like an empty one: not retried every nudge") +} + +// TestRun_ClearsEnrollOnExit: Run's defer must clear l.enroll itself, not +// just its timer. Without that, a re-entered Run on the same Loop would +// inherit a stale enrollRun with no timer left to end it: maybeEnroll's +// "enroll != nil" guard would block forever and updateSnapshot would +// report "discovering" forever, with nothing left running to finish it. +func TestRun_ClearsEnrollOnExit(t *testing.T) { + _, cfg := enrollSetup(t) + cfg.Games = []string{"A"} + cfg.EnrollWatch = time.Hour // outlives this test; only ctx cancel ends it + l, err := New(cfg) + require.NoError(t, err) + + ctx1, cancel1 := context.WithCancel(context.Background()) + done1 := make(chan struct{}) + go func() { _ = l.Run(ctx1); close(done1) }() + require.Eventually(t, func() bool { return l.Snapshot().State == "discovering" }, 2*time.Second, 2*time.Millisecond) + cancel1() + select { + case <-done1: + case <-time.After(2 * time.Second): + t.Fatal("first Run did not exit") + } + assert.Nil(t, l.enroll, "Run's defer must clear the enroll run on exit, not just its timer") + assert.Nil(t, l.enrollT) + assert.Nil(t, l.enrollC) + + // Re-entering Run must be able to enroll again (a stale enroll would + // wedge maybeEnroll's guard and Snapshot would never move off whatever + // state it last held). + ctx2, cancel2 := context.WithCancel(context.Background()) + defer cancel2() + done2 := make(chan struct{}) + go func() { _ = l.Run(ctx2); close(done2) }() + require.Eventually(t, func() bool { + s := l.Snapshot() + return s.State == "discovering" && s.Channel == "a1" + }, 2*time.Second, 2*time.Millisecond, "re-entered Run must be able to start a fresh enroll watch") + cancel2() + select { + case <-done2: + case <-time.After(2 * time.Second): + t.Fatal("second Run did not exit") + } +} + +// TestFinishEnroll_LogsDoneEvenWhenReconcileAuthBlocks: the post-enroll +// reconcile can trip the integrity wall (authBlocked). The phase=stop +// reason=done line (campaigns_found/new_drops from reconcile alone; +// refreshLive adds no rows) must still be logged, so an enroll watch that +// ends in an auth block doesn't vanish from the logs. Calls finishEnroll +// directly (no Run goroutine) to keep the integrity failure isolated to +// the reconcile finishEnroll triggers, not the loop's startup reconcile. +func TestFinishEnroll_LogsDoneEvenWhenReconcileAuthBlocks(t *testing.T) { + f, cfg := enrollSetup(t) + cfg.Backend = integrityBackend{f} // ListActiveCampaigns always integrity-blocks + + l, err := New(cfg) + require.NoError(t, err) + ctx := context.Background() + require.NoError(t, l.load(ctx)) + l.enroll = &enrollRun{game: "A", idx: 0, stream: platform.Stream{Channel: "a1", ViewerCount: 100}, started: l.cfg.Now()} + + prev := slog.Default() + t.Cleanup(func() { slog.SetDefault(prev) }) + cap := &recordCapture{} + slog.SetDefault(slog.New(cap)) + + l.finishEnroll(ctx) + + assert.True(t, l.authBlocked, "reconcile hit the integrity wall") + assert.Nil(t, l.enroll) + assert.True(t, cap.has("pipeline enroll discovery", slog.LevelInfo, map[string]string{ + "phase": "stop", "reason": "done", "game": "A", "channel": "a1", + }), "phase=stop reason=done must log even though the post-enroll reconcile auth-blocked") +} diff --git a/internal/pipeline/loop/loop.go b/internal/pipeline/loop/loop.go index 81d46bd..49ae8af 100644 --- a/internal/pipeline/loop/loop.go +++ b/internal/pipeline/loop/loop.go @@ -313,6 +313,11 @@ func (l *Loop) Run(ctx context.Context) error { l.catchUpT.Stop() } l.stopEnrollTimer() + // A Loop is meant to run once per Run call, but clear the enroll + // run itself (not just its timer) so a re-entered Run can never + // inherit a stale enrollRun and report "discovering" forever with + // no timer left to end it. + l.enroll = nil }() l.reconcile(ctx) if l.authBlocked { diff --git a/internal/pipeline/loop/loop_test.go b/internal/pipeline/loop/loop_test.go index ad2df06..bc9e5d5 100644 --- a/internal/pipeline/loop/loop_test.go +++ b/internal/pipeline/loop/loop_test.go @@ -50,6 +50,9 @@ type fakeBackend struct { // uses. gameLookups counts those calls per game. gameLive map[string][]platform.Stream gameLookups map[string]int + // gameErr, keyed by game, fails a synthetic-campaign ListEligibleChannels + // lookup instead of answering from gameLive (enroll's directory-error path). + gameErr map[string]error // onStartWatch, when set, runs (under mu) on every StartWatch. onStartWatch func(channel string) } @@ -73,6 +76,9 @@ func (f *fakeBackend) ListEligibleChannels(_ context.Context, _ platform.Session f.gameLookups = map[string]int{} } f.gameLookups[c.Game]++ + if err := f.gameErr[c.Game]; err != nil { + return nil, err + } return append([]platform.Stream(nil), f.gameLive[c.Game]...), nil } return append([]platform.Stream(nil), f.live...), nil diff --git a/internal/web/static/css/app.css b/internal/web/static/css/app.css index 873c74c..6428c57 100644 --- a/internal/web/static/css/app.css +++ b/internal/web/static/css/app.css @@ -999,7 +999,8 @@ .pill-force_watch { color: var(--green); border-color: rgba(111, 181, 107, 0.45); background: rgba(111, 181, 107, 0.06); } .pill-claiming { color: var(--amber); border-color: rgba(225, 176, 74, 0.45); background: rgba(225, 176, 74, 0.06); } .pill-pick_stream, - .pill-pick_campaign { color: var(--blue); border-color: rgba(111, 141, 181, 0.45); background: rgba(111, 141, 181, 0.06); } + .pill-pick_campaign, + .pill-discovering { color: var(--blue); border-color: rgba(111, 141, 181, 0.45); background: rgba(111, 141, 181, 0.06); } .pill-sleeping { color: var(--muted); border-color: var(--line); } .pill-awaiting_connect { color: #e8d44d; border-color: rgba(232, 212, 77, 0.5); background: rgba(232, 212, 77, 0.08); } .pill-needs_auth { color: var(--accent); border-color: rgba(212, 99, 30, 0.5); background: rgba(212, 99, 30, 0.08); } diff --git a/internal/web/templates/dashboard_mining_columns.html b/internal/web/templates/dashboard_mining_columns.html index 6e821f2..8b98754 100644 --- a/internal/web/templates/dashboard_mining_columns.html +++ b/internal/web/templates/dashboard_mining_columns.html @@ -46,6 +46,8 @@

{{t "mining.kick"}}

{{t "mining.claim_in_flight"}} {{else if eq .State "force_watch"}} {{t "mining.force_watching"}}{{if .Channel}} {{.Channel}}{{end}} + {{else if eq .State "discovering"}} + {{t "mining.discovering"}}{{if .Channel}} {{.Channel}}{{end}} {{else if eq .State "pick_stream"}} {{if .DropCampaign}}{{.DropCampaign}}{{end}} {{t "mining.scanning_channels"}} @@ -66,6 +68,7 @@

{{t "mining.kick"}}

{{else if eq .State "sleeping"}}{{t "mining.sleeping"}} {{else if eq .State "awaiting_connect"}}{{t "mining.awaiting_connect"}} {{else if eq .State "pick_stream"}}{{t "mining.scanning"}} + {{else if eq .State "discovering"}}{{t "mining.discovering"}} {{else if eq .State "pick_campaign"}}{{t "mining.picking"}} {{else if eq .State "idle"}}{{t "mining.idle"}} {{else}}{{.State}}{{end}} @@ -119,6 +122,19 @@

{{t "mining.kick"}}

{{t "mining.state"}} {{t "mining.force_watching"}}{{if .Uptime}} · {{.Uptime}} {{t "mining.on_stream"}}{{end}}
+ {{else if eq .State "discovering"}} +
+ {{t "mining.game"}} + {{if .ChannelGame}}{{.ChannelGame}}{{end}} +
+
+ {{t "mining.channel"}} + {{if .Channel}}{{.Channel}} ↗{{end}} +
+
+ {{t "mining.state"}} + {{t "mining.discovering"}}{{if .Uptime}} · {{.Uptime}} {{t "mining.on_stream"}}{{end}} +
{{else}}
{{t "mining.state"}} From dc58bec1624d443e696ca015447a6acb8cddf3a9 Mon Sep 17 00:00:00 2001 From: Alejandro Date: Wed, 30 Sep 2026 02:23:15 -0400 Subject: [PATCH 55/56] fix(i18n): lowercase discovering state label like other states Co-Authored-By: Claude Opus 5.5 (1M context) --- internal/i18n/locales/en.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/internal/i18n/locales/en.json b/internal/i18n/locales/en.json index 40ea417..db474aa 100644 --- a/internal/i18n/locales/en.json +++ b/internal/i18n/locales/en.json @@ -226,7 +226,7 @@ "mining.scanning": "scanning", "mining.scanning_channels": "scanning channels", "mining.force_watching": "watching", - "mining.discovering": "Discovering", + "mining.discovering": "discovering", "mining.picking": "picking", "mining.idle": "idle", "mining.reload_account": "Reload this account", From 89bc010da17f7c8301f13d6e82a3bb2b14062448 Mon Sep 17 00:00:00 2001 From: Alejandro Date: Wed, 30 Sep 2026 11:29:57 -0400 Subject: [PATCH 56/56] docs: allow Client-Integrity only on the web-client profile Co-Authored-By: Claude Opus 5.5 (1M context) --- AGENTS.md | 11 +++++++---- CLAUDE.md | 7 +++++-- 2 files changed, 12 insertions(+), 6 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index ab7d139..bbf61ed 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -63,10 +63,13 @@ proto/ buf config + gRPC defs for the sidecar — core, not dea ## Platform gotchas -- **Twitch = Android device-code OAuth over direct HTTP.** Never send a - `Client-Integrity` header — adding it is a self-inflicted integrity wall. Do - not re-implement logged-out catalog scraping; `/drops/campaigns` while logged - out is just a login wall. +- **Twitch clients: Android (legacy sessions) and TV (device-code logins since + v1.4.0) over direct HTTP.** Never send a `Client-Integrity` header on the + Android or TV client; adding it there is a self-inflicted integrity wall. The + `Client-Integrity` header is allowed ONLY on the web-client profile + (browser-captured sessions), where Twitch requires it. Do not re-implement + logged-out catalog scraping; `/drops/campaigns` while logged out is just a + login wall. - **Twitch credit beacon is gated by `HeartbeatInterval`** (1 minute credited per beacon), so anything above 60s under-credits. Keep it at 60s. Kick is immune. diff --git a/CLAUDE.md b/CLAUDE.md index 6dca36d..60d1948 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -57,8 +57,11 @@ the emoji-section + bold-subject format above. ## Platform gotchas -- **Twitch = Android device-code OAuth over direct HTTP.** Never send a - `Client-Integrity` header; adding it was a self-inflicted integrity wall. +- **Twitch clients: Android (legacy sessions) and TV (device-code logins since + v1.4.0) over direct HTTP.** Never send a `Client-Integrity` header on the + Android or TV client; adding it there was a self-inflicted integrity wall. + The `Client-Integrity` header is allowed ONLY on the web-client profile + (browser-captured sessions, rangermix-style), where Twitch requires it. Don't re-implement logged-out catalog scraping either — `/drops/campaigns` while logged out is just a login wall. - **Twitch credit beacon is gated by `HeartbeatInterval`** (1 minute credited