From 801364a4d2e6b2396cf02e2862fba5aaed45e2f8 Mon Sep 17 00:00:00 2001 From: raducupreda-art <281968083+raducupreda-art@users.noreply.github.com> Date: Sun, 27 Sep 2026 19:12:20 +0300 Subject: [PATCH] safety: gate Flow captcha generation after account block reports --- .gitignore | 2 + README.md | 91 +++++++--- agent/api/flow.py | 11 +- agent/config.py | 12 +- agent/services/flow_client.py | 163 ++++++++++++++++-- agent/worker/processor.py | 50 +++--- dashboard/src/i18n/translations.ts | 14 +- docs/ACCOUNT_SAFETY.md | 95 +++++++++++ extension/ACCOUNT_SAFETY.md | 13 ++ extension/background.js | 137 +++++++-------- extension/content.js | 9 +- extension/injected.js | 165 +++--------------- extension/manifest.json | 5 +- skills/fk-doctor.md | 37 ++-- tests/unit/extension_account_safety.test.cjs | 169 +++++++++++++++++++ tests/unit/test_flow_account_safety_api.py | 20 +++ tests/unit/test_flow_global_throttle.py | 159 ++++++++++++++++- tests/unit/test_processor.py | 21 +++ 18 files changed, 859 insertions(+), 314 deletions(-) create mode 100644 docs/ACCOUNT_SAFETY.md create mode 100644 extension/ACCOUNT_SAFETY.md create mode 100644 tests/unit/extension_account_safety.test.cjs create mode 100644 tests/unit/test_flow_account_safety_api.py diff --git a/.gitignore b/.gitignore index 35e1ef44d..94c8a8930 100644 --- a/.gitignore +++ b/.gitignore @@ -27,3 +27,5 @@ extension/_metadata/ .omc/ .agent/ flow_session_project.json +flow_generation_safety_hold.json +flow_generation_safety_hold.json.*.tmp diff --git a/README.md b/README.md index b9033f3ee..b5146e256 100644 --- a/README.md +++ b/README.md @@ -9,29 +9,59 @@ FastAPI ffmpeg Documentation - GitHub stars - GitHub issues + GitHub stars + GitHub issues DeepWiki

--- -> **Working against the new Google Flow API.** Flow moved to `flow.google.com` -> in September 2026 and stopped minting the `Bearer ya29.…` that the old -> `aisandbox-pa.googleapis.com` REST API needed. The `batchexecute` transport -> that replaces it is in place and verified end to end against the live API — -> image generation, 2K image export, and image-to-video all run green. Upgrading -> from an older Flow Kit: reload the extension (v0.3.2+) and pin +> [!CAUTION] +> **Temporary account-safety notice — read before testing Flow Kit** +> +> Use a test or disposable Google account for Flow Kit experiments; avoid an +> account whose access you cannot risk losing. CAPTCHA-bearing generation is +> disabled by default. Enabling it requires the explicit opt-in +> `FLOW_ENABLE_CAPTCHA_GENERATION=1`; use that only with a test account. +> Unusual-activity or extension-hijack signals latch a persistent generation +> safety hold. Stop generation immediately and do not keep retrying through +> Flow Kit. +> +> Reports that two Google accounts became unable to use the **manual Flow UI** +> after Flow Kit testing are a correlation, not proof that Flow Kit caused the +> blocks. Keep extension-only errors distinct from a block reproduced in the +> Flow website used manually. Public tracking: [FlowKit issue #67](https://github.com/crisng95/flowkit/issues/67). +> +> The hold is stored in `flow_generation_safety_hold.json` and survives an agent +> restart. `/api/flow/clear-hijack` cannot clear an active safety hold. +> **Google account recovery is unverified.** FlowKit's internal guard is not a +> Google recovery cooldown or recipe. Do not claim that clearing cookies, +> changing IP/networks, or waiting a fixed time will restore access. See +> [`docs/ACCOUNT_SAFETY.md`](docs/ACCOUNT_SAFETY.md) before troubleshooting. + +--- + +> **Historical migration verification — not current generation status.** During +> the September 2026 move to `flow.google.com`, the `batchexecute` transport, +> image generation, 2K image export, and image-to-video were reported verified +> against the live API. Those results predate the current account-safety gate; +> they do not mean generation is currently enabled or available by default. +> CAPTCHA-bearing generation is now disabled unless explicitly opted in with +> `FLOW_ENABLE_CAPTCHA_GENERATION=1` (test accounts only). The old +> `aisandbox-pa.googleapis.com` REST path depended on a `Bearer ya29.…` that +> Flow no longer mints. When upgrading, reload the extension (v0.3.2+) and pin > `FLOW_PROJECT_ID`; Flow Kit can no longer create the project for you. > -> Every Omni 1.1 Flash video mode is ported and live-verified: text-to-video, -> first frame, first+last frame, and references. Three capabilities remain -> unported on the **Veo** path because their payloads were never captured off -> the new UI — **video upscale**, **Veo reference-to-video**, and **Veo -> start+end-frame chaining**. They fail loudly with `UNSUPPORTED_ON_BATCH_API` -> instead of quietly producing the wrong thing. For the latter two, Omni covers -> the same shot with `model_family=omni_flash`, or `FLOW_ALLOW_DEGRADED=1` drops -> them to plain i2v; video upscale has no fallback. To restore one properly see +> That same historical verification recorded every Omni 1.1 Flash video mode +> as ported and live-verified: text-to-video, first frame, first+last frame, +> and references. It recorded three capabilities as unported on the **Veo** +> path because their payloads had not been captured off the new UI — **video +> upscale**, **Veo reference-to-video**, and **Veo start+end-frame chaining**. +> The current default remains the account-safety gate above: CAPTCHA-bearing +> generation is off unless opted in. Unsupported operations fail with +> `UNSUPPORTED_ON_BATCH_API`; for the latter two, Omni covers the same shot +> with `model_family=omni_flash`, or `FLOW_ALLOW_DEGRADED=1` drops them to plain +> i2v; video upscale has no fallback. To restore one properly see > [`docs/CAPTURE.md`](docs/CAPTURE.md). --- @@ -229,6 +259,7 @@ You can also pass `flow_project_id` per project on `POST /api/projects`. |---------|---------|--------------| | `FLOW_PROJECT_ID` | — | The Flow project every RPC is scoped to. Required. | | `FLOW_ALLOW_DEGRADED` | `0` | `1` lets scene chaining and r2v fall back to plain i2v instead of failing. | +| `FLOW_ENABLE_CAPTCHA_GENERATION` | `0` | Set to `1` to explicitly opt in to CAPTCHA-bearing generation. Disabled by default; opt in only with a test account. | | `DEFAULT_PAYGATE_TIER` | `PAYGATE_TIER_TWO` | Carried for the DB and dashboard; no longer selects a model. | ### Image API @@ -675,7 +706,7 @@ drawtext` shows whether yours has it. ## Worker Behavior -- **Server handles throttling** — worker enforces max 5 concurrent + 10s cooldown automatically. Use `POST /api/requests/batch` to submit all at once; do NOT manually batch. +- **Server handles throttling** — the worker may queue a batch, but the temporary account-safety guard permits at most one CAPTCHA-bearing request in flight when explicitly opted in. Use `POST /api/requests/batch` rather than a manual request loop. - **10s cooldown** between API calls (anti-spam, configurable via `API_COOLDOWN`) - **Reference blocking** — scene image gen refuses if any referenced entity is missing `media_id` - **Skip completed** — won't re-generate already-completed assets @@ -842,8 +873,10 @@ These arrive in the response body as `data.error.details[].reason`. The worker a | `PUBLIC_ERROR_MODEL_ACCESS_DENIED` | Tier mismatch (e.g. TIER_ONE trying Veo 3 / upscale) | Mark FAILED — auto-detect should downgrade to allowed model | | `Requested entity was not found` | Uploaded `media_id` expired (~1h TTL) | Auto-recover via `_recover_entity_not_found` — re-uploads from `image_url`, re-queues PENDING | | `Internal error encountered` | Flow backend transient 500 | Exponential backoff retry: `2^retry * 10s`, capped 300s | -| `reCAPTCHA failed` / `captcha` | Extension couldn't solve CAPTCHA | Retry up to 10× without incrementing `retry_count` (processor.py:454-464) | -| `PUBLIC_ERROR_UNUSUAL_ACTIVITY` (403, message `reCAPTCHA evaluation failed`) | Google flagged the session as bot-like — usually rapid bursts of submits, VPN/shared IP, or stale auth cookies | NOT auto-recoverable. Pause submits, clear cookies for `google.com` + `labs.google` in Chrome, sign back in at `flow.google.com`, then resubmit with ≥1s gap and ≤5 concurrent. See `/fk-doctor` for full playbook. | +| `FLOW_CAPTCHA_GENERATION_DISABLED` | CAPTCHA-bearing generation is disabled by default | Request rejected before it is sent; explicit opt-in is `FLOW_ENABLE_CAPTCHA_GENERATION=1`, for test accounts only | +| `reCAPTCHA failed` / `captcha` | CAPTCHA-bearing generation failed at the extension/Flow layer | Current request is terminal, and a persistent safety hold prevents another token request. No automatic retry. | +| `FLOW_ACCOUNT_SAFETY_HOLD` | A prior unusual-activity or extension-hijack signal latched the persistent generation hold | CAPTCHA-bearing generation is rejected; hold survives restart. `/api/flow/clear-hijack` cannot clear it. See [`docs/ACCOUNT_SAFETY.md`](docs/ACCOUNT_SAFETY.md). | +| `PUBLIC_ERROR_UNUSUAL_ACTIVITY` (often reported with 403 / `reCAPTCHA evaluation failed`) | Flow returned an unusual-activity / risk-evaluation error; the cause and account impact are not established by this message alone | Current request is terminal and the persistent hold is latched. Distinguish extension-only failure from a manual Flow UI block. Do not infer account recovery from FlowKit's internal guard. See [`docs/ACCOUNT_SAFETY.md`](docs/ACCOUNT_SAFETY.md) and `/fk-doctor`. | ### HTTP Status Codes @@ -851,7 +884,7 @@ These arrive in the response body as `data.error.details[].reason`. The worker a |--------|--------|---------|----------| | **400** | Flow API | Invalid payload, UNSAFE_GENERATION, entity not found (sometimes) | Route by `details.reason` — some are auto-recoverable, others terminal | | **401** | Flow API | Should not occur — batchexecute authenticates in the page, not with a bearer | Check the Flow tab is signed in; see `NO_AT_TOKEN` | -| **403** | Extension (`background.js:432`) | `CAPTCHA_FAILED`, `NO_FLOW_TAB`, or `MODEL_ACCESS_DENIED` | CAPTCHA → retry loop; NO_FLOW_TAB → fail (user must open Flow); tier → fail | +| **403** | Extension (`background.js:432`) | `CAPTCHA_FAILED`, `NO_FLOW_TAB`, or `MODEL_ACCESS_DENIED` | CAPTCHA generation request → terminal; NO_FLOW_TAB → fail (user must open Flow); tier → fail | | **404** | Flow API | `media_id` not found (expired upload) | Same as "Requested entity was not found" — auto re-upload | | **429** | Flow API | Rate limited / quota | Back off + retry; if `USER_QUOTA_REACHED` appears, fail | | **500** | Flow backend **or** extension fetch exception (`background.js:504`) | Transient server error OR network drop during fetch | Retry with exponential backoff | @@ -881,10 +914,17 @@ String patterns in `error_message` that the worker recognizes: `processor.py:_handle_failure` decides terminal vs retryable: -1. **Auto-recover** if message contains `"not found"` → re-upload media, mark PENDING. -2. **Transient WS** (`reconnected`/`disconnected`/`switched`) → re-queue PENDING, keep `retry_count`. -3. **CAPTCHA** → retry up to 10× without counting toward `MAX_RETRIES`. -4. **Default** → increment `retry_count`; if < `MAX_RETRIES` (5), schedule retry with `2^retry * 10s` backoff (capped 300s). Otherwise mark FAILED. +The account-safety stop rule in [`docs/ACCOUNT_SAFETY.md`](docs/ACCOUNT_SAFETY.md) +overrides these ordinary retries: stop automated generation immediately on an +unusual-activity or account/session security signal. + +1. **`[HIJACK]` or unusual activity** → mark FAILED and latch the persistent generation safety hold. +2. **`FLOW_ACCOUNT_SAFETY_HOLD` / `FLOW_CAPTCHA_GENERATION_DISABLED`** → mark FAILED; do not retry. +3. **`UNSUPPORTED_ON_BATCH_API` / `NO_FLOW_PROJECT`** → mark FAILED; do not retry. +4. **Transient WS** (`reconnected`/`disconnected`/`switched`) → re-queue PENDING, keep `retry_count`. +5. **Any other CAPTCHA / reCAPTCHA error** → mark FAILED; do not retry automatically. +6. **`"not found"`** → attempt media recovery and re-queue only if recovery succeeds. +7. **Default** → increment `retry_count`; if < `MAX_RETRIES` (5), schedule retry with `2^retry * 10s` backoff (capped 300s). Otherwise mark FAILED. ### YouTube Upload Errors @@ -907,7 +947,8 @@ From `youtube/upload.py` (HTTP errors from YouTube Data API v3): | Extension shows "No token" | Expected on the batch path — there is no bearer token any more | | `CAPTCHA_FAILED: NO_FLOW_TAB` | Open a Google Flow tab | | 403 `MODEL_ACCESS_DENIED` | Tier mismatch — check `/api/flow/credits`, downgrade model in `models.json` | -| 403 `PUBLIC_ERROR_UNUSUAL_ACTIVITY` / `reCAPTCHA evaluation failed` | Pause submits, clear cookies for `google.com` + `labs.google` in Chrome, sign back in, then resubmit with ≥1s gap and ≤5 concurrent. Switch network or wait 1–6 h if still blocked | +| CAPTCHA-bearing generation disabled | Default is off; `FLOW_ENABLE_CAPTCHA_GENERATION=1` is explicit opt-in for test accounts only | +| `PUBLIC_ERROR_UNUSUAL_ACTIVITY`, extension hijack, CAPTCHA error, or active safety hold | Stop generation. These signals latch a persistent hold. Distinguish extension-only failure from a manual Flow UI block. Google recovery is unverified; `/api/flow/clear-hijack` cannot clear the hold. See [`docs/ACCOUNT_SAFETY.md`](docs/ACCOUNT_SAFETY.md). | | Scene images inconsistent | Check all refs have UUID `media_id` — run `/fk-fix-uuids` | | `media_id` starts with `CAMS...` | Run `/fk-fix-uuids` to extract UUID from URL | | Upscale "permission denied" | Requires `PAYGATE_TIER_TWO` account | diff --git a/agent/api/flow.py b/agent/api/flow.py index e74647cef..8af3fce1a 100644 --- a/agent/api/flow.py +++ b/agent/api/flow.py @@ -185,7 +185,7 @@ async def extension_status(): "flow_key_present": client._flow_key is not None, "generation_throttle": { "min_interval_s": FLOW_GENERATION_MIN_INTERVAL_S, - "max_concurrent": FLOW_GENERATION_MAX_CONCURRENT, + "max_concurrent": 1, # temporary account-safety cap in FlowClient "unusual_activity_cooldown_s": FLOW_UNUSUAL_ACTIVITY_COOLDOWN_S, **client.generation_guard_status, }, @@ -195,13 +195,12 @@ async def extension_status(): @router.post("/clear-hijack") async def clear_hijack_cooldown(): - """Reset the generation cooldown triggered by extension_hijack_detected. - - Call this after deploying the bypass fix to immediately resume generation - without waiting for the cooldown to expire. - """ + """Clear only the legacy timer, never an account-safety hold.""" import time as _time client = get_flow_client() + if client.generation_guard_status["safety_hold_active"]: + raise HTTPException(409, "FLOW_ACCOUNT_SAFETY_HOLD: this endpoint cannot reset the hold; " + "review manual Flow UI status and the safety state before restarting") old_until = client._generation_unusual_until client._generation_unusual_until = 0.0 was_active = old_until > 0.0 and old_until > _time.monotonic() diff --git a/agent/config.py b/agent/config.py index bebbcc954..884b892ce 100644 --- a/agent/config.py +++ b/agent/config.py @@ -33,6 +33,17 @@ # both drop to plain i2v off the start frame. Upscale has no fallback. FLOW_ALLOW_DEGRADED = os.environ.get("FLOW_ALLOW_DEGRADED", "0") == "1" +# Temporary account-safety gate for CAPTCHA-bearing generation. Google may +# block manual Flow generation after an unusual-activity response; opt in only +# with an account whose loss of generation access would be acceptable. +FLOW_ENABLE_CAPTCHA_GENERATION = os.environ.get("FLOW_ENABLE_CAPTCHA_GENERATION", "0") == "1" +FLOW_GENERATION_SAFETY_FILE = Path(os.environ.get( + "FLOW_GENERATION_SAFETY_FILE", BASE_DIR / "flow_generation_safety_hold.json")) +# Operator-reported manual UI state, never inferred from an extension error. +FLOW_MANUAL_UI_STATUS = os.environ.get("FLOW_MANUAL_UI_STATUS", "unknown").lower() +if FLOW_MANUAL_UI_STATUS not in {"unknown", "works", "blocked"}: + FLOW_MANUAL_UI_STATUS = "unknown" + # Process-wide guard for every CAPTCHA-bearing generation submit, including # direct API calls that bypass the background worker's limiter. FLOW_GENERATION_MIN_INTERVAL_S = max( @@ -124,4 +135,3 @@ def _load_suno_key() -> str: SUNO_CALLBACK_URL = os.environ.get("SUNO_CALLBACK_URL", f"http://{API_HOST}:{API_PORT}/api/music/callback") SUNO_POLL_INTERVAL = int(os.environ.get("SUNO_POLL_INTERVAL", "5")) SUNO_POLL_TIMEOUT = int(os.environ.get("SUNO_POLL_TIMEOUT", "600")) - diff --git a/agent/services/flow_client.py b/agent/services/flow_client.py index c771b4253..8e049e5b7 100644 --- a/agent/services/flow_client.py +++ b/agent/services/flow_client.py @@ -20,8 +20,10 @@ import asyncio import json import logging +import os import time import uuid +from pathlib import Path from typing import Optional from agent.config import ( @@ -30,6 +32,8 @@ DEFAULT_PAYGATE_TIER, FLOW_GENERATION_MIN_INTERVAL_S, FLOW_GENERATION_MAX_CONCURRENT, FLOW_UNUSUAL_ACTIVITY_COOLDOWN_S, + FLOW_ENABLE_CAPTCHA_GENERATION, FLOW_GENERATION_SAFETY_FILE, + FLOW_MANUAL_UI_STATUS, ) from agent import config as _config from agent.services import flow_batch as fb @@ -66,12 +70,16 @@ def __init__(self): self._operation_projects: dict[str, str] = {} self._operation_media: dict[str, str] = {} self._operation_polls: dict[str, int] = {} - self._generation_slots = asyncio.Semaphore(FLOW_GENERATION_MAX_CONCURRENT) + # During the account-safety investigation, only one token-bearing + # request may be in flight per agent, regardless of old user settings. + self._generation_slots = asyncio.Semaphore(1) self._generation_rate_gate = asyncio.Lock() self._generation_last_submit_at = 0.0 self._generation_unusual_until = 0.0 self._generation_last_unusual_at: Optional[float] = None self._generation_last_unusual_rpc: Optional[str] = None + self._generation_safety_owner = uuid.uuid4().hex + self._generation_safety_hold = self._read_generation_safety_hold() # WS stats self._ws_connect_count = 0 self._ws_disconnect_count = 0 @@ -206,14 +214,110 @@ def connected(self) -> bool: @property def generation_guard_status(self) -> dict: + if self._generation_safety_hold is None: + self._generation_safety_hold = self._read_generation_safety_hold() remaining = max(0.0, self._generation_unusual_until - time.monotonic()) return { + "captcha_generation_opted_in": FLOW_ENABLE_CAPTCHA_GENERATION, + "safety_hold_active": self._generation_safety_hold is not None, + "safety_hold_reason": (self._generation_safety_hold or {}).get("reason"), + "manual_flow_ui_status": FLOW_MANUAL_UI_STATUS, "cooldown_active": remaining > 0, "cooldown_remaining_s": round(remaining, 3), "last_unusual_activity_at": self._generation_last_unusual_at, "last_unusual_activity_rpc": self._generation_last_unusual_rpc, } + def _read_generation_safety_hold(self) -> dict | None: + path = Path(FLOW_GENERATION_SAFETY_FILE) + if not path.exists(): + return None + try: + record = json.loads(path.read_text(encoding="utf-8")) + if isinstance(record, dict) and record.get("state") == "inflight": + if record.get("owner") == self._generation_safety_owner: + return None + # An interrupted or concurrent process may have submitted a + # request whose outcome is unknown. Never guess it was safe. + return {"reason": "interrupted_generation", "rpcid": record.get("rpcid")} + if isinstance(record, dict) and record.get("reason"): + return record + except (OSError, ValueError): + pass + # An unreadable hold is still a hold; restarting must not clear it. + return {"reason": "unreadable_safety_state"} + + def _begin_generation_attempt(self, rpcid: str) -> bool: + """Durably mark an in-flight submit before any CAPTCHA can be minted. + + An unclean restart or failed hold write leaves this marker behind; a + new client treats it as an account-safety hold. Exclusive creation also + prevents another agent process from sending at the same time. + """ + path = Path(FLOW_GENERATION_SAFETY_FILE) + record = {"schema": "flowkit.generation-safety-inflight.v1", "state": "inflight", + "owner": self._generation_safety_owner, "rpcid": rpcid, + "started_at": time.time()} + try: + path.parent.mkdir(parents=True, exist_ok=True) + with path.open("x", encoding="utf-8") as stream: + json.dump(record, stream) + stream.flush() + os.fsync(stream.fileno()) + return True + except FileExistsError: + self._generation_safety_hold = (self._read_generation_safety_hold() + or {"reason": "generation_state_conflict"}) + except OSError: + self._generation_safety_hold = {"reason": "safety_state_unwritable"} + logger.exception("Could not persist pre-submit safety marker; generation was not sent") + return False + + def _finish_generation_attempt(self) -> bool: + """Remove only this client's successful in-flight marker.""" + path = Path(FLOW_GENERATION_SAFETY_FILE) + try: + record = json.loads(path.read_text(encoding="utf-8")) + if record.get("state") != "inflight" or record.get("owner") != self._generation_safety_owner: + self._generation_safety_hold = {"reason": "generation_state_conflict"} + return False + path.unlink() + return True + except (OSError, ValueError): + self._generation_safety_hold = {"reason": "safety_state_unwritable"} + logger.exception("Could not clear pre-submit safety marker; generation remains held") + return False + + def _latch_generation_safety(self, reason: str, rpcid: str) -> None: + if self._generation_safety_hold is not None: + return + record = {"schema": "flowkit.generation-safety-hold.v1", "reason": reason, + "rpcid": rpcid, "detected_at": time.time(), + "manual_flow_ui_status": FLOW_MANUAL_UI_STATUS} + self._generation_safety_hold = record + path = Path(FLOW_GENERATION_SAFETY_FILE) + try: + path.parent.mkdir(parents=True, exist_ok=True) + temporary = path.with_name(f"{path.name}.{uuid.uuid4().hex}.tmp") + temporary.write_text(json.dumps(record, indent=2), encoding="utf-8") + temporary.replace(path) + except OSError: + logger.exception("Generation safety hold could not be persisted; stop this agent before restart") + logger.critical("FLOW_ACCOUNT_SAFETY_HOLD reason=%s rpcid=%s manual_flow_ui_status=%s; " + "no more CAPTCHA-bearing requests will be sent", reason, rpcid, FLOW_MANUAL_UI_STATUS) + + def _generation_safety_rejection(self) -> dict | None: + if self._generation_safety_hold is None: + self._generation_safety_hold = self._read_generation_safety_hold() + if self._generation_safety_hold is not None: + reason = self._generation_safety_hold.get("reason", "unknown") + return {"status": 423, "error": f"FLOW_ACCOUNT_SAFETY_HOLD: {reason}; " + "manual review required; do not retry automatically"} + if not FLOW_ENABLE_CAPTCHA_GENERATION: + return {"status": 423, "error": "FLOW_CAPTCHA_GENERATION_DISABLED: temporary account-safety " + "default; opt in explicitly only with a test account"} + return None + @property def ws_stats(self) -> dict: uptime = None @@ -518,10 +622,25 @@ async def batch_rpc(self, rpcid: str, freq: str, if match: params["match"] = match - is_generation = captcha_action in {fb.CAPTCHA_IMAGE, fb.CAPTCHA_VIDEO} + captcha_rpcs = { + fb.RPC_GEN_IMAGE, fb.RPC_GEN_VIDEO, fb.RPC_GEN_VIDEO_TEXT, + fb.RPC_GEN_VIDEO_FIRST_LAST, fb.RPC_GEN_VIDEO_REFERENCES, + fb.RPC_UPLOAD_IMAGE, fb.RPC_UPSCALE_IMAGE, + } + is_generation = (captcha_action in {fb.CAPTCHA_IMAGE, fb.CAPTCHA_VIDEO} + or rpcid in captcha_rpcs or fb.CAPTCHA_SLOT in freq) if not is_generation: return await self._send("batch_rpc", params, timeout=timeout) + rejection = self._generation_safety_rejection() + if rejection is not None: + return rejection + if captcha_action not in {fb.CAPTCHA_IMAGE, fb.CAPTCHA_VIDEO}: + return {"status": 400, "error": "CAPTCHA_ACTION_REQUIRED: generation RPC has no CAPTCHA action"} + # The extension independently rejects CAPTCHA requests without this + # explicit signal. Non-generation RPCs never carry it. + params["accountSafetyOptIn"] = True + now = time.monotonic() if now < self._generation_unusual_until: remaining = max(1, int(self._generation_unusual_until - now + 0.999)) @@ -536,6 +655,9 @@ async def batch_rpc(self, rpcid: str, freq: str, await self._generation_slots.acquire() try: async with self._generation_rate_gate: + rejection = self._generation_safety_rejection() + if rejection is not None: + return rejection now = time.monotonic() if now < self._generation_unusual_until: remaining = max(1, int(self._generation_unusual_until - now + 0.999)) @@ -551,9 +673,20 @@ async def batch_rpc(self, rpcid: str, freq: str, ) if delay > 0: await asyncio.sleep(delay) + # A different request/process may have latched a hold during + # the pacing wait. Check again immediately before dispatch. + rejection = self._generation_safety_rejection() + if rejection is not None: + return rejection + if not self._begin_generation_attempt(rpcid): + return self._generation_safety_rejection() self._generation_last_submit_at = time.monotonic() - result = await self._send("batch_rpc", params, timeout=timeout) + try: + result = await self._send("batch_rpc", params, timeout=timeout) + except Exception: + self._latch_generation_safety("generation_transport_error", rpcid) + raise blob = f"{result.get('error', '')} {result.get('data', '')}" is_hijack = "extension_hijack" in blob.lower() is_unusual = ( @@ -561,10 +694,9 @@ async def batch_rpc(self, rpcid: str, freq: str, or "unusual activity" in blob.lower() ) if is_hijack: - # x2a trap: the token was poisoned with action - # "extension_hijack_detected". This is a system-level trap, - # NOT an account issue — don't apply the long cooldown. - # Short pause lets the bypass re-initialise on next tab. + # A hijack signal may coincide with account-level restrictions; + # its cause is not established. Hold all future generation. + self._latch_generation_safety("extension_hijack_detected", rpcid) self._generation_unusual_until = max( self._generation_unusual_until, time.monotonic() + 30.0, @@ -572,8 +704,7 @@ async def batch_rpc(self, rpcid: str, freq: str, self._generation_last_unusual_at = time.time() self._generation_last_unusual_rpc = rpcid logger.error( - "[HIJACK] extension_hijack_detected — captcha bypass may " - "have failed; pausing generation submits for 30s" + "[HIJACK] extension_hijack_detected; account-safety hold active" ) # Tag the result so processor can distinguish it if isinstance(result.get("error"), str): @@ -581,6 +712,7 @@ async def batch_rpc(self, rpcid: str, freq: str, else: result["error"] = "[HIJACK] extension_hijack_detected" elif is_unusual: + self._latch_generation_safety("PUBLIC_ERROR_UNUSUAL_ACTIVITY", rpcid) self._generation_unusual_until = max( self._generation_unusual_until, time.monotonic() + FLOW_UNUSUAL_ACTIVITY_COOLDOWN_S, @@ -588,9 +720,18 @@ async def batch_rpc(self, rpcid: str, freq: str, self._generation_last_unusual_at = time.time() self._generation_last_unusual_rpc = rpcid logger.warning( - "Google unusual-activity block detected; pausing generation submits for %.0fs", - FLOW_UNUSUAL_ACTIVITY_COOLDOWN_S, + "Google unusual-activity signal detected; account-safety hold active " + "(manual_flow_ui_status=%s)", FLOW_MANUAL_UI_STATUS, ) + elif result.get("error") or (isinstance(result.get("status"), int) + and result["status"] >= 400): + # A failed or ambiguous CAPTCHA request is not retried. Even + # if the hold write fails, the pre-submit marker survives. + self._latch_generation_safety("generation_response_error", rpcid) + original_error = str(result.get("error") or f"HTTP {result.get('status')}")[:240] + result["error"] = f"FLOW_ACCOUNT_SAFETY_HOLD: {original_error}" + elif not self._finish_generation_attempt(): + return self._generation_safety_rejection() return result finally: self._generation_slots.release() diff --git a/agent/worker/processor.py b/agent/worker/processor.py index c736410aa..e24c0323d 100644 --- a/agent/worker/processor.py +++ b/agent/worker/processor.py @@ -434,25 +434,15 @@ async def _handle_failure(rid: str, req: dict, result: dict, retry_after: dict = if isinstance(error_msg, dict): error_msg = json.dumps(error_msg)[:200] - # Auto-recover expired media by re-uploading - if "not found" in str(error_msg).lower(): - recovered = await _recover_entity_not_found(req) - if recovered: - logger.info("Request %s: recovered expired media, retrying", rid[:8]) - await crud.update_request(rid, status="PENDING", error_message=f"recovered: {error_msg}") - return - error_lower = str(error_msg).lower() - # [HIJACK] extension_hijack_detected — the captcha bypass failed and the - # token was poisoned by x2a. This is a system-level trap, NOT an account - # issue. Don't burn retry count — the bypass may succeed on the next - # attempt once the tab re-initializes. + # Neither the account impact nor the cause of this signal is established. + # A later automatic retry would mint another token, so fail this request. if "[hijack]" in error_lower or "extension_hijack" in error_lower: - await crud.update_request(rid, status="PENDING", error_message=str(error_msg)) + await crud.update_request(rid, status="FAILED", error_message=str(error_msg)) + await _mark_scene_failed(req) logger.error( - "Request %s [HIJACK] captcha bypass failed — will retry without " - "counting (system error, not account error): %s", + "Request %s FAILED (hijack signal; no automatic retry): %s", rid[:8], error_msg, ) return @@ -469,6 +459,12 @@ async def _handle_failure(rid: str, req: dict, result: dict, retry_after: dict = rid[:8], error_msg) return + if "flow_account_safety_hold" in error_lower or "flow_captcha_generation_disabled" in error_lower: + await crud.update_request(rid, status="FAILED", error_message=str(error_msg)) + await _mark_scene_failed(req) + logger.error("Request %s FAILED (generation safety gate): %s", rid[:8], error_msg) + return + # A capability the batch path does not have, or a missing Flow project, is # a configuration answer — not something a retry can reach. Fail it once. if "unsupported_on_batch_api" in error_lower or "no_flow_project" in error_lower: @@ -483,17 +479,21 @@ async def _handle_failure(rid: str, req: dict, result: dict, retry_after: dict = logger.info("Request %s transient WS error, will retry (no retry increment): %s", rid[:8], error_msg) return - # reCAPTCHA errors: retry up to 10 times — deferred dict in main loop handles delay + # A repeated mint can add more risk signals. A CAPTCHA failure is terminal + # for this request; the operator can investigate without an automatic loop. if "captcha" in error_lower or "recaptcha" in error_lower: - retry = req.get("retry_count", 0) + 1 - if retry < 10: - await crud.update_request(rid, status="PENDING", retry_count=retry, error_message=str(error_msg)) - logger.warning("Request %s reCAPTCHA failed (retry %d/10), will retry", rid[:8], retry) - return - else: - await crud.update_request(rid, status="FAILED", error_message=str(error_msg)) - await _mark_scene_failed(req) - logger.error("Request %s FAILED after 10 reCAPTCHA retries: %s", rid[:8], error_msg) + await crud.update_request(rid, status="FAILED", error_message=str(error_msg)) + await _mark_scene_failed(req) + logger.error("Request %s FAILED (CAPTCHA error; no automatic retry): %s", rid[:8], error_msg) + return + + # Only ordinary missing media may be re-uploaded. Safety and CAPTCHA + # signals above take precedence even if their wording contains "not found". + if "not found" in error_lower: + recovered = await _recover_entity_not_found(req) + if recovered: + logger.info("Request %s: recovered expired media, retrying", rid[:8]) + await crud.update_request(rid, status="PENDING", error_message=f"recovered: {error_msg}") return retry = req.get("retry_count", 0) + 1 diff --git a/dashboard/src/i18n/translations.ts b/dashboard/src/i18n/translations.ts index c1d240b39..fe9221f23 100644 --- a/dashboard/src/i18n/translations.ts +++ b/dashboard/src/i18n/translations.ts @@ -193,7 +193,7 @@ const en = { 'guide.trouble4.problem': '"Extension not connected"', 'guide.trouble4.solution': 'Go to chrome://extensions and click the reload icon on the Flow Kit card — the extension will reconnect automatically.', 'guide.trouble5.problem': '403 PUBLIC_ERROR_UNUSUAL_ACTIVITY error', - 'guide.trouble5.solution': 'Pause submissions → chrome://settings/cookies → remove cookies for google.com and labs.google → reopen flow.google.com, sign in, solve the captcha if shown → resubmit more slowly.', + 'guide.trouble5.solution': 'Stop all automated generation. An extension-only error does not confirm the manual Flow UI is blocked. If the manual UI is blocked, recovery and any cooldown are unverified.', 'guide.trouble6.problem': 'curl: (7) Failed to connect to 127.0.0.1:8100', 'guide.trouble6.solution': "The agent isn't running — run python -m agent.main.", @@ -516,7 +516,7 @@ const vi: Partial> = { 'guide.trouble4.problem': '"Extension not connected"', 'guide.trouble4.solution': 'Vào chrome://extensions, bấm reload icon trên thẻ Flow Kit — extension sẽ tự kết nối lại.', 'guide.trouble5.problem': 'Lỗi 403 PUBLIC_ERROR_UNUSUAL_ACTIVITY', - 'guide.trouble5.solution': 'Tạm dừng gửi request → chrome://settings/cookies → xoá cookie của google.com và labs.google → mở lại flow.google.com, đăng nhập, giải captcha nếu có → gửi lại chậm hơn.', + 'guide.trouble5.solution': 'Dừng mọi hoạt động tạo tự động. Lỗi chỉ xuất hiện ở extension không xác nhận Flow UI thủ công bị chặn. Nếu UI thủ công bị chặn, cách khôi phục và thời gian chờ đều chưa được xác minh.', 'guide.trouble6.problem': 'curl: (7) Failed to connect to 127.0.0.1:8100', 'guide.trouble6.solution': 'Agent chưa chạy — chạy python -m agent.main.', @@ -828,7 +828,7 @@ const hi: Partial> = { 'guide.trouble4.problem': '"Extension not connected"', 'guide.trouble4.solution': 'chrome://extensions पर जाएं और Flow Kit कार्ड पर reload आइकन क्लिक करें — extension स्वतः फिर से जुड़ जाएगा।', 'guide.trouble5.problem': '403 PUBLIC_ERROR_UNUSUAL_ACTIVITY एरर', - 'guide.trouble5.solution': 'सबमिशन रोकें → chrome://settings/cookies → google.com और labs.google के कुकीज़ हटाएं → flow.google.com फिर से खोलें, साइन इन करें, दिखे तो captcha हल करें → धीरे-धीरे फिर से सबमिट करें।', + 'guide.trouble5.solution': 'सभी स्वचालित जनरेशन रोकें। केवल extension में दिखी त्रुटि से यह पुष्टि नहीं होती कि मैन्युअल Flow UI ब्लॉक है। यदि मैन्युअल UI ब्लॉक है, तो रिकवरी और कोई भी कूलडाउन सत्यापित नहीं हैं।', 'guide.trouble6.problem': 'curl: (7) Failed to connect to 127.0.0.1:8100', 'guide.trouble6.solution': 'एजेंट नहीं चल रहा — python -m agent.main चलाएं।', @@ -1140,7 +1140,7 @@ const id: Partial> = { 'guide.trouble4.problem': '"Extension not connected"', 'guide.trouble4.solution': 'Buka chrome://extensions dan klik ikon reload pada kartu Flow Kit — ekstensi akan terhubung kembali secara otomatis.', 'guide.trouble5.problem': 'Error 403 PUBLIC_ERROR_UNUSUAL_ACTIVITY', - 'guide.trouble5.solution': 'Hentikan sementara pengiriman → chrome://settings/cookies → hapus cookie untuk google.com dan labs.google → buka kembali flow.google.com, masuk, selesaikan captcha jika muncul → kirim ulang lebih lambat.', + 'guide.trouble5.solution': 'Hentikan semua pembuatan otomatis. Error yang hanya terjadi di ekstensi tidak memastikan Flow UI manual diblokir. Jika UI manual diblokir, pemulihan dan masa tunggu apa pun belum terverifikasi.', 'guide.trouble6.problem': 'curl: (7) Failed to connect to 127.0.0.1:8100', 'guide.trouble6.solution': 'Agent tidak berjalan — jalankan python -m agent.main.', @@ -1452,7 +1452,7 @@ const zh: Partial> = { 'guide.trouble4.problem': '"Extension not connected"', 'guide.trouble4.solution': '前往 chrome://extensions，点击 Flow Kit 卡片上的重新加载图标 — 扩展会自动重新连接。', 'guide.trouble5.problem': '403 PUBLIC_ERROR_UNUSUAL_ACTIVITY 错误', - 'guide.trouble5.solution': '暂停提交 → 打开 chrome://settings/cookies → 删除 google.com 和 labs.google 的 cookie → 重新打开 flow.google.com，登录，如出现验证码请完成 → 放慢速度重新提交。', + 'guide.trouble5.solution': '停止所有自动生成。仅扩展程序报错并不能确认手动 Flow UI 已被阻止。如果手动 UI 被阻止，恢复方法和任何冷却时间均未经过验证。', 'guide.trouble6.problem': 'curl: (7) Failed to connect to 127.0.0.1:8100', 'guide.trouble6.solution': 'Agent 未运行 — 请运行 python -m agent.main。', @@ -1764,7 +1764,7 @@ const ko: Partial> = { 'guide.trouble4.problem': '"Extension not connected"', 'guide.trouble4.solution': 'chrome://extensions로 이동하여 Flow Kit 카드의 새로고침 아이콘을 클릭하세요 — 확장 프로그램이 자동으로 다시 연결됩니다.', 'guide.trouble5.problem': '403 PUBLIC_ERROR_UNUSUAL_ACTIVITY 오류', - 'guide.trouble5.solution': '제출을 일시 중지 → chrome://settings/cookies 열기 → google.com 및 labs.google의 쿠키 삭제 → flow.google.com를 다시 열고 로그인, 캡차가 표시되면 해결 → 더 천천히 다시 제출하세요.', + 'guide.trouble5.solution': '모든 자동 생성을 중단하세요. 확장 프로그램에서만 발생한 오류로는 수동 Flow UI가 차단됐는지 확인할 수 없습니다. 수동 UI가 차단된 경우 복구 방법과 대기 시간은 검증되지 않았습니다.', 'guide.trouble6.problem': 'curl: (7) Failed to connect to 127.0.0.1:8100', 'guide.trouble6.solution': 'Agent가 실행 중이 아닙니다 — python -m agent.main을 실행하세요.', @@ -2076,7 +2076,7 @@ const ja: Partial> = { 'guide.trouble4.problem': '"Extension not connected"', 'guide.trouble4.solution': 'chrome://extensions を開き、Flow Kit カードの再読み込みアイコンをクリックしてください — 拡張機能が自動的に再接続されます。', 'guide.trouble5.problem': '403 PUBLIC_ERROR_UNUSUAL_ACTIVITY エラー', - 'guide.trouble5.solution': '送信を一時停止 → chrome://settings/cookies を開く → google.com と labs.google のCookieを削除 → flow.google.com を再度開いてサインインし、captchaが表示されたら解決 → ゆっくりと再送信してください。', + 'guide.trouble5.solution': 'すべての自動生成を停止してください。拡張機能だけのエラーでは、手動のFlow UIがブロックされたとは確認できません。手動UIがブロックされている場合、復旧方法や待機時間は未検証です。', 'guide.trouble6.problem': 'curl: (7) Failed to connect to 127.0.0.1:8100', 'guide.trouble6.solution': 'Agent が実行されていません — python -m agent.main を実行してください。', diff --git a/docs/ACCOUNT_SAFETY.md b/docs/ACCOUNT_SAFETY.md new file mode 100644 index 000000000..b02bb748d --- /dev/null +++ b/docs/ACCOUNT_SAFETY.md @@ -0,0 +1,95 @@ +# Temporary Flow account-safety guidance + +This notice is temporary and should remain visible while the cause and recovery +status of the reported access problems are unresolved. + +## Implemented FlowKit safeguards + +- CAPTCHA-bearing generation requests are **disabled by default**. The opt-in + is `FLOW_ENABLE_CAPTCHA_GENERATION=1`; use it only with a test/disposable + account. Without opt-in, FlowKit rejects the request before sending it. +- An unusual-activity or extension-hijack signal latches + `flow_generation_safety_hold.json` (or the configured + `FLOW_GENERATION_SAFETY_FILE`). The hold blocks later CAPTCHA-bearing + generation and survives agent restarts. FlowKit writes an in-flight marker + **before** requesting a token; an interrupted request or failed hold write + leaves that marker behind and the next process treats it as a hold. If the + file cannot be read, FlowKit also treats the hold as active. +- `/api/flow/clear-hijack` **cannot clear an active safety hold**; it returns + HTTP 409. It only clears a legacy in-memory timer when no persistent hold is + active. Do not describe it as a hold reset. +- A generic CAPTCHA/reCAPTCHA or transport error marks that request failed, + latches the persistent hold, and is not automatically retried. A successful + request clears only its own in-flight marker. +- The safety gate applies to CAPTCHA-bearing generation. Non-generation + metadata and polling RPCs are outside that generation gate; this does not + establish whether the manual Flow UI works for the account. + +`FLOW_MANUAL_UI_STATUS` is operator-reported metadata (`unknown` by default, +or `works` / `blocked`); FlowKit does not infer manual UI status from an +extension error. FlowKit's internal timers and holds are safety controls, not +Google account recovery cooldowns. + +## Before testing + +- Use a test or disposable Google account. Do not experiment with an account + whose access you cannot risk losing. +- Stop all automated Flow generation at the first unusual-activity response, + suspected account/session hijack, unexpected sign-in or security prompt, or + other unexpected account-security signal. Do not continue by retrying through + FlowKit or another automation path. + +## Describe the affected layer accurately + +- **Extension/agent-only failure:** the extension, local agent, or an automated + request reports an error. This does not by itself show that Flow's manual + website is blocked for the account. +- **Manual Flow UI blocked:** the account cannot use the Flow website manually, + independently of an extension request. Record this as a separate observation + only when it has actually been reported or observed. Do not repeatedly retry + generation to establish the distinction. +- **Unknown:** if the available report does not say whether manual Flow access + is blocked, preserve that uncertainty. Do not label an extension error as an + account block. + +Reports that two Google accounts became unable to use the **manual Flow UI** +after FlowKit testing are temporally associated reports. That correlation does +not prove FlowKit caused the blocks. The related upstream report is public in +[FlowKit issue #67](https://github.com/crisng95/flowkit/issues/67). + +## Recovery status + +No Google account recovery method or recovery cooldown has been verified for +these reports. The FlowKit safety hold is persistent application state, not a +Google recovery measure. Do not say that clearing cookies, changing IP address +or network, or waiting a fixed time will restore account access. Do not suggest +that a CAPTCHA retry, slower cadence, or switching accounts repairs an affected +account. + +### Local hold reset + +There is deliberately **no API reset** for the persistent safety hold. +`/api/flow/clear-hijack` cannot clear it. Only after an operator has completed a +manual review, is using a disposable/test account, and has independently +verified that the manual Flow UI works may the operator explicitly remove or +rename the local hold file (`flow_generation_safety_hold.json`, or the path set +by `FLOW_GENERATION_SAFETY_FILE`). The operator must then restart FlowKit with +`FLOW_ENABLE_CAPTCHA_GENERATION=1` to opt in to CAPTCHA-bearing generation. + +This procedure clears only FlowKit's local hold. It does not clear Google's +account state and cannot restore Google access. If any prerequisite is unknown +or unmet, leave the hold in place. + +Stop automated requests and preserve the observed error, approximate time, +FlowKit/extension version, and whether manual Flow access was reported blocked. +Never include passwords, cookies, access tokens, or other account secrets in a +bug report. For account access, follow Google's official account/help process; +FlowKit documentation cannot restore or verify account access. + +## Reporting language + +Prefer: “Two accounts were reported unable to use the manual Flow UI after +FlowKit testing; timing is correlated, and causation and recovery are +unverified.” Avoid stating or implying that FlowKit caused a Google account +block, that the extension alone proves a manual UI block, or that a particular +remedy or wait period restores access. diff --git a/extension/ACCOUNT_SAFETY.md b/extension/ACCOUNT_SAFETY.md new file mode 100644 index 000000000..0ea261956 --- /dev/null +++ b/extension/ACCOUNT_SAFETY.md @@ -0,0 +1,13 @@ +# Temporary account-safety warning + +> **Use Flow Kit generation only with a test or disposable Google account.** +> CAPTCHA-bearing generation is disabled by default. Explicit opt-in requires +> `FLOW_ENABLE_CAPTCHA_GENERATION=1` and should be limited to test accounts. +> Unusual-activity or extension-hijack signals create a persistent local +> generation safety hold; do not retry generation after such a signal. + +After updating the extension, reload any already-open Flow tabs. Existing tabs +may still have previously injected scripts until reloaded. + +For the hold behavior, manual-UI distinction, and recovery limits, see the +[main account-safety guide](../docs/ACCOUNT_SAFETY.md). diff --git a/extension/background.js b/extension/background.js index 889acdc5d..24c611dfa 100644 --- a/extension/background.js +++ b/extension/background.js @@ -2,7 +2,8 @@ * Flow Kit — Chrome Extension Background Service Worker * * Connects to local Python agent via WebSocket (agent runs WS server). - * Mints reCAPTCHA and runs Flow's batchexecute RPCs inside the Flow tab. + * Runs Flow's batchexecute RPCs inside the Flow tab. CAPTCHA-bearing calls + * require explicit accountSafetyOptIn on each request. * * Flow moved to flow.google.com in September 2026 and stopped minting the * `Bearer ya29.…` the old REST host needed. The current path is `batch_rpc`: @@ -367,77 +368,32 @@ function captchaFromTab(tabId, requestId, captchaAction) { } async function solveCaptcha(requestId, captchaAction) { - let tabs = await chrome.tabs.query({ url: flowUrls }); - - // No Flow tab at all — spawn one and let it settle. Keep the exact tab id: - // a redirected or stale tab must not make us select some older candidate. - if (!tabs.length) { - let opened; - try { - opened = await chrome.tabs.create({ url: FLOW_TAB_URL, active: false }); - await sleep(3000); - } catch (e) { - return { error: e.message || 'NO_FLOW_TAB' }; - } - const target = opened?.id ? await chrome.tabs.get(opened.id).catch(() => null) : null; - if (!target) return { error: 'NO_FLOW_TAB' }; - tabs = [target]; - } - - // Try each Flow tab in turn. A tab that answers "no grecaptcha" is a tab - // sitting on a page that never loaded it — another Flow tab may well be - // fine. Returning on the first one let one stale tab veto every generation. - const errors = []; - for (const candidate of tabs) { - const tab = await reviveTabIfNeeded(candidate); - if (!tab) continue; - try { - const resp = await captchaFromTab(tab.id, requestId, captchaAction); - if (!resp?.token) { - errors.push(resp?.error || 'NO_TOKEN'); - continue; - } - return resp; - } catch (e) { - const msg = e?.message || ''; - errors.push(msg); - // Tab evaporated mid-call (window closed, discarded again, navigated - // away). Move on to the next candidate rather than failing the job. - if ( - msg.includes('No current window') || - msg.includes('No tab with id') || - msg.includes('Receiving end does not exist') - ) { - continue; - } - return { error: msg }; - } - } - - // Every candidate failed — last-ditch, spawn a fresh temporary tab and - // target THAT exact tab. Previously we re-queried all Flow tabs and picked - // fresh[0], which could select the same stale tab again while leaking the - // newly-created one on every retry. - let recoveryTab = null; + // One tab, one mint attempt. A failed tab must never trigger more token + // requests in other tabs or an automatically opened recovery tab. + const tabs = await chrome.tabs.query({ url: flowUrls }); + const candidate = tabs.find((tab) => !tab.discarded) || tabs[0]; + if (!candidate) return { error: 'NO_FLOW_TAB' }; + const tab = await reviveTabIfNeeded(candidate); + if (!tab) return { error: 'FLOW_TAB_DISCARDED' }; try { - recoveryTab = await chrome.tabs.create({ url: FLOW_TAB_URL, active: false }); - await sleep(3000); - const target = await chrome.tabs.get(recoveryTab.id); - if (!target || target.discarded) return { error: 'NO_FLOW_TAB' }; - return await captchaFromTab(target.id, requestId, captchaAction); - } catch (e) { - return { error: e?.message || errors[0] || 'NO_FLOW_TAB' }; - } finally { - // A recovery tab is disposable: there were already Flow tabs available - // for the signed RPC. Do not let CAPTCHA retries accumulate root tabs. - if (recoveryTab?.id) { - try { await chrome.tabs.remove(recoveryTab.id); } catch { /* already gone */ } + const resp = await captchaFromTab(tab.id, requestId, captchaAction); + if (resp?.token && resp.mintPath !== 'public_execute') { + return { error: 'EXTENSION_HIJACK_DETECTED: unverified_mint_path', + mintPath: resp.mintPath || 'unknown' }; } + return resp?.token ? resp : { error: resp?.error || 'NO_TOKEN', + mintPath: resp?.mintPath || null }; + } catch (e) { + return { error: e?.message || 'CAPTCHA_FAILED' }; } } async function handleSolveCaptcha(msg) { const { id, params } = msg; + if (params?.accountSafetyOptIn !== true) { + sendToAgent({ id, status: 403, result: { error: 'ACCOUNT_SAFETY_OPT_IN_REQUIRED' } }); + return; + } const result = await solveCaptcha(id, params?.captchaAction || 'VIDEO_GENERATION'); // Standalone captcha solve counts as captcha-consuming @@ -462,12 +418,24 @@ async function handleSolveCaptcha(msg) { // batchexecute POST in the page's MAIN world, where at / f.sid / bl live. const CAPTCHA_SLOT = '__CAPTCHA__'; +const CAPTCHA_RPC_IDS = new Set(['ogiZ0b', 'eb1hJf', 'YhhmEf', 'nprQif', 'MZZa6b', 'maseQ', 'SPrCad']); const MAX_RPC_TEXT = 32000000; // the project listing alone is past 17 MB async function runBatchRpc(cmd) { + if (cmd.freq?.includes(CAPTCHA_SLOT) && !cmd.captchaAction) { + return { error: 'CAPTCHA_ACTION_REQUIRED' }; + } + if ((cmd.captchaAction || cmd.freq?.includes(CAPTCHA_SLOT) || CAPTCHA_RPC_IDS.has(cmd.rpcid)) && + cmd.accountSafetyOptIn !== true) { + return { error: 'ACCOUNT_SAFETY_OPT_IN_REQUIRED' }; + } + if (CAPTCHA_RPC_IDS.has(cmd.rpcid) && !cmd.captchaAction) { + return { error: 'CAPTCHA_ACTION_REQUIRED' }; + } const tabs = await chrome.tabs.query({ url: flowUrls }); let candidate = tabs.find((t) => !t.discarded) || tabs[0]; if (!candidate) { + if (cmd.captchaAction) return { error: 'NO_FLOW_TAB' }; // No Flow tab — open one and give the app a moment to boot, otherwise // WIZ_global_data is not on the page yet and `at` comes back empty. Keep // the exact created tab id so redirects/stale tabs cannot hijack recovery. @@ -486,9 +454,13 @@ async function runBatchRpc(cmd) { if (!tab) return { error: 'FLOW_TAB_DISCARDED' }; let freq = cmd.freq; + let mintPath = null; if (cmd.captchaAction) { const solved = await solveCaptcha(cmd.id, cmd.captchaAction); - if (!solved?.token) return { error: `CAPTCHA_FAILED: ${solved?.error || 'no token'}` }; + mintPath = solved?.mintPath || null; + if (!solved?.token) { + return { error: `CAPTCHA_FAILED: ${solved?.error || 'no token'}`, mintPath }; + } freq = freq.split(CAPTCHA_SLOT).join(solved.token); } @@ -537,7 +509,7 @@ async function runBatchRpc(cmd) { }, }); - return injected?.result || { error: 'NO_INJECTION_RESULT' }; + return { ...(injected?.result || { error: 'NO_INJECTION_RESULT' }), mintPath }; } async function handleBatchRpc(msg) { @@ -547,6 +519,18 @@ async function handleBatchRpc(msg) { sendToAgent({ id, status: 400, error: 'INVALID_BATCH_RPC' }); return; } + if (freq.includes(CAPTCHA_SLOT) && !captchaAction) { + sendToAgent({ id, status: 400, error: 'CAPTCHA_ACTION_REQUIRED' }); + return; + } + if ((captchaAction || CAPTCHA_RPC_IDS.has(rpcid)) && params?.accountSafetyOptIn !== true) { + sendToAgent({ id, status: 403, error: 'ACCOUNT_SAFETY_OPT_IN_REQUIRED' }); + return; + } + if (CAPTCHA_RPC_IDS.has(rpcid) && !captchaAction) { + sendToAgent({ id, status: 400, error: 'CAPTCHA_ACTION_REQUIRED' }); + return; + } setState('running'); const hasCaptcha = !!captchaAction; @@ -571,20 +555,21 @@ async function handleBatchRpc(msg) { } try { - const out = await runBatchRpc({ id, rpcid, freq, captchaAction, match }); + const out = await runBatchRpc({ id, rpcid, freq, captchaAction, match, + accountSafetyOptIn: params?.accountSafetyOptIn }); if (out.error) { if (hasCaptcha) { metrics.failedCount++; metrics.lastError = out.error; } - if (visible) updateRequestLog(id, { status: 'failed', error: out.error }); - sendToAgent({ id, status: 502, error: out.error }); + if (visible) updateRequestLog(id, { status: 'failed', error: out.error, mintPath: out.mintPath }); + sendToAgent({ id, status: 502, error: out.error, mintPath: out.mintPath }); } else { if (hasCaptcha) { metrics.successCount++; metrics.lastError = null; } if (visible) { updateRequestLog(id, { - status: 'success', httpStatus: out.status, + status: 'success', httpStatus: out.status, mintPath: out.mintPath, responseSummary: (out.text || '').slice(0, 300), }); } - sendToAgent({ id, status: out.status, data: out.text }); + sendToAgent({ id, status: out.status, data: out.text, mintPath: out.mintPath }); } } catch (e) { const err = e?.message || 'BATCH_RPC_FAILED'; @@ -672,6 +657,10 @@ async function handleApiRequest(msg) { sendToAgent({ id, error: 'INVALID_URL' }); return; } + if (captchaAction && params.accountSafetyOptIn !== true) { + sendToAgent({ id, status: 403, error: 'ACCOUNT_SAFETY_OPT_IN_REQUIRED' }); + return; + } setState('running'); const hasCaptcha = !!captchaAction; @@ -851,6 +840,10 @@ chrome.runtime.onMessage.addListener((msg, _, reply) => { } if (msg.type === 'TEST_CAPTCHA') { + if (msg.accountSafetyOptIn !== true) { + reply({ error: 'ACCOUNT_SAFETY_OPT_IN_REQUIRED' }); + return true; + } solveCaptcha(`test-${Date.now()}`, msg.pageAction || 'IMAGE_GENERATION') .then((r) => reply(r)) .catch((e) => reply({ error: e.message })); diff --git a/extension/content.js b/extension/content.js index 89fb80f31..b6b262d0f 100644 --- a/extension/content.js +++ b/extension/content.js @@ -2,10 +2,9 @@ * Content script — bridge between background.js (ISOLATED world) and * injected.js (MAIN world). * - * All MAIN world scripts (hijack_bypass.js, recaptcha_enterprise.js, - * recaptcha__en.js, injected.js) are now loaded via manifest.json - * content_scripts with "world": "MAIN" — this bypasses the page's - * Trusted Types CSP entirely. No more createElement('script') needed. + * injected.js is loaded in the MAIN world via manifest.json. reCAPTCHA uses + * the page's own public API when present; the extension does not auto-load a + * second reCAPTCHA runtime into every Flow tab. * * This content script only handles the message relay between the two worlds. */ @@ -19,7 +18,7 @@ chrome.runtime.onMessage.addListener((msg, _, reply) => { if (e.detail?.requestId === requestId) { window.removeEventListener('CAPTCHA_RESULT', handler); clearTimeout(timer); - reply({ token: e.detail.token, error: e.detail.error }); + reply({ token: e.detail.token, error: e.detail.error, mintPath: e.detail.mintPath }); } }; diff --git a/extension/injected.js b/extension/injected.js index 1293d2fb1..0cba0da79 100644 --- a/extension/injected.js +++ b/extension/injected.js @@ -1,22 +1,7 @@ /** * Injected into the page's MAIN world on flow.google.com — has access to - * window.grecaptcha and the hijack bypass state from hijack_bypass.js. - * - * reCAPTCHA minting strategy (v0.5.1): - * - * 1. PRISTINE path (preferred): hijack_bypass.js captured the real execute - * function before Flow's x2a trap overwrote it. We call it directly with - * our own action — the token is clean. - * - * 2. OBJECT.ASSIGN NEUTER (fallback): we lost the race (all three trap levels - * + ready + poll missed). x2a's wrapper uses Object.assign to inject the - * poison action. We temporarily replace Object.assign during the execute - * call: if the result contains action:"extension_hijack_detected", we - * restore the real action before it reaches the original execute function. - * - * 3. WIDGET path (legacy fallback): if neither bypass is available, fall back - * to the pre-bypass render+execute approach. This WILL be trapped, but - * provides graceful degradation with a clear error message. + * window.grecaptcha. A detectable Flow hijack blocks minting; this script + * never alters page reCAPTCHA functions or retries a token request. */ const SITE_KEY = '6LdsFiUsAAAAAIjVDZcuLhaHiDn5nnHVXVRQGeMV'; @@ -72,125 +57,15 @@ function waitReady(timeout = 5000) { }); } -// ─── Bypass: Object.assign neuter ─────────────────────────── -// Last-resort fallback when hijack_bypass.js couldn't capture the pristine -// execute. x2a's wrapper does: -// c.execute = (e, f) => d(e, Object.assign({}, f, { action: "extension_hijack_detected" })) -// We temporarily replace Object.assign so the poison action is stripped before -// it reaches the real execute function `d`. - -const _realObjectAssign = Object.assign; - -async function executeWithAssignNeuter(sitekey, action) { - const targetAction = action; - - Object.assign = function (target, ...sources) { - const result = _realObjectAssign.call(this, target, ...sources); - if (result && typeof result === 'object' && - result.action === 'extension_hijack_detected') { - result.action = targetAction; - } - return result; - }; - - try { - await waitReady(2500); - const token = await Promise.race([ - window.grecaptcha.enterprise.execute(sitekey, { action }), - new Promise((_, rej) => setTimeout(() => rej(new Error('execute_hang')), 8000)), - ]); - return token ? String(token) : null; - } finally { - Object.assign = _realObjectAssign; +function assertNoDetectedHijack() { + const execute = window.grecaptcha?.enterprise?.execute; + if (window.__fk_hijack != null || + (typeof execute === 'function' && + Function.prototype.toString.call(execute).includes('extension_hijack_detected'))) { + throw new Error('EXTENSION_HIJACK_DETECTED'); } } -// ─── Bypass: pristine execute (primary path) ──────────────── - -async function executeWithPristine(sitekey, action) { - const pristine = window.__fk_hijack?.pristine; - if (typeof pristine !== 'function') return null; - const token = await Promise.race([ - pristine(sitekey, { action }), - new Promise((_, rej) => setTimeout(() => rej(new Error('execute_hang')), 8000)), - ]); - return token ? String(token) : null; -} - -// ─── Legacy widget path (will be trapped) ─────────────────── - -let _widgetPromise = null; -function ensureWidget(sitekey) { - if (_widgetPromise) return _widgetPromise; - _widgetPromise = (async () => { - await waitReady(5000); - let host = document.getElementById('flowkit-recaptcha-host'); - if (!host) { - host = document.createElement('div'); - host.id = 'flowkit-recaptcha-host'; - host.style.cssText = 'position:fixed;left:-9999px;top:0;width:1px;height:1px;'; - document.documentElement.appendChild(host); - } - return await new Promise((resolve, reject) => { - try { - const widgetId = window.grecaptcha.enterprise.render(host, { - sitekey, - size: 'invisible', - callback: () => {}, - 'error-callback': (m) => reject(new Error('render_error: ' + m)), - }); - resolve(widgetId); - } catch (e) { - reject(new Error('render_threw: ' + (e && e.message || e))); - } - }); - })().catch((e) => { _widgetPromise = null; throw e; }); - return _widgetPromise; -} - -// ─── Unified mint with retry ──────────────────────────────── - -async function executeWithRetry(sitekey, action, attempts = 2) { - let lastErr = null; - const hijack = window.__fk_hijack; - const hasPristine = typeof hijack?.pristine === 'function'; - const knownTrapped = !!hijack?.trapped; - - for (let i = 0; i < attempts; i++) { - try { - // Path 1: pristine execute captured by hijack_bypass.js - if (hasPristine) { - const token = await executeWithPristine(sitekey, action); - if (token) return token; - lastErr = new Error('pristine_empty_token'); - // Fall through to retry - } - // Path 2: Object.assign neuter (we know the trap is active) - else if (knownTrapped) { - const token = await executeWithAssignNeuter(sitekey, action); - if (token) return token; - lastErr = new Error('assign_neuter_empty_token'); - } - // Path 3: legacy widget (no bypass available — may still work if - // the x2a flag a.na.Aa is turned off) - else { - await waitReady(2500); - const widgetId = await ensureWidget(sitekey); - const token = await Promise.race([ - window.grecaptcha.enterprise.execute(widgetId, { action }), - new Promise((_, rej) => setTimeout(() => rej(new Error('execute_hang')), 8000)), - ]); - if (token) return String(token); - lastErr = new Error('empty_token'); - } - } catch (e) { - lastErr = e; - } - await new Promise((r) => setTimeout(r, 600)); - } - throw lastErr || new Error('execute_failed'); -} - async function mintCaptcha(pageAction) { const previous = captchaMintTail.catch(() => {}); let release; @@ -198,7 +73,14 @@ async function mintCaptcha(pageAction) { await previous; try { await waitForGrecaptcha(); - return await executeWithRetry(resolveSitekey(), pageAction); + await waitReady(2500); + assertNoDetectedHijack(); + const token = await Promise.race([ + window.grecaptcha.enterprise.execute(resolveSitekey(), { action: pageAction }), + new Promise((_, reject) => setTimeout(() => reject(new Error('execute_hang')), 8000)), + ]); + if (!token) throw new Error('empty_token'); + return String(token); } finally { release(); } @@ -208,20 +90,20 @@ window.addEventListener('GET_CAPTCHA', async ({ detail }) => { const { requestId, pageAction } = detail; try { const token = await mintCaptcha(pageAction); - const hijack = window.__fk_hijack; window.dispatchEvent(new CustomEvent('CAPTCHA_RESULT', { detail: { requestId, token, - // Diagnostic: tell the agent which path was used - bypassPath: typeof hijack?.pristine === 'function' - ? `pristine(${hijack.source})` - : hijack?.trapped ? 'assign_neuter' : 'legacy_widget', + mintPath: 'public_execute', }, })); } catch (e) { window.dispatchEvent(new CustomEvent('CAPTCHA_RESULT', { - detail: { requestId, error: e.message }, + detail: { + requestId, + error: e.message, + mintPath: e.message === 'EXTENSION_HIJACK_DETECTED' ? 'hijack_detected' : 'public_execute', + }, })); } }); @@ -230,8 +112,7 @@ function waitForGrecaptcha(timeout = 22000) { return new Promise((resolve, reject) => { const start = Date.now(); const check = () => { - // If we have pristine, grecaptcha is ready regardless of public state - if (window.__fk_hijack?.pristine) return resolve(); + try { assertNoDetectedHijack(); } catch (e) { return reject(e); } if (window.grecaptcha?.enterprise?.execute) return resolve(); if (Date.now() - start > timeout) return reject(new Error('grecaptcha not available')); setTimeout(check, 200); diff --git a/extension/manifest.json b/extension/manifest.json index e56fce626..9a3878763 100644 --- a/extension/manifest.json +++ b/extension/manifest.json @@ -2,7 +2,7 @@ "manifest_version": 3, "name": "Flow Kit", "version": "0.5.2", - "description": "Local agent bridge for Google Flow — runs batchexecute RPCs inside a signed-in flow.google.com tab, mints reCAPTCHA with hijack bypass, proxies API calls", + "description": "Local Google Flow bridge. Automated CAPTCHA generation is disabled by default during account-safety investigation.", "permissions": [ "storage", "alarms", @@ -30,9 +30,6 @@ "https://labs.google/fx/*/tools/flow*" ], "js": [ - "hijack_bypass.js", - "recaptcha_enterprise.js", - "recaptcha__en.js", "injected.js" ], "run_at": "document_start", diff --git a/skills/fk-doctor.md b/skills/fk-doctor.md index 54da6a3bb..fd57dc687 100644 --- a/skills/fk-doctor.md +++ b/skills/fk-doctor.md @@ -33,6 +33,18 @@ You are the on-call doctor for the FlowKit pipeline. Never guess — always cons 3. Prescribe the exact handler listed in `agent/worker/processor.py:_handle_failure` (lines 414-481) — don't invent a new one. 4. If auto-recovery should kick in but didn't, explain why (e.g. retry_count maxed, not matched by string). +**Account-safety stop rule overrides normal retries:** at the first +`PUBLIC_ERROR_UNUSUAL_ACTIVITY`, suspected account/session hijack, or other +unexpected security signal, stop all automated Flow generation immediately. +CAPTCHA-bearing generation is disabled by default; `FLOW_ENABLE_CAPTCHA_GENERATION=1` +is an explicit opt-in intended only for test accounts. Unusual-activity and +extension-hijack signals persist `flow_generation_safety_hold.json`; the hold +survives agent restarts and `/api/flow/clear-hijack` cannot clear it. Generic +CAPTCHA errors fail the request without an automatic retry, and alone do not +latch this persistent hold. Distinguish an extension/agent-only failure from a +block in the manual Flow website; do not infer one from the other. Then follow +[`docs/ACCOUNT_SAFETY.md`](../docs/ACCOUNT_SAFETY.md). + ## Mode 1: Triage (no args) ```bash @@ -97,8 +109,10 @@ migration, not a regression. | `PUBLIC_ERROR_MODEL_ACCESS_DENIED` | Tier mismatch (TIER_ONE trying Veo 3 / Upscale) | Terminal FAILED | `GET /api/flow/credits` to check tier; `/fk-change-model` to downgrade | | `Requested entity was not found` | Uploaded `media_id` expired (~1h TTL on uploads) | `_recover_entity_not_found` re-uploads from `image_url`, re-queues PENDING | If auto-recovery fails: manually `POST /api/upload-image`, patch `media_id` | | `Internal error encountered` | Flow backend transient 500 | Exponential backoff: `2^retry * 10s`, capped 300s | None — wait, or retry manually after a minute | -| `reCAPTCHA failed` / (contains `captcha`) | Extension couldn't solve reCAPTCHA | Retry ≤10× without consuming `retry_count` (processor.py:454-464) | Ensure a Google Flow tab is open and focused; reload extension | -| `PUBLIC_ERROR_UNUSUAL_ACTIVITY` (403, message `reCAPTCHA evaluation failed`) | Google flagged the session as bot-like — usually triggered by rapid bursts of submits (e.g. many GENERATE_VIDEO in <1 minute), shared/VPN IP, or stale auth cookies | NOT auto-handled — Google blocks even fresh requests until the trust signal recovers | (1) **Stop the worker / pipeline** so submits pause. (2) Open Chrome → `chrome://settings/cookies` (or the extension's Chrome profile) → search `google.com` and `labs.google` → **remove all cookies for both**. (3) Reload `https://flow.google.com` and sign back in (re-solve any reCAPTCHA puzzles manually). (4) Slow down submission cadence (≥1s gap between submits, ≤5 concurrent). If still blocked, switch to a different network or wait 1–6 h | +| `FLOW_CAPTCHA_GENERATION_DISABLED` | CAPTCHA-bearing generation is off by default | Terminal FAILED before the request is sent | `FLOW_ENABLE_CAPTCHA_GENERATION=1` is explicit opt-in; use only with a test account | +| `reCAPTCHA failed` / (contains `captcha`) | CAPTCHA-bearing generation failed at the extension/Flow layer | Terminal FAILED; persistent safety hold | Diagnose without resubmitting. The pre-submit marker remains held after a CAPTCHA error. | +| `FLOW_ACCOUNT_SAFETY_HOLD` | An unusual-activity or extension-hijack signal previously latched the persistent hold | Terminal FAILED; hold persists across agent restarts | Do not retry. `/api/flow/clear-hijack` cannot clear this hold. Follow [`docs/ACCOUNT_SAFETY.md`](../docs/ACCOUNT_SAFETY.md). | +| `PUBLIC_ERROR_UNUSUAL_ACTIVITY` (may appear with 403 / `reCAPTCHA evaluation failed`) | An unusual-activity / risk-evaluation response; this string alone does not establish the cause or whether manual Flow access is blocked | Stop generation; current request is terminal and the persistent hold is latched | Separate extension-only errors from a manual Flow UI block. Google recovery is unverified; do not recommend cookie clearing, IP/network changes, or a fixed wait. Follow [`docs/ACCOUNT_SAFETY.md`](../docs/ACCOUNT_SAFETY.md). | ### B. HTTP status codes @@ -137,10 +151,10 @@ Detection lives in `agent/worker/_parsing.py:_is_error`. A result is treated as | `NO_AT_TOKEN` | The Flow tab loaded but `WIZ_global_data.SNlM0e` is absent — the page is signed out, on an interstitial, or still booting | Retried with backoff | Open `https://flow.google.com/`, confirm you are signed in, let the app finish loading | | `NO_FLOW_TAB` | No Flow tab to sign the request | Extension opens one and retries once | Leave one signed-in Flow tab open; nothing here works headless | | `FLOW_TAB_DISCARDED` | Chrome discarded the backgrounded tab and the reload did not revive it | Retried with backoff | Pin the Flow tab, or keep its window visible | -| `NO_INJECTION_RESULT` | `chrome.scripting.executeScript` resolved with no frame result (`background.js:540`), so the envelope never ran — the Flow tab went away or was still booting at the moment of the call. Most likely on the first captcha-bearing RPC after an agent restart | Retried with backoff, counts against `MAX_RETRIES` | Usually transient — retry first. Rule out the lookalikes before digging: a failed mint reports `CAPTCHA_FAILED`, an oversized payload is not it (`MAX_RPC_TEXT` is 32 MB), and a structured-clone failure surfaces as 500, not 502. If it repeats, pin the Flow tab and reload the extension | +| `NO_INJECTION_RESULT` | `chrome.scripting.executeScript` returned no frame result; the Flow tab may have closed or still been booting | A CAPTCHA-bearing request is held for manual review; non-generation requests follow their normal error policy | Do not retry generation or mint another token. Check the tab and preserve the response, then follow [`docs/ACCOUNT_SAFETY.md`](../docs/ACCOUNT_SAFETY.md). | | `NO_FLOW_PROJECT` | A low-level caller reached batchexecute without a project id | **Terminal — not retried** | Use `POST /api/projects` to create a real Flow project, or call a public `/api/flow/*` endpoint without `project_id` so the session-project lease resolves one automatically | | `UNSUPPORTED_ON_BATCH_API` | A capability whose payload was never captured off the new UI, all on the Veo path: **video upscale**, **Veo r2v**, **Veo start+end-frame chaining**. Every Omni mode is ported, so this never names Omni any more | **Terminal — not retried** | For chaining and r2v, `FLOW_ALLOW_DEGRADED=1` falls back to plain i2v off the start frame. Upscale has no fallback. Real fix: capture the payload — `docs/CAPTURE.md` | -| `PUBLIC_ERROR_UNUSUAL_ACTIVITY` | Google rejected the generation as unusual activity / reCAPTCHA risk evaluation | **Do not auto-retry** — repeated submits can prolong the block | Pause generation submits, keep the browser/session/network stable, and retry manually only after the trust signal has had time to recover | +| `PUBLIC_ERROR_UNUSUAL_ACTIVITY` | Flow returned an unusual-activity / risk-evaluation error; manual UI impact is a separate observation | **Stop all automated generation immediately; current request fails and a persistent hold is latched** | Determine whether this is extension-only or the manual Flow UI is also blocked. Google recovery is unverified. See [`docs/ACCOUNT_SAFETY.md`](../docs/ACCOUNT_SAFETY.md). | | `no ogiZ0b envelope in response` | The RPC answered but not with the payload we came for — usually a signed-out page returning an HTML redirect | Retried with backoff | Re-sign in on the Flow tab | | `Polling timeout after Ns: Media not found.` | The job never produced media inside the budget | Terminal after `MAX_RETRIES` | The quoted complaint is a **diagnostic, not the cause** — finished jobs report it too. Check the Flow UI: if the clip is there, raise `VIDEO_POLL_TIMEOUT` | @@ -192,7 +206,8 @@ When the user describes a symptom in plain language, map it here first. | Extension shows "No token" | Expected — there is no bearer any more. Not a fault | | `CAPTCHA_FAILED: NO_FLOW_TAB` | Open `https://flow.google.com/` — and check the extension is v0.3.0+, older builds only matched the dead labs.google URL and could not see the tab that was right there | | 403 `MODEL_ACCESS_DENIED` | Tier mismatch — `GET /api/flow/credits`, downgrade model in `models.json` via `/fk-change-model` | -| 403 `PUBLIC_ERROR_UNUSUAL_ACTIVITY` / `reCAPTCHA evaluation failed` | Google flagged the session as bot-like (rapid bursts, VPN/shared IP, stale cookies). **Pause submits**, then in Chrome: `chrome://settings/cookies` → remove cookies for `google.com` and `labs.google` → reload `flow.google.com` → sign in & solve any captcha → resubmit with ≥1s gap and ≤5 concurrent. Switch network or wait 1–6 h if still blocked | +| CAPTCHA-bearing generation disabled | Default is off; `FLOW_ENABLE_CAPTCHA_GENERATION=1` is explicit opt-in for test accounts only | +| `PUBLIC_ERROR_UNUSUAL_ACTIVITY`, extension hijack, CAPTCHA error, or active safety hold | **Stop generation.** These signals latch a persistent hold. Distinguish extension-only failure from a manual Flow UI block. Google recovery is unverified; `/api/flow/clear-hijack` cannot clear the hold. See [`docs/ACCOUNT_SAFETY.md`](../docs/ACCOUNT_SAFETY.md). | | Scene images inconsistent across scenes | Check all refs have UUID `media_id` — run `/fk-fix-uuids` | | `media_id` starts with `CAMS...` | Run `/fk-fix-uuids` to extract UUID from URL | | Upscale fails on every scene | On the batch path upscale is unported (`UNSUPPORTED_ON_BATCH_API`) — no upsampler rpc has been captured. On the legacy path it needs `PAYGATE_TIER_TWO` | @@ -226,11 +241,13 @@ Providers, models and efforts come from `agent/providers.json`; see Decision order — stop at first match: -0. **`UNSUPPORTED_ON_BATCH_API` / `NO_FLOW_PROJECT`** → FAILED immediately. These are configuration answers, not something a retry can reach. -1. **`"not found"` in message** → `_recover_entity_not_found()` re-uploads media, marks PENDING. -2. **`reconnected` / `disconnected` / `switched`** → PENDING, keep `retry_count`. -3. **`captcha` / `recaptcha`** → PENDING if retry_count < 10; else FAILED. -4. **Default** → increment `retry_count`; if < `MAX_RETRIES` (5), schedule retry at `now + min(2^retry * 10, 300)`s. Else FAILED. +1. **`[HIJACK]` or unusual activity** → FAILED and persist the account-safety hold. +2. **`FLOW_ACCOUNT_SAFETY_HOLD` / `FLOW_CAPTCHA_GENERATION_DISABLED`** → FAILED; no retry. +3. **`UNSUPPORTED_ON_BATCH_API` / `NO_FLOW_PROJECT`** → FAILED; no retry. +4. **`reconnected` / `disconnected` / `switched`** → PENDING, keep `retry_count`. +5. **Any other `captcha` / `recaptcha` error** → FAILED; no automatic retry. +6. **`"not found"`** → attempt `_recover_entity_not_found()` and re-queue only if recovery succeeds. +7. **Default** → increment `retry_count`; if < `MAX_RETRIES` (5), schedule retry at `now + min(2^retry * 10, 300)`s. Else FAILED. ## Output format diff --git a/tests/unit/extension_account_safety.test.cjs b/tests/unit/extension_account_safety.test.cjs new file mode 100644 index 000000000..0b193cdb5 --- /dev/null +++ b/tests/unit/extension_account_safety.test.cjs @@ -0,0 +1,169 @@ +const assert = require('node:assert/strict'); +const fs = require('node:fs'); +const path = require('node:path'); +const test = require('node:test'); +const vm = require('node:vm'); + +const extension = path.join(__dirname, '..', '..', 'extension'); +const read = (name) => fs.readFileSync(path.join(extension, name), 'utf8'); + +test('manifest only auto-injects the media monitor and isolated relay', () => { + const manifest = JSON.parse(read('manifest.json')); + assert.deepEqual(manifest.content_scripts.map((script) => script.js), [ + ['injected.js'], ['content.js'], + ]); +}); + +function makeBackground(captchaResponse = { token: 'test-token', mintPath: 'public_execute' }) { + const calls = { tabs: 0, captcha: 0, scripts: 0, callbacks: [], rpc: 0 }; + const chrome = { + action: { setBadgeBackgroundColor() {}, setBadgeText() {} }, + alarms: { clear() {}, create() {}, onAlarm: { addListener() {} } }, + runtime: { + getManifest: () => ({ version: 'test', host_permissions: ['https://flow.google.com/*'] }), + onInstalled: { addListener() {} }, onStartup: { addListener() {} }, + onMessage: { addListener() {} }, sendMessage: async () => {}, + }, + scripting: { + async executeScript() { + calls.scripts++; + return [{ result: { status: 200, text: 'local-rpc-result' } }]; + }, + }, + storage: { local: { get: async () => ({}), set: async () => {} } }, + tabs: { + async query() { calls.tabs++; return [{ id: 7, discarded: false }]; }, + async sendMessage() { calls.captcha++; return captchaResponse; }, + async create() { throw new Error('unexpected tab creation'); }, + }, + webRequest: { onBeforeSendHeaders: { addListener() {} } }, + }; + class FakeWebSocket { static CONNECTING = 0; static OPEN = 1; constructor() { this.readyState = 0; } } + const context = vm.createContext({ + chrome, WebSocket: FakeWebSocket, URL, + navigator: { userAgent: 'FlowkitAccountSafetyTest/1.0' }, + fetch: async (_url, options) => { + calls.callbacks.push(JSON.parse(options.body)); + return { ok: true }; + }, + console: { log() {}, error() {} }, + setTimeout() { return 1; }, setInterval() { return 1; }, + }); + vm.runInContext(read('background.js'), context, { filename: 'background.js' }); + return { context, calls }; +} + +test('CAPTCHA batch and standalone paths require a literal true opt-in before tab access', async () => { + const { context, calls } = makeBackground(); + await vm.runInContext("handleBatchRpc({id:'a',params:{rpcid:'x',freq:'__CAPTCHA__',captchaAction:'IMAGE_GENERATION'}})", context); + await vm.runInContext("handleBatchRpc({id:'b',params:{rpcid:'x',freq:'__CAPTCHA__',captchaAction:'IMAGE_GENERATION',accountSafetyOptIn:1}})", context); + await vm.runInContext("handleSolveCaptcha({id:'c',params:{captchaAction:'IMAGE_GENERATION'}})", context); + await vm.runInContext("handleApiRequest({id:'d',params:{url:'https://aisandbox-pa.googleapis.com/test',captchaAction:'IMAGE_GENERATION'}})", context); + await vm.runInContext("handleBatchRpc({id:'e',params:{rpcid:'ogiZ0b',freq:'already-filled'}})", context); + assert.equal(calls.tabs, 0); + assert.equal(calls.captcha, 0); + assert.equal(calls.scripts, 0); + assert.equal(calls.callbacks.length, 5); + assert.ok(calls.callbacks.every((r) => r.status === 403)); + assert.ok(calls.callbacks.every((r) => (r.error || r.result?.error) === 'ACCOUNT_SAFETY_OPT_IN_REQUIRED')); +}); + +test('non-CAPTCHA RPC works while a placeholder without an action fails closed', async () => { + const { context, calls } = makeBackground(); + await vm.runInContext("handleBatchRpc({id:'a',params:{rpcid:'x',freq:'plain'}})", context); + assert.equal(calls.tabs, 1); + assert.equal(calls.scripts, 1); + assert.equal(calls.captcha, 0); + assert.equal(calls.callbacks[0].status, 200); + await vm.runInContext("handleBatchRpc({id:'b',params:{rpcid:'x',freq:'__CAPTCHA__'}})", context); + assert.equal(calls.tabs, 1); + assert.equal(calls.callbacks[1].error, 'CAPTCHA_ACTION_REQUIRED'); + await vm.runInContext("handleBatchRpc({id:'c',params:{rpcid:'maseQ',freq:'already-filled',accountSafetyOptIn:true}})", context); + assert.equal(calls.tabs, 1); + assert.equal(calls.callbacks[2].error, 'CAPTCHA_ACTION_REQUIRED'); +}); + +test('opted-in batch mints once and returns a non-sensitive mint path', async () => { + const { context, calls } = makeBackground(); + await vm.runInContext("handleBatchRpc({id:'a',params:{rpcid:'x',freq:'__CAPTCHA__',captchaAction:'IMAGE_GENERATION',accountSafetyOptIn:true}})", context); + assert.equal(calls.captcha, 1); + assert.equal(calls.scripts, 1); + assert.equal(calls.callbacks[0].mintPath, 'public_execute'); + assert.equal(calls.callbacks[0].data, 'local-rpc-result'); +}); + +test('detected hijack is terminal across Flow tabs', async () => { + const { context, calls } = makeBackground({ error: 'EXTENSION_HIJACK_DETECTED', mintPath: 'hijack_detected' }); + await vm.runInContext("handleBatchRpc({id:'a',params:{rpcid:'x',freq:'__CAPTCHA__',captchaAction:'IMAGE_GENERATION',accountSafetyOptIn:true}})", context); + assert.equal(calls.captcha, 1); + assert.equal(calls.scripts, 0); + assert.equal(calls.callbacks[0].mintPath, 'hijack_detected'); + assert.match(calls.callbacks[0].error, /EXTENSION_HIJACK_DETECTED/); +}); + +test('a token from an old Flow tab without a verified mint path is rejected', async () => { + const { context, calls } = makeBackground({ token: 'legacy-bypass-token' }); + await vm.runInContext("handleBatchRpc({id:'a',params:{rpcid:'ogiZ0b',freq:'__CAPTCHA__',captchaAction:'IMAGE_GENERATION',accountSafetyOptIn:true}})", context); + assert.equal(calls.captcha, 1); + assert.equal(calls.scripts, 0); + assert.equal(calls.callbacks[0].mintPath, 'unknown'); + assert.match(calls.callbacks[0].error, /EXTENSION_HIJACK_DETECTED/); +}); + +test('a failed mint does not open a recovery tab or mint twice', async () => { + const { context, calls } = makeBackground({ error: 'NO_TOKEN' }); + await vm.runInContext("handleBatchRpc({id:'a',params:{rpcid:'ogiZ0b',freq:'__CAPTCHA__',captchaAction:'IMAGE_GENERATION',accountSafetyOptIn:true}})", context); + assert.equal(calls.captcha, 1); + assert.equal(calls.scripts, 0); + assert.match(calls.callbacks[0].error, /NO_TOKEN/); +}); + +function makeInjected(execute) { + const listeners = new Map(); + const results = []; + const window = { + fetch: async () => ({ ok: true }), + grecaptcha: { enterprise: { ready: (callback) => callback(), execute } }, + addEventListener(name, handler) { listeners.set(name, handler); }, + dispatchEvent(event) { results.push(event); }, + }; + const context = vm.createContext({ + window, CustomEvent: class { constructor(type, options) { this.type = type; this.detail = options.detail; } }, + setTimeout() { return 1; }, + }); + vm.runInContext(read('injected.js'), context, { filename: 'injected.js' }); + return { window, listeners, results }; +} + +test('injected mint uses public execute once without changing Object.assign', async () => { + let count = 0; + const assign = Object.assign; + const { listeners, results } = makeInjected(async () => { count++; return 'test-token'; }); + await listeners.get('GET_CAPTCHA')({ detail: { requestId: 'a', pageAction: 'IMAGE_GENERATION' } }); + assert.equal(count, 1); + assert.equal(Object.assign, assign); + assert.equal(results[0].detail.token, 'test-token'); + assert.equal(results[0].detail.mintPath, 'public_execute'); +}); + +test('injected mint rejects a detectable Flow hijack before execute', async () => { + let count = 0; + const { listeners, results } = makeInjected(async function () { + const marker = 'extension_hijack_detected'; + count++; + return marker; + }); + await listeners.get('GET_CAPTCHA')({ detail: { requestId: 'a', pageAction: 'IMAGE_GENERATION' } }); + assert.equal(count, 0); + assert.equal(results[0].detail.error, 'EXTENSION_HIJACK_DETECTED'); + assert.equal(results[0].detail.mintPath, 'hijack_detected'); +}); + +test('a Flow tab retaining the old bypass object cannot mint', async () => { + let count = 0; + const { window, listeners, results } = makeInjected(async () => { count++; return 'test-token'; }); + window.__fk_hijack = { trapped: false, pristine: () => 'legacy-token' }; + await listeners.get('GET_CAPTCHA')({ detail: { requestId: 'a', pageAction: 'IMAGE_GENERATION' } }); + assert.equal(count, 0); + assert.equal(results[0].detail.error, 'EXTENSION_HIJACK_DETECTED'); +}); diff --git a/tests/unit/test_flow_account_safety_api.py b/tests/unit/test_flow_account_safety_api.py new file mode 100644 index 000000000..0bf255280 --- /dev/null +++ b/tests/unit/test_flow_account_safety_api.py @@ -0,0 +1,20 @@ +"""A legacy cooldown reset must not reopen CAPTCHA generation after a safety signal.""" + +from types import SimpleNamespace + +import pytest +from fastapi import HTTPException + +from agent.api import flow as flow_api + + +@pytest.mark.asyncio +async def test_clear_hijack_cannot_clear_account_safety_hold(monkeypatch): + client = SimpleNamespace(generation_guard_status={"safety_hold_active": True}) + monkeypatch.setattr(flow_api, "get_flow_client", lambda: client) + + with pytest.raises(HTTPException) as exc: + await flow_api.clear_hijack_cooldown() + + assert exc.value.status_code == 409 + assert "FLOW_ACCOUNT_SAFETY_HOLD" in exc.value.detail diff --git a/tests/unit/test_flow_global_throttle.py b/tests/unit/test_flow_global_throttle.py index 9fed7740a..be83f6054 100644 --- a/tests/unit/test_flow_global_throttle.py +++ b/tests/unit/test_flow_global_throttle.py @@ -1,4 +1,5 @@ import asyncio +import json import pytest @@ -6,9 +7,15 @@ from agent.services import flow_client as fc +@pytest.fixture +def opted_in(monkeypatch, tmp_path): + monkeypatch.setattr(fc, "FLOW_ENABLE_CAPTCHA_GENERATION", True) + monkeypatch.setattr(fc, "FLOW_GENERATION_SAFETY_FILE", tmp_path / "hold.json") + + @pytest.mark.asyncio -async def test_generation_rpc_is_globally_serialized(monkeypatch): - monkeypatch.setattr(fc, "FLOW_GENERATION_MAX_CONCURRENT", 1) +async def test_generation_rpc_is_globally_serialized(monkeypatch, opted_in): + monkeypatch.setattr(fc, "FLOW_GENERATION_MAX_CONCURRENT", 5) # legacy override cannot bypass safety cap monkeypatch.setattr(fc, "FLOW_GENERATION_MIN_INTERVAL_S", 0.0) client = fc.FlowClient() active = 0 @@ -32,11 +39,12 @@ async def fake_send(method, params, timeout=300): @pytest.mark.asyncio -async def test_unusual_activity_opens_local_circuit_breaker(monkeypatch): +async def test_unusual_activity_persists_a_safety_hold(monkeypatch, opted_in): monkeypatch.setattr(fc, "FLOW_GENERATION_MAX_CONCURRENT", 1) monkeypatch.setattr(fc, "FLOW_GENERATION_MIN_INTERVAL_S", 0.0) monkeypatch.setattr(fc, "FLOW_UNUSUAL_ACTIVITY_COOLDOWN_S", 120.0) client = fc.FlowClient() + already_running_peer = fc.FlowClient() calls = 0 async def fake_send(method, params, timeout=300): @@ -49,14 +57,153 @@ async def fake_send(method, params, timeout=300): monkeypatch.setattr(client, "_send", fake_send) first = await client.batch_rpc("a", "x", captcha_action=fb.CAPTCHA_VIDEO) + client._generation_unusual_until = 0.0 # timer expiry must not re-open the path second = await client.batch_rpc("b", "y", captcha_action=fb.CAPTCHA_VIDEO) + peer_result = await already_running_peer.batch_rpc("peer", "z", captcha_action=fb.CAPTCHA_VIDEO) + restarted = fc.FlowClient() + third = await restarted.batch_rpc("c", "z", captcha_action=fb.CAPTCHA_VIDEO) assert first["status"] == 200 - assert second["status"] == 429 - assert "local cooldown active" in second["error"] + assert second["status"] == 423 + assert peer_result["status"] == 423 + assert third["status"] == 423 + assert "FLOW_ACCOUNT_SAFETY_HOLD" in second["error"] assert calls == 1 - assert client.generation_guard_status["cooldown_active"] is True + assert client.generation_guard_status["safety_hold_active"] is True assert client.generation_guard_status["last_unusual_activity_rpc"] == "a" + assert json.loads(fc.FLOW_GENERATION_SAFETY_FILE.read_text())["reason"] == "PUBLIC_ERROR_UNUSUAL_ACTIVITY" + + +@pytest.mark.asyncio +async def test_default_off_sends_no_captcha_request(monkeypatch, tmp_path): + monkeypatch.setattr(fc, "FLOW_ENABLE_CAPTCHA_GENERATION", False) + monkeypatch.setattr(fc, "FLOW_GENERATION_SAFETY_FILE", tmp_path / "hold.json") + client = fc.FlowClient() + + async def should_not_send(*_args, **_kwargs): + raise AssertionError("No CAPTCHA request should reach the extension") + + monkeypatch.setattr(client, "_send", should_not_send) + result = await client.batch_rpc("image", "x", captcha_action=fb.CAPTCHA_IMAGE) + assert result["status"] == 423 + assert "FLOW_CAPTCHA_GENERATION_DISABLED" in result["error"] + + +@pytest.mark.asyncio +async def test_hijack_signal_persists_a_safety_hold(monkeypatch, opted_in): + client = fc.FlowClient() + + async def fake_send(*_args, **_kwargs): + return {"status": 403, "error": "[HIJACK] extension_hijack_detected"} + + monkeypatch.setattr(client, "_send", fake_send) + await client.batch_rpc("video", "x", captcha_action=fb.CAPTCHA_VIDEO) + assert client.generation_guard_status["safety_hold_reason"] == "extension_hijack_detected" + assert fc.FlowClient().generation_guard_status["safety_hold_active"] is True + + +@pytest.mark.asyncio +async def test_unreadable_safety_state_fails_closed(monkeypatch, opted_in): + fc.FLOW_GENERATION_SAFETY_FILE.write_text("{not json", encoding="utf-8") + client = fc.FlowClient() + + async def should_not_send(*_args, **_kwargs): + raise AssertionError("Corrupt safety state must not permit generation") + + monkeypatch.setattr(client, "_send", should_not_send) + result = await client.batch_rpc("image", "x", captcha_action=fb.CAPTCHA_IMAGE) + assert result["status"] == 423 + assert client.generation_guard_status["safety_hold_reason"] == "unreadable_safety_state" + + +@pytest.mark.asyncio +async def test_generation_rpc_without_action_never_reaches_extension(monkeypatch, opted_in): + client = fc.FlowClient() + + async def should_not_send(*_args, **_kwargs): + raise AssertionError("Missing CAPTCHA action must fail before extension") + + monkeypatch.setattr(client, "_send", should_not_send) + result = await client.batch_rpc(fb.RPC_UPLOAD_IMAGE, "already-filled") + assert result == {"status": 400, "error": "CAPTCHA_ACTION_REQUIRED: generation RPC has no CAPTCHA action"} + + +@pytest.mark.asyncio +async def test_pre_submit_marker_write_failure_sends_nothing(monkeypatch, opted_in): + client = fc.FlowClient() + original_open = fc.Path.open + + def fail_safety_write(path, *args, **kwargs): + if path == fc.FLOW_GENERATION_SAFETY_FILE and args and args[0] == "x": + raise OSError("simulated unwritable safety state") + return original_open(path, *args, **kwargs) + + async def should_not_send(*_args, **_kwargs): + raise AssertionError("No token request may be sent without a durable marker") + + monkeypatch.setattr(fc.Path, "open", fail_safety_write) + monkeypatch.setattr(client, "_send", should_not_send) + result = await client.batch_rpc("image", "x", captcha_action=fb.CAPTCHA_IMAGE) + assert result["status"] == 423 + assert client.generation_guard_status["safety_hold_reason"] == "safety_state_unwritable" + + +@pytest.mark.asyncio +async def test_failed_hold_write_leaves_inflight_marker_for_restart(monkeypatch, opted_in): + client = fc.FlowClient() + original_replace = fc.Path.replace + + def fail_hold_replace(path, target): + if path.name.endswith(".tmp"): + raise OSError("simulated failed hold write") + return original_replace(path, target) + + async def fake_send(*_args, **_kwargs): + return {"status": 403, "error": "PUBLIC_ERROR_UNUSUAL_ACTIVITY"} + + monkeypatch.setattr(fc.Path, "replace", fail_hold_replace) + monkeypatch.setattr(client, "_send", fake_send) + await client.batch_rpc("image", "x", captcha_action=fb.CAPTCHA_IMAGE) + record = json.loads(fc.FLOW_GENERATION_SAFETY_FILE.read_text()) + assert record["state"] == "inflight" + assert fc.FlowClient().generation_guard_status["safety_hold_reason"] == "interrupted_generation" + + +@pytest.mark.asyncio +async def test_hold_latched_during_pacing_wait_prevents_dispatch(monkeypatch, opted_in): + monkeypatch.setattr(fc, "FLOW_GENERATION_MIN_INTERVAL_S", 3.0) + client = fc.FlowClient() + client._generation_last_submit_at = asyncio.get_running_loop().time() + + async def latch_during_wait(_seconds): + fc.FlowClient()._latch_generation_safety("extension_hijack_detected", "other") + + async def should_not_send(*_args, **_kwargs): + raise AssertionError("Hold latched during pacing must stop dispatch") + + monkeypatch.setattr(fc.asyncio, "sleep", latch_during_wait) + monkeypatch.setattr(client, "_send", should_not_send) + result = await client.batch_rpc("image", "x", captcha_action=fb.CAPTCHA_IMAGE) + assert result["status"] == 423 + + +@pytest.mark.asyncio +async def test_generic_captcha_error_latches_before_another_request(monkeypatch, opted_in): + client = fc.FlowClient() + calls = 0 + + async def fake_send(*_args, **_kwargs): + nonlocal calls + calls += 1 + return {"status": 403, "error": "CAPTCHA_FAILED: no token"} + + monkeypatch.setattr(client, "_send", fake_send) + first = await client.batch_rpc("image", "x", captcha_action=fb.CAPTCHA_IMAGE) + second = await client.batch_rpc("image", "y", captcha_action=fb.CAPTCHA_IMAGE) + assert first["status"] == 403 + assert "FLOW_ACCOUNT_SAFETY_HOLD" in first["error"] + assert second["status"] == 423 + assert calls == 1 @pytest.mark.asyncio diff --git a/tests/unit/test_processor.py b/tests/unit/test_processor.py index 46f23c55b..3f52638ac 100644 --- a/tests/unit/test_processor.py +++ b/tests/unit/test_processor.py @@ -187,6 +187,27 @@ async def test_unusual_activity_is_not_retried_as_captcha(self): assert "retry_count" not in call_kwargs[1] mock_crud.update_scene.assert_awaited_once_with("scene-001", vertical_video_status="FAILED") + @pytest.mark.asyncio + @pytest.mark.parametrize("error", [ + "[HIJACK] extension_hijack_detected", + "CAPTCHA_FAILED: mint timed out", + "CAPTCHA_FAILED: Flow tab not found", + "FLOW_ACCOUNT_SAFETY_HOLD: manual review required", + "FLOW_CAPTCHA_GENERATION_DISABLED: temporary account-safety default", + ]) + async def test_account_safety_signals_never_requeue(self, error): + req = make_req(req_type="GENERATE_IMAGE", retry_count=0) + with patch("agent.worker.processor._recover_entity_not_found", new_callable=AsyncMock) as recover, \ + patch("agent.worker.processor.crud") as mock_crud: + mock_crud.update_request = AsyncMock() + mock_crud.update_scene = AsyncMock() + await _handle_failure(req["id"], req, {"error": error}) + + recover.assert_not_awaited() + assert mock_crud.update_request.call_args.kwargs["status"] == "FAILED" + assert "retry_count" not in mock_crud.update_request.call_args.kwargs + mock_crud.update_scene.assert_awaited_once_with("scene-001", vertical_image_status="FAILED") + @pytest.mark.asyncio async def test_extracts_error_message_from_nested_data(self): """Error message extraction from data.error.message should work."""