web-search: alert on the fallback rate, and pay the Brave plan #27

Activity

1. sre-helmcode

   assigned

   barckcode

   on Aug 19, 2026
2. sre-helmcode commented on Aug 19, 2026

   

   sre-helmcode

   on Aug 19, 2026

   ContributorAuthor

   More actions

   Measured post-deploy: the fallback-returned-zero signal is not theoretical

   Verifying the fix on 2026-08-19 I ran ~20 queries against the live SearXNG in a couple of minutes and took all three carriers down at once:

   duckduckgo → CAPTCHA                  (suspended_time=0, retried per request)
   yep        → HTTP 403 access denied   (suspended_time=180)
   google cse → unusual traffic detected (suspended_time=180)
   

   Result: n=0 for every search over roughly three minutes. It recovers on its own — after waiting out the 180s window, yep came back with 20 relevant results — so this is transient and self-inflicted by the test burst, not a standing outage.

   But it measures the real posture after disabling bing:

   • The scraped path now rests on duckduckgo + yep. mojeek returns 0 from our egress IPs and google cse is rate-limited most of the time, so those two are the whole capacity.
   • Both are scrapers, so both can be blocked at the same time by the same burst.
   • With Brave answering 402 there is no primary to absorb it, which is item 1 of this issue.

   The failure mode is the good one — empty rather than wrong, and the chat tool now tells the model to rephrase rather than asserting nothing exists on the topic — but a customer sees a 200 with an empty results array, or a 503.

   Current traffic is nowhere near this (community tier is 20 rpm / 500 daily, and the platform logged 4 searches in a day), so nothing is being hit today. It does mean the fallback-returned-zero alert has a concrete trigger to be built against, and that restoring a primary — Brave or otherwise — is what actually removes the single point of failure.
3. sre-helmcode commented on Sep 12, 2026

   

   sre-helmcode

   on Sep 12, 2026

   ContributorAuthor

   More actions

   Closing: the whole web-search surface is being retired (owner decision, 2026-09-12). Both
   items this issue asked for are moot.

   1. Pay the Brave plan — no. It is still answering 401/402 today: searchtool_brave_denied
      fired in the last 24h in BOTH tenants (enterprise-api and platform-api), which means
      both have been running 100% on scraped engines since 2026-08-19. The plan is not being
      renewed for something being decommissioned.

   2. Alert on the fallback rate — no. Instrumenting the demotion path of a service that is
      going away buys nothing. The design work is recorded below in case any of it is reusable.

   What the counter would have been, if anyone needs it later

   route() in internal/searchtool/service.go has eleven distinguishable outcomes, and the
   two the incident demanded must not share a label:

   primary:   brave_served · brave_denied · brave_failed · brave_empty
              brave_all_filtered · upstream_ceiling
   fallback:  fallback_served · fallback_empty · fallback_failed
              fallback_rate_limited · no_upstream
   

   The fallback RATE is then derivable, which is the point: every one of those paths returns
   200 to the caller, so an error-rate alert cannot see any of it. That was the incident's
   lesson and it is why this could not have been a Grafana rule over existing signals.

   Note for whoever does the decommission: enterprise-api exposes /metrics via promauto
   and cloud-api does NOT expose the endpoint at all, so community could not have been
   instrumented without first serving it. That asymmetry outlives this issue.

   What I could NOT measure, and nobody should quote as if I had

   I tried to establish current usage and failed, in a way worth writing down:

   • searxng's pod log records zero served HTTP requests (grep '"(GET|POST) /' → 0 lines).
     Its 7,508 lines in 24h are engine errors, and SearXNG probes its engines on a timer, so
     those are consistent with nobody searching AND with somebody searching. It is not an
     instrument for this question.
   • Traefik's access log retains ~110 seconds, so it cannot answer a 24h question either.

   So "usage is zero" is unproven. What IS established: /v1/search is a published route on
   api.nan.builders (ingress platform-api-images), so retiring it is member-facing
   regardless of volume.

   Item 3 does not die with this

   Two of the smaller findings are about other systems and should not be lost:

   • deploy.yml detects the VERSION bump with git diff HEAD~1. On a rebase merge of a
     multi-commit branch whose bump is not in the last commit, the deploy silently does not
     fire — a failure shaped exactly like a success. Affects enterprise-api and
     nan-cloud-api, and has nothing to do with search.
   • cacheKey has no product prefix, so community and enterprise share one Valkey
     keyspace separated only by a tenant UUID. Moot if the keyspace goes with SearXNG, but the
     same pattern may exist elsewhere.

   Filed separately as #57 so they survive this closure.
4. sre-helmcode

   closed this as completedon Sep 12, 2026