Temporary (2026-09-22): GitHub Actions billing is maxed out. The
mroconnellaccount has used 100% of its included Actions minutes for this billing cycle, and adding a payment method or raising the budget is currently blocked on GitHub's side. CI (ruff,pytest,alembic check) has kept completing successfully so far, just sometimes with a long delay before a run even registers — it has not been disabled. The repo's required-check ruleset ("main — require tests") is staying on deliberately; every merge still needs it to pass, and nobody should bypass or remove it just because a check is slow. Until this is resolved, be extra careful verifying a change locally (all five CI gates — seeCLAUDE.md) before pushing, since GitHub's own check is a slower and less certain backstop right now than usual. Remove this note once Actions billing is fixed.
Paste the URL of a public government meeting recording. Get back the video and its transcript, side by side, with every line clickable — and a URL you can share that lands someone at that exact moment.
No background jobs. Given a meeting URL, the app resolves its video and transcript on demand and renders them. Deep-linking to an exact moment is the primary goal; the transcript is a nice-to-have on top of that.
There's an optional database (see Caching and reporting
below) that caches resolved meetings and logs every resolve attempt for
admin reporting — but it's not required to run the app. With no DATABASE_URL
set, the app falls back to a local SQLite file, and if the database is ever
unreachable, /api/resolve degrades silently to its original zero-persistence
behavior rather than failing.
Permanent, publicly shareable meeting pages (with multiple transcript
versions/languages) live in a separate app, archive/ — see
Permanent pages (the Archive) below. This
resolver itself still never hosts public content pages directly; that's
the whole reason the Archive is a separate app rather than a feature bolted
onto this one.
Red Tape Recordings builds power tools for the advocates, grassroots
organizers, and everyday citizens doing the real digging into local
government — people who need to skip straight to the moment, transcript
line, or agenda item that matters, and search across hundreds of
jurisdictions and meetings at once for a topic they're tracking, rather
than sitting through hours of video. Journalists are a good example user
of this (real distribution/credibility value) but a smaller group than the
grassroots/advocacy base this is actually built for — not the product's
core definition. rtr-deeplink is the tool that saves that time.
Lean, fast, and useful is the standing bar, not a phase we graduated out of. Accounts, permanent pages, and site-wide search were all once "nice to have," and they're real now — but the core function (resolve a meeting, share a deep link) still works with no account and no signup, in the lightest format that does the job.
Who it's for, and how this is meant to eventually pay for itself.
Advocates, grassroots organizers, and everyday citizens are the intended
core users, outnumbering local journalists by a wide margin even though a
journalist is a great example case. Two real usage modes are expected once
there's a regular user base: someone lightly following a meeting or two a
month for one city council or planning commission, and someone tracking
meaningfully more than that. The free tier is meant to comfortably cover
the light case; a paid individual tier removes the ceiling for a heavy
user, and a separate institutional/B2B tier is the intended answer for an
organization tracking a specific topic across many jurisdictions at once
(e.g. a company's PR team following a specific kind of siting decision
across city councils) — a different usage shape from an individual power
user. None of this is priced or built yet — see BACKLOG.md's "Accounts +
token billing" section for the fuller, still-directional thinking.
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
uvicorn app.main:app --reload --port 8010Then open http://localhost:8010, paste a meeting URL, and go.
No further setup needed — with no .env, the app uses a local SQLite file
for caching/reporting. Copy .env.example to .env and fill in DATABASE_URL
only if you want to point at a real Postgres instance instead.
pip install -r requirements-dev.txt
pytestBoth this suite and the JS suite below also run in CI on every push/PR
(.github/workflows/test.yml, added 2026-08-14), pinned to Python 3.12.3
to match render.yaml.
tests/ covers the platform-independent utilities (app/utils/vtt_parser.py,
app/platforms/media_scan.py, app/platforms/base.py's detect_platform)
directly, and exercises every adapter in the "Supported platforms" table
below end-to-end against real fixture files saved under tests/fixtures/
(fetched live from real government sites, not synthetic
— see each fixture directory for where it came from; tests/fixtures/ civicplus/README.md explains the one exception, hand-built to match a
real site's confirmed structure since that live site has since changed).
HTTP calls are mocked via a small in-repo tests/aiohttp_mock.py, not
aioresponses — its latest release doesn't support the aiohttp version
this project's unpinned aiohttp>=3.9 resolves to today. Also covers
archive/utils/search.py's exact/fuzzy matching logic directly (pure
functions, no DB or mocking needed — tests/test_archive_search.py) and
worker/segment_utils.py + app/platforms/media_probe.py's duration-
plausibility check the same way (tests/test_worker_segment_utils.py,
tests/test_media_probe.py) — that first file also carries the
hallucination detector's coverage, backed by real Archive transcript
excerpts under tests/fixtures/hallucination_runs/ (see its own
README.md): six real hallucination loops that must be flagged, and real
decoder stutters and roll calls that must not be.
tests/test_highlights.py covers the featured-snippet heuristic
(archive/utils/highlights.py), the curated topic list, and the
government-type classifier the same pure-function way. Its snippet cases
are frozen strings the heuristic actually produced against the live
archive on 2026-08-23 — both the good ones and the two garbled ones
that reached a rendered page during in-browser verification and are why
the coherence guards exist. Tuning the scoring without re-running those
is how a fix for one meeting silently ruins twenty.
tests/test_transcription_failure_analysis.py is the same pure-function
shape and is deliberately kept that way: it exercises
crud.summarize_failure_rows() (the chunk-failure diagnostic behind
/internal/transcription-failure-analysis) against constructed rows
rather than the shared test database, which several other modules write
real failures into — any whole-table assertion there would be
order-dependent. tests/test_transcription_jobs.py is a
different shape — real integration tests against an isolated SQLite file
(set up once per test session by tests/conftest.py, not mocked), since
the transcription job lifecycle is genuinely DB-state-machine logic
(claim/report/finalize/promote) that a pure-function test can't exercise
honestly. Every adapter now has real fixture-backed test coverage.
tests/test_youtube_channel.py is a fourth shape again — it runs the
Legistar→YouTube-channel matcher (WO-30) against real, live-captured
channel listings saved as JSON under tests/fixtures/youtube_channel/,
because that matcher's whole risk is picking the wrong real meeting,
which only real titles can exercise; two of its cases have
independently-known right answers (BACKLOG.md's recorded Phoenix pairing,
and a Baltimore meeting where the city itself attached the video).
tests/test_retry_async.py covers app/utils/retry.py, the one shared
backoff policy scripts/transcribe_backlog_locally.py and
worker/main.py both use around their live-source calls — a
counting fake rather than a real site, because "fails once, then succeeds
unchanged" is exactly the behavior under test and no real government
source can be asked to do that on cue.
The suite is deliberately network-free, which leaves one thing it can't
catch: a government site quietly changing structure under a working
adapter. That's what scripts/adapter_canary.py is for — it re-resolves
one real, known-good meeting URL per platform against the live site
(in-process, so no production cache/Archive noise) and exits non-zero if
any comes back empty or broken. .github/workflows/adapter-canary.yml
runs it daily at 15:00 UTC. Adding a new platform adapter therefore means
adding a live-verified URL to that script's CANARY_URLS — or an entry to
its CANARY_EXCLUSIONS saying why no such URL exists;
tests/test_adapter_canary.py fails the build if a registered platform
has neither.
python scripts/adapter_canary.py # real network calls, ~1 minAdding a platform adapter has a second registry obligation of the
same shape (WO-35, 2026-08-21): every platform register_all_finders()
registers must also appear in archive/db/crud.py's DIRECT_PLATFORMS,
CUSTOM_PLATFORMS, or COVERAGE_EXCLUSIONS — otherwise its rows fall
through get_platform_coverage()'s if/elif chain with no matching
branch and vanish from /coverage silently, with no error at all.
tests/test_coverage_platform_registry.py fails the build if one has no
decision, and (like CANARY_EXCLUSIONS) requires every exclusion to
state its reason. Four adapters shipped without this between 2026-08-19
and 2026-08-21 — see BACKLOG_DONE.md.
shared_static/deep_link.js (the t/line/version deep-link contract
both app/static/player.js and archive/static/meeting_page.js depend
on) has its own separate JS suite, since it's the one piece of this repo
with no Python equivalent:
npm install
npm testtests_js/helpers.js loads the real file into a fresh jsdom window as
an actual <script> element per test (not vm.runInContext or eval) so
top-level let/function declarations land in real global script scope
— the same behavior two separate classic <script> tags share in an
actual browser, which matters here since player.js/meeting_page.js
both assign into deep_link.js's module-level segments variable this
same way in production.
When you paste a meeting URL, the app has to figure out which government video platform it's looking at, fetch the video and transcript from that platform's own site, and hand back something the page can render — all in the few seconds you're waiting. Here's that path in detail:
- The frontend (
app/static/player.js) POSTs the pasted URL to/api/resolve. app/platforms/base.py'sdetect_platform(url)looks at the URL's domain (and sometimes path) and classifies it as one of the platforms below.- The matching
AssetFinder(one class per platform, all inapp/platforms/) fetches whatever it needs — usually the meeting page's HTML, sometimes a small REST API — and returns aResolvedMeeting: title, date, jurisdiction, a playable video URL, transcript segments ({start, end, text}) if any were found, and agenda/chapter-marker items (agenda_items, same{start, end, text}shape) if any were found — kept as a separate field so agenda data is never mistaken for a real transcript. /meeting?url=<source>(served byapp/templates/meeting.html+player.js) calls/api/resolveclient-side and renders the result: video player (hls.js for.m3u8, native<video>otherwise), then an Agenda section (if any agenda items were found), then a clickable Transcript section. Agenda is populated independently of transcript availability, so a meeting with both shows both at once.
The rendered page itself is never cached — every page load re-resolves from
the source URL. What can be cached is the resolve result: if a database is
configured, /api/resolve checks it first (keyed on a normalized version of
the URL) and serves a prior successful resolve instead of re-fetching from
the government source. Reloading a deep link re-runs the resolve either way
(live or cached) and lands you back at the same moment.
Not every pasted URL is a clean win — sometimes it points at a list of
meetings instead of one specific meeting, and sometimes it's a platform
this app doesn't know how to read yet. /api/resolve always returns one
of these three shapes, so the frontend knows exactly how to react instead
of guessing from an error string:
- A resolved meeting — the normal case: a
ResolvedMeetingJSON blob (seeapp/platforms/models.py). {"error": "calendar_page", "candidates": [...]}— the URL was a calendar/listing page (e.g. a LegistarCalendar.aspxor a CivicPlus AgendaCenter category) rather than one specific meeting. Instead of failing, the adapter pulls every meeting it can find on that page (title, date, direct URL) and the frontend shows a pick-list (renderCalendarPage()inplayer.js).{"error": "unsupported_platform" | "resolve_failed", ...}— the platform isn't recognized, or resolution threw. Shown as a plain message.
Linking someone straight to the exact moment in a meeting — not just the video in general — is the entire reason this app exists. That link is just a couple of URL query parameters, read back out on page load:
A URL like /meeting?url=<source>&t=630&line=seg-42 means: seek the video
to 630 seconds and highlight transcript segment 42. t always wins for the
actual seek position; line is only used to decide which row to highlight
(see the comment above applyDeepLink() in player.js for why — line
used to take priority and silently truncate precision on coarse-grained
sources like chapter markers).
Every interaction that produces a link (clicking a timestamp, the per-line
link icon, "Copy link to current time", the manual "Go to time" box) goes
through the same updateUrlParams() helper, so all four stay consistent.
Skip this if you're already comfortable with SQLAlchemy/Alembic — it's here because it's easy to lose track of which database is which, and what actually happens when a new table or column shows up, once there are two separate apps each with their own database plus a migration tool layered on top of one of them.
There are two separate databases, one per app, and they don't talk to each other directly:
app/db/— the resolver's own database. Two tables:meeting_resolutions(a log of every URL anyone's ever tried to resolve, success or failure — powers the cache and the/admin/statsreporting) andproblem_reports(the "something's wrong with this meeting" form submissions). See "Caching and reporting" below.archive/db/— the Archive's own database, holding the actual permanent-page content. Four tables:meeting_pages(one row per permanent/m/{slug}page, including which government it belongs to -- see "Government identity" below),transcript_versions(a page can have several — original scrape, a self-transcribed replacement, etc.),transcription_jobs(the on-demand-transcription queue, see below), andmeeting_page_url_aliases(every URL that's ever successfully matched to a page, so a repeat paste of a Legistar link finds the same page even though its real identity lives on Granicus). See "Permanent pages" below.worker/(the background transcription service) doesn't have its own database at all — it reaches directly into the Archive's database using the same models asarchive/db/, since it's really just Archive logic that needs a long-running process instead of a web request.
How a table used to get created: create_all() — no longer true in
production. Every app startup calls Base.metadata.create_all()
(app/db/engine.py's and archive/db/engine.py's init_models()), which
looks at every model class defined in models.py and runs CREATE TABLE IF NOT EXISTS for each one. That made adding a brand-new table
zero-friction: write the class, deploy, and the table just appeared in
production on the next restart. ProblemReport and TranscriptionJob
both shipped that way. Both services' init_models() now return early
on Postgres (archive 2026-08-17, resolver 2026-08-21) — create_all()
runs only on the local/test SQLite path, and Alembic is the sole writer
to the production schema. The convenience was exactly what let
alembic_version drift silently behind reality; see the incident below
and CLAUDE.md's migration bullet.
Where create_all() stopped working even before that: changing a
table that already has real rows in it. It only ever adds tables it
doesn't see yet — it has
no idea how to add a column to a table that already exists, so if you add
a field to an existing model and just redeploy, nothing happens to
production at all; the app will start throwing "column does not exist"
errors the moment it tries to use that field. This actually happened
twice in this project's life (a materialized search column, then a
priority column on transcription_jobs) before Alembic was adopted for
exactly this reason.
What Alembic actually does, without the jargon:
- A migration is a small Python file that says how to change the
schema one step at a time — "add this column," "rename that table" —
each with matching instructions to undo itself. They're generated
automatically (
alembic revision --autogenerate) by diffing the real models against whatever the database currently looks like, though autogenerate is a first draft, not gospel — always read the file before committing it. - Migrations chain together in a straight line, oldest to newest, like a
numbered list. The most recent one is called
head. - The database keeps exactly one row, in a table Alembic manages itself
(
alembic_version), recording which migration it's currently at. That one row is Alembic's entire memory of what's already been applied — it never inspects the actual table structure to guess. alembic upgrade headis the "do the work" command: it looks at that one row, then actually runs every migration between there and the latest one — realALTER TABLEstatements against the real database.alembic stamp <revision>is the "just update the bookkeeping" command: it overwrites that one row directly and runs no SQL at all. This only exists for bootstrapping — telling Alembic "trust me, this database already matches this point in the history" for a database that had tables before Alembic was ever introduced (which was every table in this project, since they all started life viacreate_all()).
The real incident this caused (2026-08-09), as a concrete example:
production had four tables, all created by create_all(), with real rows
in them, and had never been stamped at all. The plan was alembic stamp head as a one-time step to say "you're already caught up, nothing to
run." That was true for about twenty minutes — right up until a second
migration (the priority column) was added to the history afterward.
"head" isn't a fixed point, it's whichever migration is newest right
now — so running stamp head after that second migration existed
incorrectly told production "you already have the priority column" when
it didn't, and the deployed app immediately started failing every query
with column transcription_jobs.priority does not exist. The fix was to
stamp the specific baseline revision id instead of the word head
(alembic stamp a8dc5aad7eff), then actually run the real migration on
top of that (alembic upgrade head) — see archive/alembic/README.md
for the exact recovery sequence, and BACKLOG_DONE.md for how this was
confirmed fixed.
One more practical note: each service that has real rows to protect keeps
its own Alembic config, not a single shared one — archive/
(archive/alembic.ini, archive/alembic/) for archive/db, and app/
(app/alembic.ini, app/alembic/, added 2026-08-10 after app/db hit
this exact same wall for the first time — see BACKLOG_DONE.md) for the
resolver's own app/db. Every alembic command has to be run from
inside the matching directory (cd archive or cd app) — running it
from the repo root fails with No 'script_location' key found in configuration, and running it from the wrong service's directory would
point at the wrong models/database entirely.
Backups and recovery (WO-4, AUDIT_EXECUTION_BRIEF.md, confirmed live
2026-08-17). All persistent state (both app/db and archive/db) lives
in one Render Postgres instance, rtr-deeplink-db (Basic-256mb, Oregon).
Render takes continuous backups automatically on paid instances — nothing
to configure — but two things about how that actually works matter more
than "backups exist":
- The recovery window is 3 days, not a generic "we have backups." This workspace is on Render's Hobby tier, and PITR (point-in-time recovery) retention follows the workspace tier, not the database's own plan — confirmed live via Render's dashboard, not assumed from their docs. Upgrading the database's own plan later would not extend this; only upgrading the whole workspace to a paid Team/Pro tier would (7 days). Concretely: a bad migration that lands on a Friday and isn't noticed until the following Wednesday is already past the window by the time anyone looks.
- Restoring is a swap, not a rewind. Render's PITR spins up a brand
new database instance at the chosen point in time — it does not roll
the existing
rtr-deeplink-dbback in place. Recovering for real means:- In Render's dashboard, open
rtr-deeplink-db→ Recovery → pick a timestamp within the last 3 days → this creates a new Postgres instance (a new name, a new internal hostname). - Copy the new instance's connection string.
- Update
DATABASE_URLon all three services that use it (rtr-deeplink,rtr-deeplink-archive,rtr-transcription-worker) to point at the new instance, and redeploy each. - If
EXPECTED_DB_HOSTis set onrtr-deeplink-archive/rtr-transcription-worker(archive/db/engine.py's_assert_expected_db_host(), WO-4), update it too, or the two services will refuse to start against the recovered database — this is the check working as intended (a hostname that doesn't match is exactly the signal a manual recovery step got missed), not a bug to work around. - Once confirmed working, update the old
rtr-deeplink-dbRender Blueprint entry (render.yaml'sdatabases:block) to the new instance's name so a future Blueprint sync doesn't try to reconcile against the now-abandoned original.
- In Render's dashboard, open
Not yet done: an actual test restore. This procedure is written from
Render's documented PITR behavior, cross-checked against this workspace's
real dashboard (tier, retention window, instance details) — but nobody
has actually clicked through a real recovery yet. An unexercised restore
procedure is a hypothesis, not a backup plan; doing one throwaway PITR
restore to a scratch instance (never repointing any real service at it)
to confirm these steps are accurate is a real, still-open follow-up, not
a formality — see BACKLOG.md.
Re-scraping a government site every single time someone revisits the same
meeting is wasteful and slow, and nobody can tell what's actually working
across hundreds of different city websites without some kind of record of
every attempt. app/db/ solves both, optionally: /api/resolve can skip
the live fetch when it's already resolved a URL successfully before, and
every attempt — success or failure — gets logged for later review.
app/db/ adds an optional Postgres-backed (SQLite locally) layer with two
jobs: a read-through cache in front of the live resolve, and a log of every
resolve attempt for admin reporting. Neither is required — see the note in
the intro above about graceful degradation with no DATABASE_URL set or an
unreachable database.
-
Cache:
/api/resolvechecks for a prior successful resolve of the same normalized URL (app/utils/url_normalize.py) before doing a live fetch. Failed/calendar/unsupported attempts are never cached, so a currently-broken URL always re-fetches live — that's deliberate, both so a fixed bug or a transient outage doesn't get stuck serving a stale failure, and so every real failure still gets logged for reporting. -
Reporting log: every resolve attempt — success or failure — is logged unconditionally to the
meeting_resolutionstable (app/db/models.py).app/db/outcomes.py'sclassify_outcome()buckets each logged row by actual content quality, not just whetherresolve()raised:success— a real transcript in the target language, not garbledagenda_fallback— no real transcript, but agenda/chapter-marker data was found instead (Granicus, CivicClerk, or Swagit) — still deep-linkable, but not a real transcriptblank_transcript— video found, nothing usable for a transcript at allcaptions_disabled— a YouTube video confirmed (via a real yt-dlp metadata check) to have no captions at all, or to be removed/private (WO-135, 2026-09-09) — distinct fromblank_transcript: this is a confirmed permanent answer, not "the government source hasn't posted captions yet"garbled_transcript— a real transcript, but flagged as likely garbled at the source (seeis_likely_garbled()invtt_parser.py)non_english_transcript— a real transcript, just not in the target languageno_video— resolved without a playable videoresolve_failed/calendar_page/unsupported_platform— the existing/api/resolveerror shapes
Agenda/chapter-marker data lives in its own
ResolvedMeeting.agenda_itemsfield, separate fromsegments(see "Supported platforms" below), and is fetched regardless of whether a real transcript was also found. Soclassify_outcome()checksresolved_payload["agenda_items"]directly to decideagenda_fallbackvs.blank_transcript— only reached whentranscript_foundis false, since a meeting with both a real transcript and agenda data still classifies assuccess.
Admin endpoints, all gated by ADMIN_STATS_TOKEN (?token=..., returns 404
rather than 401/403 on a missing/wrong token so the route isn't
distinguishable from a typo):
GET /admin/stats— aggregates: totals, success rate, cache hits, average resolve duration, counts by platform × outcome, recent non-success rows.GET /admin/log?limit=&format=json|csv— the unaggregated per-attempt list (URL, platform, outcome, language, timestamp), most recent first.GET /admin/problem-reports?limit=— user-submitted "something's wrong with this meeting" reports, most recent first.GET /admin/recheck-archive-page?url=— force an immediate re-resolve + Archive push for one meeting, instead of waiting for the passive 30-dayARCHIVE_RECHECK_AFTERrecheck. For when a permanent page needs a fix (e.g. an adapter bug fix, or a source that's since added captions) to land sooner. Returns what it found (segment/agenda counts, warnings, whether anything was pushed) synchronously, unlike the passive recheck.GET /admin/sweep-pending-pushes— on-demand version of the durable Archive-push retry mechanism (below), for checking on or forcing it directly. Returns exactly which resolutions it found and retried.GET /admin/schema-info— read-only introspection of this service's real live schema: the actual reflected columns on every table (via SQLAlchemy'sInspectoragainst a real connection) next to whatapp/db/models.pyexpects, withmismatched_tables/schema_matches_modelscalling out any difference, plus whateveralembic_versioncurrently says as context rather than as truth. The resolver's port of the Archive's/internal/schema-info(described under "Permanent pages" below), added 2026-08-21 so confirming production's real schema — specifically whethermeeting_resolutionshasjurisdiction_confidence, and what the resolver's untouchedalembic_versionactually holds — doesn't need Render shell access. Seeapp/alembic/README.md's runbook for how to read it. Noteactual_columnsreflects the whole database, which the Archive shares, so its tables appear here too; they aren't counted as mismatches sinceapp/db/models.pysays nothing about them.
Durable Archive pushes: a successful resolve's push to the Archive is
fired via BackgroundTasks, so a resolver process restart (a deploy, a
crash) between the response returning and the task actually running could
previously lose the push silently — no exception, no log line, since the
process that would have logged it is the one that got killed (real
incident, see BACKLOG_DONE.md's 2026-08-10 entry). MeetingResolution
now tracks archive_pushed_at/archive_push_attempts, and an
opportunistic sweep (fired from /api/resolve itself, at most once every
few minutes — this app has no background job queue by design, so this
isn't a real scheduler, just the same pattern ARCHIVE_RECHECK_AFTER's
stale-page recheck already uses) retries any row with real content that's
stayed unpushed past a grace period. /admin/stats' pending_archive_pushes
count and the /admin/sweep-pending-pushes endpoint above give direct
visibility into this instead of relying on someone happening to notice a
meeting missing from /meetings.
See .env.example for the two env vars (DATABASE_URL, ADMIN_STATS_TOKEN).
The resolver above is deliberately disposable — it re-fetches from
scratch every time and remembers nothing public. But some meetings
deserve a permanent, linkable, search-engine-indexed page of their own,
the way a news article does — so a second, separate app (archive/)
takes a successfully resolved meeting and gives it a real, permanent home
at redtaperecordings.com/m/{slug}.
This resolver stays deliberately stateless per meeting — the meeting_resolutions
table above is a private cache/log, not a public archive. Permanent,
SEO-indexed meeting pages live in a separate app, archive/ (its own
FastAPI service, own database, own deploy) — not grown into this resolver,
so this app keeps its single job: resolving, not hosting public content
pages. See archive/ for that app's own structure; this section covers
only the handoff between the two.
Domain: permanent pages are reachable at redtaperecordings.com/m/{slug}
— same domain as everything else, for SEO and sharing consistency. This
resolver's service holds the custom domain, so it reverse-proxies /m/*
and /archive-static/* through to the Archive's own Render service
(app/archive_client.py's proxy_get(), wired up in app/main.py). A
failure to reach the Archive here returns a clean 503, not a hang or a raw
exception — these are public, potentially-indexed pages.
Lookup, before resolving: /api/resolve calls archive_client.lookup()
before checking its own local cache or doing a live resolve. If a
permanent page already exists for the normalized input URL, the response is
{"redirect_url": "/m/{slug}"} instead of a ResolvedMeeting — player.js
sends the browser there (preserving ?t=/?line= from the current URL),
consolidating traffic and sharing on the canonical permanent URL rather than
re-scraping. This lookup only has the raw pasted URL to go on, which is why
the Archive keeps a MeetingPageUrlAlias table recording every URL that's
ever successfully pushed — without it, a Legistar/CivicPlus/PrimeGov URL
(whose real identity lives on the platform it delegates to, not the URL the
user pasted) would never match on a repeat paste.
Push, after resolving: after a live resolve succeeds with real content
(segments or agenda_items non-empty — blank/failed resolves are never
pushed, so test pastes and broken URLs don't create junk permanent pages),
/api/resolve fires archive_client.push() via FastAPI's BackgroundTasks
(not a bare asyncio.create_task, which risks the task being
garbage-collected mid-flight). The Archive matches the push against an
existing MeetingPage by (platform, external_id) when available, else by
the resolved URL, and either creates a new page or attaches a new
TranscriptVersion to an existing one (deduped by a content hash, so
re-pushing an unchanged meeting doesn't pile up duplicate versions). Both
lookup() and push() are wrapped in the same safe() pattern as the DB
calls above — a down/misconfigured Archive degrades silently, never breaks
/api/resolve.
Agenda and packet text, read once at ingest (WO-1966, 2026-10-16):
when an ingest carries a CivicPlus (/AgendaCenter/ViewFile/Agenda/...)
or CivicClerk (*.api.civicclerk.com/v1/Meetings/GetMeetingFileStream(...),
the only link CivicClerk's API gives since 2026-10; an older stored
GetMeetingFile(...) link is still accepted)
agenda_link or packet_link, /internal/ingest queues
archive/utils/agenda_ingest.read_and_store() as a BackgroundTask. It
reads each document's text with app/agenda_text.py (ported from
rtr-upcoming) and stores it in agenda_versions: one row per distinct
text per page and role (agenda or packet), deduplicated by a sha256
content hash. An unchanged re-read stores nothing; an amended agenda is a
new row. A page is re-read on a later ingest, but not within 24 hours of
the last read. There is no schedule and no sweep. CivicClerk is read
through its own plain-text route first, then the file itself. CivicPlus
is read as the PDF first, then its HTML copy. PDFs are parsed in a
separate process (app/agenda_extract_worker.py) under a memory ceiling
(AGENDA_EXTRACT_MEMORY_MB, default 1,000) and a time limit
(AGENDA_EXTRACT_TIMEOUT_S, default 300), one at a time per instance; a
body over 100 MB is refused, never truncated. Every failure is logged and
never touches the ingest. Every other platform's agenda_link is ignored
for now. Nothing displays or searches this text yet, and it must
never be loaded into search_corpus or any query that spans every page
(packet text runs to 1.8 million characters). See BACKLOG.md's "Agenda
text as a first-class, versioned asset" entry for what comes next.
YouTube transcripts — fetched locally, not server-side: YouTube
blocks caption requests from cloud-provider IPs (confirmed live
2026-08-10: yt-dlp, plain timedtext requests, and youtube-transcript-api
all fail from Render's IP while the same calls work from a residential
connection — see BACKLOG_DONE.md's experiment entry). So YouTube-backed
pages get their transcripts through a two-part loop instead:
GET /internal/transcript-wanted (token-gated like every /internal/*
route) lists every YouTube-backed page with no default transcript, and
scripts/fetch_youtube_transcripts.py — run locally, from a residential
connection, same .env setup as scripts/bulk_ingest.py plus
pip install -r requirements-dev.txt — drains that queue via
youtube-transcript-api and pushes results back through the normal
POST /internal/ingest (idempotent, content-hash-deduped, matched to
the existing page). Supports --dry-run and --limit; aborts the whole
run on an IP-level block rather than failing every queue entry
identically. Prints per-item timing (wall-clock timestamp + elapsed
seconds) and a final total/average — fetching an already-generated
caption track is one API call, so run time is independent of how long
the actual meeting is.
Permanent failures are recorded and never re-queued (WO-135,
2026-09-09). Before this, a page whose channel disables captions, or
whose video was removed or made private, was re-attempted from scratch
every single day forever — real, confirmed counts: 8 TranscriptsDisabled,
4 VideoUnplayable, 3 VideoUnavailable in the first 15 real failures of
one run (see BACKLOG_DONE.md). Before every real transcript request,
the script runs a metadata-only yt-dlp check
(YouTubeAssetFinder.check_permanent_failure(), zero rate-limit cost —
no captions are actually downloaded) and skips the request entirely if
it already confirms a permanent failure; if the check comes back clean
but the real youtube-transcript-api fetch itself still raises one of
those three exceptions, that's treated the same way. Either path records
one of three exact marker strings via POST /internal/pages/{slug}/ video-status (not the ordinary /internal/ingest path — see
archive/db/crud.py's record_youtube_video_status() docstring for why
a permanently-empty transcript has no way to record why through that
route) and the page then drops out of GET /internal/transcript-wanted
for good:
YouTube: captions are disabled by the channel(transcript_warnings) — confirmed via yt-dlp: no manual or auto-generated captions exist in any language.YouTube: embedding is disabled by the channel; watch on YouTube(video_warnings) — the video may still have real, fetchable captions; this does not skip the transcript attempt (a real, confirmed-live case has both facts true at once).YouTube: video is unavailable (removed or private)(transcript_warnings) — a real yt-dlp/youtube-transcript-apierror confirms the video is genuinely gone, distinguished from a merely scheduled-but-not-yet-live stream (which must never be marked permanent).
A marked page still counts as "no good transcript" everywhere else on
the site (a captions-disabled page never falsely shows a transcript
checkmark) — the marker only changes whether fetch_youtube_ transcripts.py and the cloud auto-transcription worker
(find_auto_transcription_candidate()) keep trying it. It stays
eligible for a local Whisper run (scripts/ transcribe_backlog_locally.py, WO-136) from this same residential Mac,
since the channel not having its own captions says nothing about
whether this app can still generate one from the audio track — only the
cloud worker is excluded, since it runs from a server IP YouTube blocks
and could never fetch that audio anyway.
Partial-transcript warning (WO-923, 2026-09-20). A caption feed can
stop partway through a meeting with nothing on the page to say so (the
Rhode Island Senate page: captions to 81 minutes of a 149-minute video).
Every registered adapter's resolve() now ends with
app/platforms/coverage_check.py: when the video's duration is known
(ResolvedMeeting.video_duration_seconds from the adapter, or a
header-only ffprobe of an HLS/mp4/mp3 file; never guessed, and never an
ffprobe of a YouTube page — since WO-935 a YouTube resolve carries the
length yt-dlp already returns) and the last caption ends under 90% of it
with 10+ minutes uncovered, it adds a reader warning containing "may end
before the meeting did". That is the Archive's existing
_EARLY_TRUNCATION_MARKER, so the page reports as truncated_transcript
and stays eligible for re-transcription with no new plumbing. RTR_PARTIAL_TRANSCRIPT_CHECK=0 switches the check off.
The same warning on a transcript we made ourselves, with a tail-silence
check (2026-10-16). The cloud worker (before it reports a job's last
chunk), scripts/transcribe_backlog_locally.py and
scripts/retranscribe_first_chunk.py all call one shared function,
coverage_check.own_transcript_early_end_warning(), on the finished
transcript and the video's real probed length. Same threshold, same
marker, only the last cue compared. Before warning, it samples 30 seconds
of audio halfway between the last cue and the video's end (volume only,
media_probe.sample_mean_volume_db()). A mean under -60 dB is silence:
dead air after the meeting, so no warning. That rule comes from the two
real pages WO-935 measured: their tails read -83 and -71 dB, while speech
read -22 to -41 dB. No length, or no reading of the tail, means no
warning. The same switch turns it off.
A transcript the local scripts push has no job row. It now waits 30 days
before the finders pick its page again, the same wait as a page whose
newest job completed (crud._jobless_own_transcript_waiting()).
scripts/count_own_transcript_early_end_flags.py counts flagged own
transcripts, read-only, from the Archive's Render shell.
Superseded on the dedicated Mac by scripts/youtube_drip.py (2026-09-11) —
one always-on process that fetches captions for waiting pages, feeds the
YouTube lines of the tier-3 queue, and downloads audio for captions-disabled
pages for local Whisper, all under one paced YouTube request budget with a
long escalating back-off on a block. The launchd job below stays as the
fallback for a Mac where the drip is not running; never run both on the
same internet connection. Plain-language operating instructions:
docs/YOUTUBE_DRIP_RUNBOOK.md.
Runs automatically once a day via launchd on the user's own Mac (must
be that machine specifically — the residential IP is the whole point).
scripts/com.redtaperecordings.fetch-youtube-transcripts.plist has the
real install/test/uninstall commands in its own header comment; installed
copy lives at ~/Library/LaunchAgents/, logs at ~/Library/Logs/ fetch-youtube-transcripts.log.
Every real (non-dry-run) run emails a report to YOUTUBE_FETCH_REPORT_EMAIL
(default ryan@how-to-adu.com) via the Archive's existing Resend
integration (archive/utils/email.py, same RESEND_API_KEY/
RESEND_FROM_ADDRESS the Archive service already uses) — every transcript
actually added, with a real clickable link, even an empty report, so
silence itself is a signal the job stopped firing. A run that fails to
complete at all (an IP-level block, or any unhandled exception) sends a
different, explicitly-flagged failure email instead.
The page itself degrades gracefully too, instead of staying dead
(WO-136, 2026-09-09), consuming the YouTube: embedding is disabled by the channel; watch on YouTube marker from the block above.
archive/templates/meeting_page.html renders a "Watch on YouTube" link
(honoring the page's own ?t= deep link) in place of the dead iframe,
server-side, whenever that marker is already known; app/static/ player.js renders the identical link both proactively (marker present
in the resolve response) and reactively, from the YouTube IFrame
Player's own onError — live-verified against a real embedding-disabled
video that by the time onError fires had already had its <div>
replaced by YT.Player's own <iframe> of the same id, so the fallback
swaps that iframe out for a fresh element rather than writing invisible
fallback content into it. A page in this state is folded into the
existing "empty page" rule (crud._is_empty_page_condition(), and its
Python twin in archive/main.py's /m/{slug} route — the same
predicate that already noindexes and hides from /meetings, the
sitemap and the feed a page with no video, no agenda and no transcript
at all, tests/test_thin_page_audit.py's own lockstep test keeps the
two copies honest) only while it has no transcript — scripts/ transcribe_backlog_locally.py's local Whisper run (below) can still
produce one by downloading the audio directly, since an embedding
restriction doesn't stop an audio download the way it stops playback or
the caption endpoint — and the page returns to /meetings, the sitemap
and the feed on its own the moment one lands, with no un-hide step.
Checking the Archive's real production schema: GET /internal/schema-info
(token-gated the same way as every other /internal/* route — a bearer
token matching ARCHIVE_INGEST_TOKEN, 404 rather than 401/403 on a
missing/wrong token) reflects the actual live columns on every table via
SQLAlchemy's Inspector against a real connection, next to what
archive/db/models.py's Base.metadata currently expects, and reports
any mismatch directly (mismatched_tables, schema_matches_models) —
plus whatever alembic_version currently says, as context only, not as
the source of truth. Exists specifically so confirming production's real
schema state doesn't require someone with DATABASE_URL access to run
psql/alembic commands by hand and paste the output back — added
2026-08-10 after a stale doc's account of "production has never been
stamped" turned out to be wrong and caused a real (contained) Alembic
mistake; see archive/alembic/README.md and BACKLOG_DONE.md for that
incident. Note this hits the Archive service's own base URL directly
(the same one ARCHIVE_BASE_URL points at) — /internal/* is
deliberately not one of the paths redtaperecordings.com proxies
through (only /m/* and /archive-static/* are, see "Domain" above),
so it isn't reachable at the public custom domain. Example:
curl -H "Authorization: Bearer $ARCHIVE_INGEST_TOKEN" "$ARCHIVE_BASE_URL/internal/schema-info"Auditing what the pipeline published unverified: GET /internal/low-trust-pages (token-gated the same way, and reachable only
at the Archive service's own base URL for the same reason as above)
lists every archived page whose provenance was never actually confirmed
— platform == "unknown" (the name generic_fallback.py registers
under), best_effort (the resolver's own flag for that path, which also
covers the fallback results that delegate to YouTube and therefore
report platform = "youtube"), or a jurisdiction_confidence of
unverified/blank. Each row carries a reasons list saying which of
the three caught it, plus slug, title, platform, jurisdiction, source
URL, creation date and reviewed_at; ?limit=/?offset= paginate and
total is the full match count. Exists because the resolve → Archive →
public page → social announcement path is fully automatic end to end,
with nothing that could otherwise answer "what has this published that
nobody looked at?" It's read-only and changes nothing: these pages stay
live, indexed, and in the sitemap by design (see BACKLOG.md) — the
best_effort flag's one enforcement effect is that social auto-posting
refuses to announce them.
What's actually in the queue, measured (first-ever production call,
2026-08-21): 474 rows — 470 unverified_jurisdiction, 7
unknown_platform (3 overlapping), and zero best_effort. So in
practice this reads as a data-quality queue — meetings whose
jurisdiction couldn't be determined, on real live pages with real video
— and as a trust/spoofing queue only prospectively. best_effort can't
be backfilled onto rows archived before its column existed, so it only
starts appearing on pages ingested from 2026-08-21 onward.
Two filters make that volume workable (added 2026-08-21, WO-38); both are optional and an unfiltered call returns exactly what it always did:
?unreviewed=true— only rows nobody has marked reviewed yet (reviewed_at IS NULL).?reason=unknown_platform|best_effort|unverified_jurisdiction— one reason at a time, which matters given the skew above: the 4 rows that aren'tunverified_jurisdictionare otherwise invisible in practice. An unrecognised value is a 400, not a silently-unfiltered result.
Marking pages reviewed: POST /internal/low-trust-pages/mark-reviewed?ids=1,2,3 (same token gate)
stamps meeting_pages.reviewed_at on exactly the ids given and nothing
else — it does not hide, de-index, edit or delete anything, and the
public page is served exactly as before. ids is required and there is
deliberately no "mark everything" mode. Idempotent: an id already in the
requested state comes back under already_reviewed with its original
timestamp untouched, and an unknown id under missing_ids rather than
failing the batch. ?dry_run=true is the default (returns the exact
diff it would write); ?unreview=true clears the stamp back to NULL,
the undo for a mis-pasted id list.
# newest unreviewed rows
curl -H "Authorization: Bearer $ARCHIVE_INGEST_TOKEN" \
"$ARCHIVE_BASE_URL/internal/low-trust-pages?limit=50&unreviewed=true"
# the handful flagged for something other than a missing jurisdiction
curl -H "Authorization: Bearer $ARCHIVE_INGEST_TOKEN" \
"$ARCHIVE_BASE_URL/internal/low-trust-pages?reason=unknown_platform"
# mark three of them reviewed (drop dry_run=false to preview first)
curl -X POST -H "Authorization: Bearer $ARCHIVE_INGEST_TOKEN" \
"$ARCHIVE_BASE_URL/internal/low-trust-pages/mark-reviewed?ids=2215,2201,2200&dry_run=false"The meeting inventory report (WO-124, 2026-09-09): GET /internal/meeting-inventory (same token gate, keyset-paginated with
after_id/limit capped at 500, format=csv for a CSV page with the
cursor in X-Next-After-Id) returns one flat review row per archived
page -- the stored jurisdiction next to the government registry's name
for its gov_id and whether they agree, the "City, ST" convention check,
the stored meeting_body (blank when nothing was stored -- nothing here
is guessed from a title), video/transcript presence and outcome bucket,
the page-vs-video platform split /coverage/detail uses plus the raw
video host domain, and the archive/source/video links. GET /internal/meeting-inventory/summary is one aggregate query returning
the headline missing-field counts (no stored body, no date, no gov id,
no video, no transcript). scripts/export_meeting_inventory.py --out-dir DIR walks both over HTTP and writes meeting_inventory.csv, a
self-contained sortable/filterable meeting_inventory.html (sticky
headers, per-column filters, click-a-tile-to-filter, CSV download), and
summary.json. Columns and derivations live in
archive/utils/meeting_inventory.py.
The unidentified-pages view (WO-256, 2026-09-12): GET /internal/unidentified-pages (same token gate) groups every archived page
that has no government yet -- a blank gov_id, or the
rtr:unknown:<host> placeholder -- by the host it came from, biggest host
first. Each row carries the page count, the blank-versus-placeholder
split, whether the host is one of the confirmed multi-government hosts
(MULTI_GOV_HOSTS, which can only ever be pinned per video or per
channel), which real governments already have pages on that host, the one
government whose hub already adopts those pages under the §5 inclusion
rule (adopted_by, null for a shared host or a host with two
governments), and a few example pages to recognise it by. ?host= narrows
to one host, ?limit= caps the host rows. This replaces the
export-and-grep step every identity work order used to repeat by hand --
see docs/investigations/hub_architecture_audit.md §6.
Diagnosing transcription chunk failures: GET /internal/transcription-failure-analysis?days=N (token-gated the same
way, reachable only at the Archive service's own base URL) groups every
recorded chunk failure by media host, page host, platform, and
position-within-job. It needs no new instrumentation —
TranscriptionJob.failure_history has stored a real per-attempt
{chunk_index, error, at} since 2026-08-19 — so this is a query over
data that already exists.
Two things about how to read it, both load-bearing:
- Read
failure_rate_by_chunk_index/failure_rate_by_decile, not the rawfailure_positioncounts. A job has exactly one chunk 0 but many later chunks, so raw counts always make later chunks look dominant. Normalized per attempt, the shape is diagnostic: rising with chunk index means an accumulating rate limit, a spike at chunk 0 means cold storage/rehydration, flat-and-high throughout means a source that's simply slow against the fixed 120s ffmpeg timeout. - Grouping is keyed on the media URL's host, not the page's. Granicus
serves ~300 distinct
{tenant}.granicus.compage hosts off a handful of sharedarchive-video/archive-stream/archive-media.granicus.commedia hosts — the throttling party would be the CDN, not the tenant. Both groupings are returned so the difference stays visible.
concurrency_pairs_within_10min directly tests whether workers are
contending for one host across different jobs. Measured across all 514
production jobs on 2026-08-21 that bucket was 0, which is what ruled
out reordering the queue by host — see BACKLOG_DONE.md's WO-40 entry.
curl -H "Authorization: Bearer $ARCHIVE_INGEST_TOKEN" "$ARCHIVE_BASE_URL/internal/transcription-failure-analysis"Redirect hits are logged too — status="archive_redirect" — so
/admin/stats' totals don't develop a blind spot as more traffic migrates
to Archive-redirects over time; classify_outcome() in app/db/outcomes.py
returns it directly (any non-"success" status passes through as-is).
Local dev runs both services side by side:
# terminal 1
uvicorn app.main:app --reload --port 8010
# terminal 2
ARCHIVE_INGEST_TOKEN=devtoken uvicorn archive.main:app --reload --port 8020With ARCHIVE_BASE_URL=http://localhost:8020 and a matching
ARCHIVE_INGEST_TOKEN set for the resolver (see .env.example), pasting a
URL twice will resolve live the first time and redirect to /m/{slug} the
second. Leaving ARCHIVE_BASE_URL unset disables the integration entirely
— /api/resolve just always live-resolves, same as before this feature
existed.
Discoverability: GET /meetings (proxied like /m/*) is a paginated,
server-rendered index of every permanent page (crud.list_pages(),
20/page) with a search box and jurisdiction/date-range/has-transcript/
has-agenda filters — all plain GET params, so results are
shareable/bookmarkable URLs with no JS required. A keyword search's
results carry the same deep-link treatment as the state/hub featured
cards (WO-48, 2026-08-24): each row whose match lands in a real
transcript segment gets a "Play from 25:17" link into that second plus
the stored frame from that moment, while the excerpt itself stays
query-matched rather than switching to the heuristic pick — a result's
job is showing why it matched. A match that can't be tied to one segment
(an agenda-only hit, or a quoted phrase split across two caption cues)
still gets its excerpt, just no timestamp. Empty pages (no video,
no agenda, no transcript) are excluded from the default browse, the
sitemap and the feed at query time — not deleted, and still shown under
an explicit has_transcript=false filter (that's how gaps get found) —
and rows/pages carry an "Upcoming"/"Recent" pill when the meeting
hasn't happened yet or happened within the last 30 days without a
transcript (archive/utils/date_status.py; see BACKLOG_DONE.md's
2026-08-17 entry). GET /sitemap.xml and
GET /robots.txt (the latter lives on the resolver, not proxied, since
robots.txt has to be at the domain root) give search engines an actual
crawl path to /m/{slug} pages, which previously had none —
robots.txt also disallows /meeting (the ephemeral resolver page) so
it doesn't compete with the permanent version of the same content once
one exists.
GET /coverage (proxied like /meetings, replacing a 2026-08-10
placeholder) is a public, per-platform table — one row per real,
distinct meeting platform (archive/db/crud.py's
get_platform_coverage() + DIRECT_PLATFORMS/CUSTOM_PLATFORMS; the
line those two dicts draw is "one product many jurisdictions buy" vs. "a
bespoke scraper this app wrote for one government", not "hosts video"
vs. "doesn't" — Hyland, Destiny AgendaQuick and open.media are all
agenda/CMS front-ends with their own row), each linking to a real
/m/{slug} permanent page as proof when one exists, with the same
rubber-stamp "Transcript" badge /meetings uses. Deliberately excludes
calendar-tool detection routers that only ever delegate (Legistar,
CivicPlus, PrimeGov, CivicWeb) — on a real successful resolve their own
ResolvedMeeting.platform is always the delegated platform's, never
their own (see the "Supported platforms" table above), so a row for one
of them could never have a real example; a one-line note on the page
covers them instead of a permanently-empty row. Every platform is listed
even with zero live examples yet ("Supported, but no example archived
yet") rather than silently omitted, per this file's "don't claim a data
path works without a positive example" convention — the thing being
shown here is "does a real page exist," not "is this code path
exercised," which are different claims.
/coverage also has a per-government-body table ("Every place we've
covered", crud.get_jurisdiction_coverage()) below the per-platform one,
sorted alphabetically for Ctrl+F discoverability ("only software
engineers think platform first").
Reworked 2026-08-26 for mobile length (the page had grown to stack
two ~1,600-row tables): the "Every place we've covered" table stays on
/coverage and keeps its flat, unpartitioned alphabetical order on
purpose — see CLAUDE.md's reasoning on why that ordering itself (a random
small town sitting next to Los Angeles) is a deliberate brand choice, not
just a sort default — but now renders inside a capped-height,
visibly-scrollable pane with a sticky header row and an "All N
jurisdictions supported so far…" callout above it, so its real length
reads as obviously-more-below rather than dictating the page's own
scroll length. A new lightweight, server-backed "Find your government"
search box (GET /api/jurisdictions?q=, crud.search_jurisdictions())
sits above it — bounded on both the SQL fetch and the response size, and
deliberately never ships the roster itself to the client, unlike the
table below it.
The fuller "Full jurisdiction detail table" (crud.get_full_jurisdiction_coverage(),
added 2026-08-17), one sortable/filterable row per successfully-archived
jurisdiction with: video-embeds / agenda-embedded /
instant-transcript-from-source / transcript-from-audio-possible (yes/no
each — the last is derived from video_format not in _IFRAME_EMBED_VIDEO_FORMATS (youtube/vimeo/viebit), mirroring
app/main.py's own _unreadable_media_message() reasoning that an
iframe-embed page is structurally unprobeable by ffprobe, not a live
check; every other stored video_format — mp4/m3u8/mp3/wav — is a
genuine fetchable media URL), a two-column "Detail page" vs. "Video"
provider split (recovers PrimeGov/CivicWeb/LIMS/SLC/ClerkBase/open.media's
real identity from source_url_normalized even though
MeetingPage.platform says "youtube" for all of them — see this file's
"when a platform turns out to be a wrapper around another" note in
CLAUDE.md for the Legistar/CivicPlus case this can't recover, since
their delegation overwrites source_url with the delegated platform's
own URL), an outcome bucket (mirrors app/db/outcomes.py's
classify_outcome()), and a last-verified date — moved 2026-08-26 to
its own page, GET /coverage/detail, noindex'd and linked only from
/coverage's "What about Platform XYZ?" FAQ answer, since it turned out
to be mostly a personal/power-user tool rather than something the public
page needed to carry. Sorting reuses archive/static/coverage.js's
existing client-side pattern (now generalized to any table.sortable-table,
shared unchanged by both /coverage and /coverage/detail); filtering
is plain client-side JS (dropdowns + a jurisdiction search box + yes/no
checkbox filters), unchanged by the move — the whole roster is ~1,600
rows in production as of 2026-08-26, small enough to render server-side
in one page load and filter/sort entirely in the browser, same reasoning
the sort code already relied on.
GET /state/{slug} (per-state landing pages, added 2026-08-17) —
server-rendered, indexable SEO pages ("California public meeting videos &
transcripts", /state/california), proxied like /coverage. Each lists
that state's covered governments (same table shape as /coverage's
"Every place we've covered", grouped from the stored jurisdiction's
canonical ", ST" suffix via
archive/utils/jurisdiction_format.py's state_abbr_from_jurisdiction())
plus featured transcript snippets (see below), with a self-referential
canonical (deliberately unlike /meetings' filter-blind one) and a link
to the pre-filtered /meetings?jurisdiction={StateName} search. Backed by
crud.get_state_page_data() (anchored LIKE '%, CA' suffix match — not
list_pages()'s substring ilike, which would leak "Decatur, GA" into
California via its "ca" substring) and crud.get_state_coverage_index()
(one row per covered state; also feeds /coverage's "Browse by state"
section and per-state sitemap.xml entries with real lastmod values).
Both queries exclude platform == "unknown" pages — the same trust
posture as the sitemap, which noindexed pages shouldn't leak through.
Jurisdictions without a recognized ", ST" suffix (school districts,
state agencies, non-US) don't appear on any state page — a documented
limitation. A state with zero indexable meetings 404s rather than
rendering an empty shell, and every /m/{slug} page whose jurisdiction
has a state now links "More {State} meetings" to its state page.
GET /state/all-50 (added 2026-08-26) — a national version of the
same pattern, deliberately scoped to the 50 US states only for its first
cut (no DC, no Canada/territories — broader "all jurisdictions"/"all
states + territories" pages are later work). Reuses the featured-snippet
feed, topic chips, "most watched governments," and the shared
_government_groups.html sidebar partial (also now used by
/state/{slug}) unchanged, but is not get_state_page_data() with
its state filter simply dropped — that function pulls every matching page
into Python unbounded, fine at one state's scale and a real corpus-wide
scan at national scale (its own sibling get_home_highlights() already
documents this). crud.get_all50_page_data() instead composes three
separately-bounded pieces: a LIMIT-based highlight pool (same shape as
get_home_highlights(), scoped to the 50 states), a recent-activity
window bounded by MOST_ACTIVE_WINDOW_DAYS (90 days, not the whole
history) for the freshness count and most-active ranking, and
crud.get_national_government_list() (get_jurisdiction_coverage()'s
already-proven-at-full-corpus-scale query shape, not
get_state_page_data()'s heavy per-row one) for the government list.
Drops the two footer links /state/{slug} has (a jurisdiction-scoped
/meetings search and RSS feed) rather than pointing them at their
unscoped equivalents — neither has an honest single value meaning "all 50
states," and an unscoped /meetings link on this page could surface
Canadian results, which would read as wrong.
Rebuilt 2026-08-23 around real quoted transcript text. See
STATE_HUB_PAGES.md for the full reference — how each piece works,
why, what was tried and rejected with the measurements, a tuning table,
how to verify a change, and the ranked future improvements. The summary:
because
Search Console measured Google selectively declining these two page
types: /j/ hubs appeared at 3.6x and /state/ at 3.1x their sitemap
share among non-indexed URLs, while /m/ meeting pages indexed better
than theirs (0.5x). That ruled out crawl budget and domain age and left
thin, templated, near-duplicate content — a list of meeting titles is
templated; a resident explaining why they came to a 9pm hearing is not.
Both surfaces now lead with:
- Featured meetings with real transcript snippets — a genuine quote
from each meeting, deep-linked to the second it was said
(
/m/{slug}?t={start}), transcribed meetings only. Picked byarchive/utils/highlights.py, which scores every candidate window of a transcript against what a reader finds substantive: procedural language (roll call, "motion", "all in favor", "you have three minutes") is heavily negative, dollar figures and "residents"/ "concerned" are positive, a curated topic hit is the strongest signal, the first 8%/last 3% of the meeting are skipped as ceremony, and windows after a "public comment" marker get a bonus. Deterministic arithmetic, not an LLM call — it runs for every meeting at ingest and has to be explainable when a bad snippet reaches a public page. Two coherence guards (_repetition_penalty()) reject a hammered content word and an interleaved roll-up caption phrase; both were added against snippets this heuristic actually produced on a live page, and the real strings are frozen intests/test_highlights.py. Featured sets are date-ordered but carry a topic diversity cap (crud.MAX_FEATURED_PER_TOPIC): at most two cards may share a topic, so one busy subject cannot take over a page. The case that prompted it — San Diego's hub showed two cannabis cards and two housing cards while a public comment delivered in character as Darth Vader about flock-camera surveillance sat unfeatured in the same pool; recency alone had no way to prefer it. Implemented as two passes rather than a sort, so the cap reorders without ever shrinking the set: a page whose meetings genuinely all share one topic still fills up. Cards with no topic are never constrained (they cannot cluster), and a?topic=view is exempt (every card is about that topic by construction). Exactly one topic is<mark>ed per snippet, the rarest on the card — the same Darth Vader quote also matchedlibraries-parkson the word "playground" and rendered with three highlighted "playground"s burying theflockthe reader came for. A rarity-ratio filter was tried first and rejected by measurement: rarity is counted over the page's own pool, and in a six-meeting hub pool both topics have a count of 1, so no ratio can separate them. The tiebreak falls through to curatedTOPICSorder, which is roughly newsworthiness-ordered and does the real work at that scale. - Topic chips — curated subjects (
archive/topics.py, edited by hand) that actually appear in the page's recent transcribed meetings, ranked by how many mention each, top 12 shown. A topic may be pinned (Topic.pinned), which keeps its chip even below the cutoff:data-centersandflock-camerasare surfaced for being newsworthy, not frequent, and both measurably lose a pure popularity contest to "property taxes" on the live California page. Pinning never overrides the count > 0 rule — a state with neither shows neither, since a chip leading to an empty page is worse than no chip. Topics deliberately overlap:flock-cameraswas split out ofsurveillance-cameras(2026-08-23) so Flock — a named vendor residents show up to speak about by name — has its own findable chip, but Flock terms remain insurveillance-camerastoo, because Flock is surveillance and a reader clicking either chip should find those meetings. The bareflockpattern was checked against the real corpus before shipping: 131 meetings mention it and all 14 stored highlights containing it are about the company, not birds. Real?topic=links, server-rendered, so a crawler follows them and each variant carries its own real snippets; the canonical stays the bare URL. Discovery is deliberately not unsupervised — an uncurated "trending terms" pass over council transcripts surfacesitem,supervisor,motion. - Meeting-card thumbnails at the quoted moment (
/m/{slug}/card.jpg ?t=), andVideoObjectstructured data for each featured meeting. Both only wherecrud.pages_with_thumbnails()confirms a stored frame — advertising a card URL that would 404 is worse than advertising none. - Grouped government list (state pages) — State government /
Counties & regions / Cities & towns / School districts / Agencies &
special districts / Courts / Other public bodies, via
archive/utils/gov_groups.py, which maps each page's storedgov_type(the Census-of-Governments type the registry assigned at ingest) to a heading. It replacedgov_classify.py, a regex over the display name that filed "Broward County Public Schools" and "West County Wastewater District" under counties and "Minnesota Senate" under cities, and disagreed with the registry on 388 of 5,929 rows (WO-99, 2026-09-02). Nothing here guesses: an unidentified government reads "Other public bodies" rather than being called a city or an agency. A sticky sidebar beside the results on desktop; below them on mobile, where the results lead. Every/j/link stays in the initial HTML in both layouts. - "Most watched governments" (state pages, ≥8 governments) — ranked by meetings archived in the last 90 days. "Watched" is meant literally and in this site's own voice: Red Tape Recordings is the one doing the watching. It claims no per-visitor view count (there isn't one), and the subtitle keeps the basis explicit regardless.
Snippets are precomputed and stored in meeting_highlights, not
computed per request: the heuristic needs a meeting's segments, and a
long meeting's segment JSON is a six-figure-byte blob (San Diego's Board
of Directors: 6,313 segments), so computing on demand would decode
megabytes per render on the exact surface built for crawlers. Kept in
sync from crud._refresh_search_corpus() — the same single choke point
that already recomputes search_corpus and upserts search_vocabulary
— so a highlight cannot go stale the way the pre-2026-08-17 corpus could.
A page with nothing quotable simply has no row, and every consumer
renders fine without one. scripts/backfill_meeting_highlights.py is
the retroactive sweep and the way to re-run after a change to the
heuristic (--force) or to archive/topics.py (which bumps
TOPICS_VERSION, making stale rows self-identify).
search_queries (added alongside) logs every /meetings keyword with
no user identity at all — keyword, optional jurisdiction filter,
result count, timestamp; no IP, user id, session, or user agent. It
exists so the curated topic list can eventually be ranked by real demand
rather than guesswork (crud.top_search_keywords()); nothing renders it
yet. Written from a FastAPI background task, so a logging failure can
never break a search.
Government identity (gov_id, WO-99, 2026-09-02) — meeting_pages
carries three columns that say whose meeting a page is, alongside the
jurisdiction string that says what to call it:
| column | meaning |
|---|---|
gov_id |
the identity of the government whose event this is |
gov_type |
its Census-of-Governments type (county, municipality, township, school_district, special_district, state, court, other) |
meeting_kind |
what kind of event the page is; NULL means an ordinary meeting |
jurisdiction did both jobs before, and a string cannot be an identity.
Two measured consequences, both live on the site until this landed:
/state/california listed both "County of Fresno" and "Fresno
County" — thirteen California counties on twenty-six hub pages, each pair
one government — and, worse, a government whose page merely mentioned
its host city was filed under that city. LADWP's meetings sat on the City
of Los Angeles's hub; SANDAG's on San Diego's; Menlo Park Fire's on
Atherton's. Nine confirmed tenants. A validated name is not a validated
identity.
The id is namespaced and deterministic — us:place:0627000,
us:county:06019, us:cousub:5502517200, us:state:06,
us:sd:0622710, ca:csd:3518013, ca:cd:3521, ca:pr:35, or a minted
rtr:us:ca:<slug> where no national table covers the government. Not a
surrogate key, because nothing in this estate shares a sequence: this app
is Postgres on Render, rtr-discovery and rtr-upcoming are local SQLite,
the feed is flat files and rtr-business is CSVs. An id every tool derives
from (type, name, state) plus the same committed data files is the only
one all five can assign independently and still agree on.
app/utils/gov_registry/ assigns it, from a seven-rung ladder whose
order is the fix: a pin, then finalize_jurisdiction()'s string repair,
then the government's TYPE classified before any place lookup — a
name that reads as a special district is looked up in the district path
and can never fall through to the place table, however place-like it
reads — then the matching national table (exactly one match or nothing),
then a fallback pin, then minting, then blank. jurisdiction_confidence
holds the resolution tier now (pinned / registry / inferred /
unverified / unresolved / blank, plus the existing
manual_override), which its "plain string, not an enum" decision
already anticipated.
jurisdiction is the display name, generated from the registry
row for every page that keyed a real government — pinned,
registry, unverified (a minted rtr: id) and inferred alike
(gov-id audit, 2026-09-10; before that only the first two, which left a
minted government's meeting page reading "County of Santa Clara, CA /
Housing Authority" while its own hub read "Housing Authority of the
County of Santa Clara, CA"). So "County of Fresno, CA" and "Fresno
County, CA" both read and group as Fresno County, CA, and a page never
disagrees with its hub. The string the adapter actually extracted is
kept verbatim in jurisdiction_raw (nullable; NULL on pages archived
before 2026-09-10) — the evidence a reviewer needs for a borrowed or
pinned identity, now that the display no longer carries it. A page the
resolver declines to key (unresolved) keeps the string it had and its
gov_id stays NULL — a real, distinguishable state, and deliberately not
a minted id, because an id nobody can look up is worse than an honest
gap. A blank page (nothing extracted at all, gov_id = rtr:unknown:<host>) shows no government and no hub link, exactly
like a page with no id: the "Unidentified government (host)" placeholder
that used to render there linked to a hub the hub list never built.
How identity flows, in one paragraph. Four words that used to be
interchangeable are kept apart: a place is a geography with a national
code (enrichment only); a government is the legal entity whose meeting
this is (the identity — one per page); a tenant is the host where the
recording lives (many-to-many with governments: a county's Granicus
site hosts a city, one city has three hosts); a meeting body is the
council or commission inside a government (its own column, never part
of the identity). The resolver app never assigns an identity — its
on-demand page shows the adapter's string. Identity enters exactly once,
at Archive ingest in _find_or_create_page(), through the seven-rung
ladder above, and everything a reader sees springs from the gov_id
that produced: the display name, the /j/ hub and its slug, the
state-page heading, the breadcrumb. Shared hosts (YouTube, Vimeo,
TelVue, ClerkBase, ChampDS, DestinyHosted, TownHall Streams) are told
apart by a match discriminator on the override row — a channel, an org
token, a customer id — and the page's own host always outranks the
video's channel: a township that posts to a shared county channel is
still the township's page. Enumeration runs the other way round: it
starts from a known gov_id and goes looking for a website, so by the
time a candidate reaches ingest the government was never in doubt —
which is why POST /internal/ingest accepts a gov_id directly
(gov-id audit, 2026-09-10): a supplied id is treated as a pin (tier
pinned, display name from the registry row; 400 if the registry cannot
render it), and if the page the push matched already carries a
different real government the push is refused with 409 rather than
overwriting — which is also the detector for two unrelated governments
sharing a generic embed id (youtube:videoseries). Absent, every existing
caller behaves exactly as before. The same pass stores the video's
publishing account on the page (video_channel — a YouTube handle or
Vimeo owner slug — and video_channel_id) and passes the handle to
the resolver as a page_hint, so a match=channel=@Handle override row
fires for a bare YouTube/Vimeo paste; YouTube's three host names
(www.youtube.com, youtube.com, youtu.be) and Vimeo's two are looked
up as one host family so a rule written against one fires for all.
The word is "government" in code and column names — gov_id,
gov_type — because that is the Census of Governments' own term for
exactly this unit (a mosquito abatement district and a city are both
"governments" there), and "public body" would collide with body, which
already means the council/commission level. "Public bodies" is fine as
display copy on the site, and /state/* uses it for the catch-all
heading; headings and column names do not have to match.
scripts/backfill_gov_id.py is the retroactive sweep (dry-run by
default, commit per row, Render shell only), and
scripts/score_gov_registry.py regenerates the registry files, the
scoring report in reports/gov_registry_scoring_<date>/ and the hub-slug
alias map. POST /internal/jurisdiction/override?ids=&gov_id= sets the
identity by hand and appends a tenant_overrides.csv-shaped rule for a
human to commit, so an override becomes a rule rather than a stamp on N
rows — BACKLOG.md's Santa Clara entry is a hub that re-fragmented
within two days of a row-level fix.
scripts/repair_wrong_pages.py (WO-934) repairs named pages in bulk from
one reviewed sheet, reports/wrong_page_worklist.csv. A row either
re-keys a page (that same override route) or deletes it
(POST /internal/admin/delete-pages). It is a dry run by default, writes at
most 5 rows per run, and refuses any row whose live page no longer matches
what the row expected, so a stale row never acts. Run it from the Archive's
Render shell; the commands are in BACKLOG_DONE.md's WO-934 entry.
scripts/wrong_page_screen.py is the read-only title screen that finds
candidates for the sheet.
reports/gov_id_problem_cases.csv + scripts/check_gov_id_problem_ cases.py (gov-id audit, 2026-09-10) — a hand-picked regression corpus
of ~80 archived pages whose identity a human verified and which are
known to be hard: the shared TelVue token that filed Half Moon Bay under
Pacifica, Juneau AK keyed to Juneau WI, a special district under its
county, subdomains whose two letters are a type ("arkansas-sc"), the
consolidated city-counties, Lloydminster's two provinces, the nine
tenants that are not the government whose name they carry (LADWP,
SANDAG, ...), and controls that must not move (South San Francisco is
not San Francisco). The checker compares each case with a fresh export
and, given a backfill_gov_id.py --report file, with what the backfill
would write — run it before every --apply, since that is the moment a
resolver change re-breaking a settled case is still cheap. Rows carry a
status: verified rows fail the run on drift; known_wrong (the
generic-embed collisions) and open (honestly unkeyed) rows are reported
but never fail it. Expected values are what a person verified, never
what the resolver happens to say today.
GET /j/{slug} (per-government hub pages, added 2026-08-17) — one
landing page per government ("Napa, CA public meeting videos &
transcripts", /j/napa-ca), proxied like /state/*. Grouped by
gov_id since WO-99 (2026-09-02), with the slug and the title
generated from the registry row rather than from whatever string an
adapter extracted. That is what finally collapsed the fragmentation the
old rule left behind: grouping by the slug of the display form got
"City of Napa, CA" and "Napa, CA" right, and got the thirteen California
counties wrong, because "County of Fresno, CA" and "Fresno County, CA"
are two legitimate display forms of one government. 179 merges over 365
hub pages in the scoring run; the state-page tables group by the same id
and link each government to its hub. Retired slugs 301 rather than 404
(archive/data/hub_slug_aliases.csv, 702 of them, generated by
scripts/score_gov_registry.py — see archive/utils/hub_aliases.py for
why that map is a committed file and not a table). Each hub lists every archived
meeting for that government newest-first with transcript badges, a
meeting-body breakdown ("City Council (30) · Planning Commission (12)"
from meeting_body), date range, the same featured snippets and topic
chips the state pages carry (6 cards rather than 12 — a hub is one
government, so after a handful the full meeting list below serves the
reader better; a hub whose own pool is too thin to produce any chips
borrows its state's, labelled "Being discussed across {State}:" and
linking to /state/{slug}?topic= rather than to the hub, which has no
meetings for those topics — that is the common case, not an edge one,
since 439 of 574 stateful jurisdictions had exactly one meeting when
last measured; those 6 cards are also capped at
MAX_FEATURED_PER_BODY per meeting_body since a hub is one government
with many bodies, deliberately unlike the state pages, where nearly
every card would be a "City Council" — see STATE_HUB_PAGES.md §6), a
BreadcrumbList (Home › State › Jurisdiction),
VideoObject structured data and breadcrumb nav, and links to /state/{slug} and the
pre-filtered /meetings?jurisdiction= search; every /m/{slug} page
links "More {Jurisdiction} meetings" to its hub. Thin-content
threshold: the archive is wide and shallow (measured 2026-08-17: 574
stateful jurisdictions, 439 with exactly one meeting), so every hub
renders but only hubs with ≥ crud.JURISDICTION_HUB_MIN_INDEXABLE (2)
meetings are indexable and listed in sitemap.xml (real lastmod);
below that the page carries noindex and a "know of another?" note.
Evaluated live per request — a singleton hub becomes indexable by
itself when its second meeting lands. Backed by
crud.get_jurisdiction_hub_data() / crud.list_indexable_hub_entries()
(one GROUP BY jurisdiction over indexable, non-empty pages; no schema
change). Same platform == "unknown" / empty-page exclusions as the
sitemap and state pages.
Search covers title, jurisdiction, agenda item text, and every
transcript version's segment text (so a demoted version's text still
counts toward a match, though the listing's badge and snippet reflect the
default version). All of that is materialized at ingest into one
lowercased meeting_pages.search_corpus column (compute_search_corpus()
in archive/utils/search.py, refreshed by crud._refresh_search_corpus()
on every ingest and when a Whisper transcription completes), and every
filter runs in SQL with LIMIT/OFFSET + a windowed count — see
list_pages()'s docstring. Query syntax: bare words are ANDed,
"quoted phrase" requires adjacency, -word / -"phrase" excludes.
Three code paths, chosen per request:
- Full-text (Postgres, once Alembic revision
c1d2e3f4a5b6has been applied) — the default in production:search_tsv @@ websearch_to_tsquery('english', q)against aGENERATEDtsvector column with a GIN index, answered from the index without reading the corpus, so a common word costs the same as a rare one. Adds stemming (budget/budgets/budgeting), stopword removal andOR. Word match, not substring.?sort=relevance("Sort by relevance" checkbox) orders byts_rank_cd; the default stays newest-first.list_pages()feature-detects the column (crud._fts_available()) and falls back to the next path when it's absent, so code and migration can deploy in either order. - Exact substring (SQLite in dev/CI, or Postgres before that
migration):
search_corpus LIKE '%term%'— byte-for-byte the predicatearchive/utils/search.py'smatches()computes, GIN-trigram indexed on Postgres (revisionbf4f54a11e5f). - Fuzzy (
fuzzy=true, "Fuzzy search (…slower)" checkbox): each query word must be within a small edit distance of a real word in the meeting's text — so "traffic" still finds a transcript that says "trafic" or "traffiq" (real transcription errors). On Postgres withsearch_vocabularypresent (revisionc684908ce5ff), this is SQL-authoritative too: each word longer than 4 chars is trigram-matched against that small, GIN-indexed table of distinct real words (crud._vocab_candidate_stmt()), every candidate re-verified with the same bounded-Levenshtein check for exact semantic parity, and the confirmed words checked againstsearch_corpusvia the substring path above — no full-archive scan.list_pages()feature-detects the table (crud._vocab_available()) the same way it doessearch_tsv, falling back to a Python-streamed scan oversearch_corpustext (SQLite dev/CI, or Postgres before that migration) otherwise. Still opt-in and slower than exact/full-text search — common short words can still take a few seconds since several individually-common real-word matches may need checking — but no longer scales with total archive size.
BACKLOG_DONE.md's "Search: move to a materialized/indexed column — full
saga, closed" entry has the complete history, from the 2026-08-17 day
this went from a Python scan over transcript JSON (which OOM-crashed the
Archive on common terms) through exact-mode, full-text, and fuzzy all
becoming SQL-backed.
Bulk read-only export (GET /internal/export/pages, WO-93, 2026-09-01).
The one endpoint for pulling archived pages out in bulk over HTTP --
metadata plus light per-version summaries (id, language, source,
segment_count computed in SQL, warnings, content hash), and the default
version's raw segments only when include_segments=true. Keyset-paginated
(after_id/limit, next_after_id in the response), with created_after,
has_transcript and ids filters, capped at 500 pages per request (100
with segments). Token-gated like every other /internal/* route. Built for
the data-product sample exports in ~/Documents/rtr-business/data-product/
so production is only ever read over HTTP, never through a database
connection (BACKLOG.md's standing decision); see
crud.list_pages_for_export()'s docstring for the shape and the explicit
column allowlist (never search_corpus, never any account/PII column).
Sometimes the government site's own captions are missing, garbled, or in the wrong language — this app can't fix that at the source, but a viewer can ask for a real transcript made from the meeting's own audio instead. Click "Transcribe this meeting from audio" on any meeting page (ephemeral or permanent), and — once a couple of quick checks pass — leave an email address to be notified when it's ready. No account required — a Clerk session isn't checked or needed; the email address (confirmed by a one-click link the first time) is the only gate. It's added to the permanent page alongside the original, not instead of it, so nothing is ever silently replaced.
Why this needs a third service. Transcribing a multi-hour meeting is
real, sustained work — nothing like the sub-second checks the resolver and
Archive web services already handle per request. Neither of those is a
place to run something that might take hours, so a third, persistent
service (worker/, a Render Background Worker — the first paid, always-on
piece of infrastructure this project has needed) exists just to grind
through transcription jobs in the background. A second, identically-
configured replica (rtr-transcription-worker-2 in render.yaml) can run
alongside it during a backlog catch-up window — real, distinct Render
services rather than numInstances scaling on one, since the two need to
differ in exactly one env var; see that file's own comment on the second
service block for why, and claim_next_chunk()'s docstring
(archive/db/crud.py) for why job/chunk claiming is already safe for any
number of concurrent worker processes.
The flow, end to end:
- Feasibility check (
POST /api/transcription/check-feasibility,app/main.py) — live-resolves the meeting fresh, then probes the discovered media URL's real duration viaffprobe(app/platforms/media_probe.py). Rejects anything implausibly short (under 5 minutes — almost certainly the wrong asset, not a real meeting) or implausibly long (over 14 hours). This is also what makes the button itself real friction against abuse: nothing past this point happens for a meeting with no usable source. A YouTube-hosted meeting stops here (WO-1969). The server never resolves or probes it (that would be a YouTube request from Render). It records the press in the Archive (POST /internal/youtube-audio-requests, one row per video, repeats counted) and tells the reader it is noted, with no date. The drip Mac's audio lane (scripts/youtube_drip.py) reads the open requests and takes one a day ahead of its backlog; seedocs/YOUTUBE_DRIP_RUNBOOK.md. - Submit (
POST /api/transcription/submit) — re-runs the entire feasibility check server-side (never trusts a client-supplied "it passed" flag) and asks the Archive to create a job (POST /internal/transcription/create-job, token-gated like every other resolver↔Archive call). The Archive checks whether this email is already a newsletter audience member: if so, the job is queued immediately; if it's a first-time address, the job waits atpending_confirmationuntil a confirmation-email link is clicked (GET /confirm-transcriptionon the resolver) — one click, once, ever, per address, since confirming also opts them into the audience. - Processing (
worker/main.py) — a persistent loop: claim the oldest pending chunk (archive/db/crud.py'sclaim_next_chunk()/report_chunk_result()— the worker reaches into the Archive's database directly here, the one deliberate exception to the resolver↔Archive HTTP-only rule, since this process is Archive backend logic, just running in a process shape the Archive's own web dyno can't offer), extract that chunk's audio withffmpeg(re- resolving a fresh media URL first — HLS/signed URLs can go stale over a long-running job; a chunk whose fast input-side seek comes back undecodable gets one retry with a slower output-side seek, which is what makes Cablecast's fMP4 VOD work at all — seemedia_probe.py's_extract_chunk_once(); since WO-935 a chunk that decodes but is more than 15 seconds shorter than asked for is treated the same way, and a slice of the cached whole-file audio gets the same decode check — the last chunk of a file is exempt, since its asked-for length is only what is left of the probed duration), transcribe it with a self-hostedfaster-whispermodel (loaded once at process startup, reused for every job), shift its timestamps from chunk-relative to full-meeting-relative seconds (worker/segment_utils.py'sshift_segments()— the same{start, end, text}convention every adapter's real segments already use, so the existing deep-link/seek logic needs no changes to work on a transcribed transcript), and persist the result before moving to the next chunk. Checkpointed after every chunk specifically so a worker restart or redeploy loses at most one in-flight chunk, never the whole job. - Completion — the finished transcript becomes a new
TranscriptVersion(source="transcribed", language detected from its own real text the same way every scraped-caption adapter already does) and is promoted to the page's default (closing a real, previously- unaddressed gap: earlier, only a page's very first transcript version ever became default — seeBACKLOG_DONE.md). If its last cue stops well short of the video and the audio after it is not silent, it carries the "may end before the meeting did" warning (see the partial-transcript paragraph above). Nothing is deleted — the original scraped version (if any) stays reachable through the existing version picker. An email goes out with an excerpt and a link to the permanent page.
Speaker labels come only from the source's own captions (WO-1967,
2026-10-16). TranscriptSegment.speaker (app/platforms/models.py) is
filled by parse_vtt() from WebVTT voice tags (<v Speaker 1>), the
shape TelVue and Orange County FL send. Room or device labels are dropped,
and generic "Speaker N" labels are dropped when the same file also names a
person. The meeting page and the .txt download show a label where it
changes; the .srt download does not. Our own AI transcriptions carry no
labels: diarization is not built (self-hosted faster-whisper is the same
base WhisperX builds diarization on, via pyannote.audio). Pages archived
before WO-1967 have no labels, and a re-ingest with the same text keeps the
old version, so they are not backfilled.
Working the existing backlog locally, on a bigger model than the cloud
worker can afford. worker/'s faster-whisper model size is forced
down to "tiny" by Render's 2GB worker plan (real OOM crashes on
"small", not a quality choice — see worker/transcription_engine.py's
own docstring), and "tiny"'s real accuracy against actual meeting audio
has two confirmed failure modes documented in BACKLOG_DONE.md's "Worker
Render plan sizing" entry (a meaning-changing mistranscription, and a
near-total transcription failure on a real stretch of English speech). A
local Mac isn't under that ceiling, so scripts/transcribe_backlog_ locally.py works the archived-but-untranscribed backlog
(/meetings?has_transcript=false, ~209 meetings as of 2026-08-16) from
here instead:
python scripts/transcribe_backlog_locally.py --dry-run
python scripts/transcribe_backlog_locally.py --limit 5
python scripts/transcribe_backlog_locally.py --model-size medium --limit 1
python scripts/transcribe_backlog_locally.py --url "https://..." # one specific meeting, bypassing the queue
# A real overnight/multi-day unattended batch (see "Thermal pacing" below):
caffeinate -s python scripts/transcribe_backlog_locally.py --cpu-threads 2 --chunk-cooldown-seconds 30- Model size is auto-picked from this Mac's real total RAM (
"small"at ≥16GB,"medium"at ≥32GB,"base"otherwise — see_pick_default_model_size()'s own docstring for the exact reasoning), not guessed — override with--model-size.device="cpu"stays correct even here:faster-whisper's CTranslate2 backend has no Apple Silicon GPU acceleration, so this is still CPU inference, just on real multi-core hardware instead of Render's box. - Candidates come from
GET /internal/transcription-backlog(archive/main.py, token-gated like every other/internal/*route) — the any-platform, batch counterpart to/internal/transcript-wanted's YouTube-only queue, reusingfind_auto_transcription_candidate()'s own quality/cooldown checks (archive/db/crud.py) so this script and the worker's own idle-time auto-generation never duplicate feasibility-probe effort on the same page.probe_duration()/is_plausible_meeting_ duration()(the same 5-minute-to-14-hour bounds the worker already uses) skip an infeasible candidate cheaply, before spending real transcription time on it. - No full download for most platforms — reuses
extract_chunk_audio()(app/platforms/media_probe.py) for direct remote extraction (an HTTP Range fetch for a direct file, just the covering.tssegments for HLS), same as the worker. Chunking is kept (900 seconds — no longer the same as the worker, which dropped to 450s in WO-94 for RAM reasons that don't apply here; this script calls the sharedchunk_size_seconds_for_platform()withmemory_constrained=False) for a different reason than the worker's real one — RAM isn't the local constraint — see the script's own module docstring: it bounds each individualffmpeg/ffprobecall undermedia_probe.py's shared 120-second subprocess timeout, proven safe at 900s in production but untested at a full multi-hour single pass. - YouTube is the one platform that IS a full download (WO-136,
2026-09-09). A
youtube.com/embed/{id}URL isn't ffmpeg-readable directly, sotranscribe_meeting()downloads the video's audio once via yt-dlp (_yt_dlp_download_best_audio(), sameplayer_clientfallback order asYouTubeAssetFinder), converts it through the sameextract_full_audio()step every other platform's whole-audio-cache path uses, and every chunk after that is a local slice — no per-chunk network. This exists specifically for a YouTube channel that disabled embedding elsewhere (dead in our iframe, see "Permanent pages" above) or disabled captions: an audio download isn't blocked by either restriction the way playback and the caption endpoint are, and this is a genuinely different request shape from the caption-fetch endpoint that's separately, sometimes IP-blocked (seedocs/investigations/ youtube_429_block.md) — confirmed live, 2026-09-09, on a Mac where caption fetches were failing that same day: the audio download worked fine.--urls-file path.txt(one URL per line,#-comments and blank lines skipped) feeds a specific pre-filtered candidate list through this same pipeline in one run, bypassing the oldest-first backlog queue the same way--urldoes for a single meeting. - Pushes with
"source": "transcribed"explicitly viaPOST /internal/ingest(now accepts an optionalsourcefield, default"scraped"for every other caller) — the same real AI-transcript disclaimer the worker's own output gets, not a silent "scraped" (i.e. authoritative government caption) mislabel. Dedup is scoped bysourcetoo, so a repeat run against an already-transcribed meeting is a no-op rather than a duplicate version. - Never touches
transcription_jobs/claim_next_chunk()— this script is not a worker. It runs its own Whisper and never claims a queued chunk. It discovers and pushes purely over the same token-gated/internal/*HTTP surfacescripts/fetch_youtube_transcripts.pyalready established, so it can safely run at the same time as both cloud workers. - Built for an unattended overnight run someone who isn't a developer
checks on, not just an interactive one — added 2026-08-17 after a
real multi-hour run showed zero output in its redirected log file
despite being alive (
print()fully buffers on a redirected stream; seeBACKLOG_DONE.md). Progress now goes through Python'sloggingmodule (same conventionworker/main.pyalready uses), which flushes every line immediately even when piped to a file — plain-English, timestamped lines for the run's real config, each meeting starting/ finishing, running ingested/skipped/failed totals after every meeting (not just at the end), and retries. Every call to the Archive's own/internal/*API (candidate list fetch, ingest push, promote) retries a 5xx or connection-level failure with exponential backoff instead of crashing the whole run — real incident: a transient 502 on the very first call used to end the entire batch before the main loop even started. If an ingest push still fails after retries, the finished transcription (the expensive part) is saved tolocal_transcription_ backups/rather than discarded, recoverable with a plaincurlonce the Archive is reachable again. A detected wall-clock-vs-processing-time gap (the machine likely slept, or a request stalled) gets logged explicitly rather than passing silently. Seetests/test_transcribe_ backlog_locally.pyfor retry/gap-detection coverage against a real local HTTP server (not a mocked session). - One transient failure no longer costs a whole meeting (2026-08-22,
after ten-plus confirmed cases across three sessions — see
BACKLOG_DONE.md). The three calls that reach a live government source —finder.resolve(),probe_duration(),extract_chunk_audio()— now retry with backoff through the sharedapp/utils/retry.pypolicy instead of recording a meeting as unusable on the first failure: four meetings marked permanently unresolvable all succeeded on an unchanged re-run minutes later, and 3 of 4 failednew.swagit.comextractions cleared on an immediate retry. Genuinely permanent failures (an unregistered platform,ffmpegmissing from PATH) still fail on the first attempt with no delay, the same discipline the Archive-API retry applies to a 4xx. And a chunk that still fails checkpoints the chunks already transcribed tolocal_transcription_backups/partial/, so the next run resumes from there rather than starting over — a real 55-chunk meeting previously discarded 50 finished chunks (~44 minutes of Whisper compute) after failing on chunk 51.--no-resumeignores a checkpoint; one is refused automatically if--chunk-secondsor the source's duration has changed since it was written.MEDIA_ATTEMPTSis 2 (one retry) and deliberately a tunable constant, not a claim that one retry is always enough — one confirmed case (Brookhaven NY) failed two identical back-to-back retries.worker/main.py's idle-time auto-generation got the same treatment for its own feasibility check; its chunk-processing path never had the gap (see "On-demand transcription" above — chunk failures already get three tries plus job-level retries, and partial segments are persisted per job). - Thermal pacing, for a real multi-day unattended run on an older/
fanless Mac (added 2026-08-21). Nothing above throttles CPU usage by
default — a genuinely long batch can run the CPU at a sustained high
clock for hours at a stretch otherwise, a real overheating risk on
hardware without much thermal headroom. Two independent knobs, meant to
be used together:
--cpu-threads Ncaps CTranslate2 to N CPU threads (default: half this machine's real physical core count);--chunk- cooldown-seconds Nrests N seconds after every ~900s chunk — a real duty-cycle rest at the natural per-chunk boundary, not just between meetings — and also polls macOS's ownCPU_Speed_Limit(pmset -g therm, no sudo needed) after each cooldown, waiting longer if the OS has already started throttling. Live-verified against a real ~5.25-hour meeting (Albemarle, NC) at--cpu-threads 2:CPU_Speed_ Limitstayed at 100% (no throttling) for the entire run. Wrap the whole invocation incaffeinate -s(prevents system sleep on AC power only) so a multi-day run survives the machine's own idle-sleep timer. - A fresh Homebrew-Python venv has an empty default SSL trust store,
breaking every
aiohttpcall in this script (and six other local scripts) withSSLCertVerificationError— easy to mistake for a real network outage. Fixed by settingSSL_CERT_FILEfromcertifibeforeimport aiohttpspecifically (not just before first use — aiohttp caches its defaultSSLContextat import time). SeeCLAUDE.md's matching convention bullet for the full incident and why the ordering matters.
Live-verified 2026-08-16 against a real backlog meeting (Welland/Elgin
County, ON — welland-2026-01-27-county-council-meeting, a real 783-second
eScribe recording with no prior transcript): --model-size small produced
102 real, coherent segments (language detected en) in 113 seconds,
pushed successfully, and the AI TRANSCRIPT disclaimer + real timestamped
segments (starting "We're live." at 0:00, ending with real adjournment/
motion dialogue) are live on the actual public page. The meeting no
longer appears in a follow-up /internal/transcription-backlog call.
Giving both cloud workers real concurrent work (2026-08-21; moved into
the worker itself 2026-09-27, WO-1148). The idle-time trickle
(maybe_generate_auto_job()) adds one job at a time, and only when this
worker finds nothing to claim. A single job's chunks run one after
another, so the second worker (rtr-transcription-worker-2) only gets
real parallel work once two or more different jobs are queued.
The batch top-up closes that gap. maybe_generate_batch_auto_jobs()
(worker/main.py) runs every 30 minutes on rtr-transcription-worker
only, gated by BULK_TOPUP_ENABLED in render.yaml. Each pass pulls up
to 100 candidates from crud.list_transcription_backlog_candidates(),
runs the same live feasibility check the on-demand path uses, and creates
up to 8 low-priority jobs. YouTube-backed pages are skipped: the cloud
worker can't fetch YouTube (see "Permanent pages" above).
Why it runs on the worker, not somewhere else. Until 2026-09-27 this ran
as an hourly GitHub Actions cron
(.github/workflows/bulk-queue-transcription-backlog.yml). Granicus,
Cablecast, IQM2, ChampDS and CivicClerk all refused GitHub's servers with
a 403, 404 or timeout, while serving the workers' own address normally.
Each refusal was recorded as a failed probe, which puts a page into
cooldown for 1 to 30 days. So the cron quietly cooled down most of the
backlog while both workers sat idle. The workflow is now disabled in
GitHub. Pages it cooled down still wait out their cooldown (see
BACKLOG.md's matching [NEEDS-AUDIT] entry).
scripts/bulk_queue_transcription_backlog.py still exists for a manual
or dry run:
python scripts/bulk_queue_transcription_backlog.py --dry-run
python scripts/bulk_queue_transcription_backlog.py --limit 4Run it from a Mac on a normal connection, never from GitHub Actions, for
the reason above. Its batch size defaults to 8, well under
archive/db/crud.py's global MAX_CONCURRENT_TRANSCRIPTION_JOBS = 15
(shared across every priority tier). That leaves headroom, so a visitor's
own request never hits too_many_active_jobs, and PRIORITY_LOW means a
visitor's job still goes ahead at the next claim. clerk_verified=True
on each created job skips the confirmation email: this script holds
ARCHIVE_INGEST_TOKEN, the same trusted-caller position as the
resolver. Without it, a job would wait at pending_confirmation until
someone clicked a link.
New meetings now come in through the YouTube drip's direct lane, not
GitHub, added 2026-09-29 (WO-1168). The two paragraphs above keep the
cloud workers busy once a meeting is on the site with no transcript.
Something else has to put new meetings on the site in the first place.
That used to be a GitHub Actions job
(.github/workflows/feed-tier3-transcription.yml), feeding 12 lines from
scripts/tier3_auto_transcription_queue.txt every 6 hours. On
2026-09-29 both cloud workers sat idle — 0 active jobs, 1 finished in a
day — because Granicus and Cablecast, most of that queue, refuse GitHub's
own IP addresses with an outright "no" (HTTP 403), even with the same
request headers that work fine from an office connection. The same push,
run by hand from an office Mac that day, ingested 31 of 40 lines. So this
now runs from scripts/youtube_drip.py's own direct lane (see
docs/YOUTUBE_DRIP_RUNBOOK.md) — the always-on Mac that already handles
YouTube for the same "GitHub gets refused" reason. It only feeds while
fewer than 10 (by default) non-YouTube, non-Vimeo pages are waiting for a
transcript, so it tops the queue up instead of dumping hundreds of lines
in at once. The GitHub workflow file stays in the repo, disabled, in case
this ever needs to move again.
Daily activity report, added 2026-08-21. GET /internal/send-worker- daily-report (Archive service, token-gated like every other
/internal/* route) composes and emails a plain-text-style HTML digest —
chunks completed, jobs finished, and (Postgres only) segments transcribed
in the last 24 hours, plus a snapshot of what's still ahead: active jobs,
remaining chunks in those jobs, meetings on the site with no transcript,
and how many URLs are still sitting in the tier-3 discovery queue
(scripts/tier3_auto_transcription_queue.txt, read directly off disk —
this service's own deploy already checks out the whole repo). Triggered
daily by .github/workflows/worker-daily-report.yml, which is a plain
curl ping — same pattern /admin/send-search-alerts already
established: GitHub Actions never touches RESEND_API_KEY or
DATABASE_URL directly, it just pings an already-running Render service
that already holds those credentials, rather than a new script
duplicating them as fresh GitHub secrets.
The report also carries a failure digest, added 2026-08-23 (WO-46) —
every job that reached failed in the last 24 hours, grouped by reason,
each row linking both the Archive page and the source URL that actually
failed, with a chunks n/N marker (0/1 means nothing was ever
attempted). It exists because the per-job failure emails cover only half
the failures: worker/main.py's _send_failure_email() has exactly one
call site, reachable only from the report_chunk_result(success=False)
paths, so anything that dies earlier — a re-resolve finding no media,
ffprobe failing, the is_plausible_meeting_duration() gate — is
recorded in the database and never emailed at all. That is not a
small blind spot: on 2026-08-23 an IQM2 cluster of ~20 jobs, the second
largest of three that day, produced zero emails while a handful of
multi-chunk Cablecast jobs emailed repeatedly. A clean day renders an
explicit "None" line rather than dropping the section, and a bad day is
capped at MAX_FAILURES_LISTED with the dropped count stated, so a
truncated digest never understates it. See
crud.list_recent_transcription_failures() and
email._render_failure_digest().
The 24h chunk-completion figure needs a real reference point to diff
against, since TranscriptionJob.chunks_completed has no per-chunk
timestamp anywhere in the schema — WorkerReportSnapshot
(archive/db/models.py, one row, overwritten on every send) holds
exactly that: the cumulative all-time totals as of the last report. The
first-ever send has nothing to diff against and reports "n/a (first
report)" for that one figure rather than a misleading number.
GET /internal/transcription-queue-stats exposes the same live summary
read-only (no snapshot advance), for anything else that wants it later
(a dashboard, say) without needing to trigger a real email.
Shipped 2026-08-11, phase 1: sign in and save meetings/searches to your
own account. Deliberately narrow scope — no public profile pages, no
posts/reposts, no billing yet (see BACKLOG.md's "Accounts + token
billing" section for what's still ahead). A hard non-goal, tested for:
nothing that worked anonymously before this shipped requires an account
now — browsing, searching, watching a meeting, and reading a transcript
are all completely unaffected by whether you're signed in. The only new,
purely additive things a signed-in visitor sees are a "Save this
meeting"/"Save this search" button, a bookmark icon by the meeting title,
a "My Saved Items" nav link, and a user avatar.
| Works for everyone, no sign-in | Requires a signed-in Clerk session |
|---|---|
| Resolving/watching a meeting, transcript, and agenda | Saving a meeting (POST /api/account/save-meeting) |
| Deep-linking — seeking, "Copy link to current time", "Go to time" | Saving a search (POST /api/account/save-search) |
| In-page transcript search, transcript download (Text/SRT) | Unsaving either of the above |
/meetings site-wide search across the Archive |
"My Saved Items" (/account/saved) |
| "Report a problem with this meeting" | |
Reading the Full Context feed (/context, /context/feed.xml) |
Posting to Full Context (/context/new) — needs an account and editor status (CONTEXT_EDITOR_CLERK_IDS), see "Full Context feed" below |
| Requesting on-demand transcription from audio — email only, see "On-demand transcription" below |
Requesting a transcription is not Clerk-gated — by deliberate design,
not as a stepping-stone toward eventually requiring a full account. It's
the app's most cost-intensive feature by far (see "On-demand transcription"
below for real dollar/compute figures), and email confirmation (a one-click
click-through the first time) is the intentional middle path between
"fully open" and "requires a real account" — real friction against abuse
without putting a login wall in front of the app's single costliest
feature. See BACKLOG.md's "Accounts + token billing — phases 2-6" entry
(under "Roadmap & strategy") for the broader account/billing thinking
this sits alongside.
(This paragraph used to flag an open UX gap — the "Save this meeting"/
"Save this search" buttons rendering for signed-out visitors. That
premise was investigated 2026-08-13 and turned out to be false: both
templates have gated on {% if active_account %} since the first
accounts commits. Corrected here 2026-08-21; see BACKLOG_DONE.md.)
Why Clerk, not a hand-rolled session system. The user explicitly chose
a third-party auth provider over building/maintaining login, sessions, and
password security by hand ("I'm kind of leaning away from becoming a
security expert" — see BACKLOG.md for the full tradeoff discussion).
Clerk also gives a real privacy-posture improvement as a side effect: this
app's own database never stores an email address, name, or any other
PII for an account — only Clerk's opaque user id (e.g. user_2abc...).
Clerk holds the actual identity data; this app just remembers what that id
saved.
Architecture. Clerk issues a signed session JWT in a __session
cookie on the shared public domain. Both services verify it locally and
independently — no session table, no per-request DB lookup, no
cross-service "who owns this session" problem:
app/utils/clerk_auth.py/archive/utils/clerk_auth.py(deliberately duplicated, same reasoning as this repo's other cross-service duplication) —get_clerk_user_id(request)returns the signed-in visitor's Clerk user id orNone, never raises, and returnsNoneimmediately (no verification work at all) wheneverCLERK_SECRET_KEYisn't set or there's no session cookie/Authorization header present — so a Clerk outage or missing config degrades to "everyone looks signed out," never a broken page.clerk_frontend_api_url(publishable_key)derives Clerk's per-account "Frontend API" domain from the publishable key itself (matching Clerk's own@clerk/sharedparsePublishableKey), so the ClerkJS script tag can be built without a hand-copied snippet.shared_static/clerk_nav.js— loads ClerkJS via a CDN script tag (using that derived Frontend API domain), mounts the nav's sign-in link/user avatar, and exposeswindow.RTRClerkfor other page scripts. Entirely client-side and entirely optional: ifCLERK_PUBLISHABLE_KEYisn't set, this does nothing at all, no script load attempted.- Cookie forwarding through the reverse proxy:
/meetings,/m/*, and/account/savedare all rendered by the Archive service but reached through the resolver's proxy (see "Permanent pages (the Archive)" above) —app/archive_client.py'sproxy_get()forwards the incoming request's rawCookieheader to Archive on exactly these three routes (not static assets/sitemap/feed, which don't need it), so Archive can verify the session itself with zero extra network round-trip.
Data model. One new table, SavedItem (archive/db/models.py) — it
lives in Archive's database (not the resolver's), since it needs a real
same-database foreign key to MeetingPage.id. Columns: clerk_user_id
(indexed, opaque — never an email), item_type (saved_meeting /
saved_search), meeting_page_id (nullable FK, set only for
saved_meeting), search_params (nullable JSON — the same filter dict
/meetings already accepts, set only for saved_search), created_at.
Unsaving is a hard delete; no soft-delete/undo.
Routes:
- Resolver:
POST /api/account/save-meeting/unsave-meeting/save-search/unsave-search(public, 401not_logged_inif no valid session) — each verifies the session locally, then calls a bearer-gated/internal/account/*route on Archive with the already-verifiedclerk_user_id, the same trust pattern every other resolver→Archive internal call already uses.POST /api/clerk/webhookhandles Clerk'suser.created(auto-subscribes the address to the Resend newsletter audience, sends the "Thanks" email — see "Lifecycle emails" below) anduser.deleted(purges the account'sSavedItemrows — the right-to-deletion story on this app's side, since it stores no other PII for an account at all). Verified viasvixbefore trusting anything in the payload. - Archive: the bearer-gated
/internal/account/*counterparts, plusGET /account/saved(the "My Saved Items" page — a friendly "sign in to save things" state for an anonymous visitor, not an error).
Lifecycle emails. Five of the six emails from rtr-business's
marketing/LIFECYCLE_EMAILS.md (approved copy/voice) are live: "Thanks"
(account created), "Welcome" (newsletter-only signup), "Goodbye for now"
(unsubscribed), a branded "Your transcript's ready" (rewrite of the
existing completion email, keeping its AI-transcript disclaimer), and "We
couldn't cook this one" (new — fires when a transcription job gives up
after repeated chunk failures). The resolver gained its own transactional
Resend-send capability for this (app/main.py's _resend_send() +
branded template, duplicated from archive/utils/email.py's equivalent)
— previously it only ever upserted Resend audience contacts. Saved-search
alert emails (the doc's sixth entry, "People are talking about…") are
also live: archive/search_alerts.py, run daily by
.github/workflows/send-search-alerts.yml via
scripts/send_search_alerts.py, matches new results against saved
searches and sends a per-alert email with its own unsubscribe token.
Env vars — see .env.example / archive/.env.example for the full,
current list and setup notes. CLERK_PUBLISHABLE_KEY + CLERK_SECRET_KEY
on both services; CLERK_JWT_KEY (optional — leave it blank unless you
have a specific reason not to, see the env file's comment for why a
malformed value here is strictly worse than an absent one);
CLERK_WEBHOOK_SIGNING_SECRET on the resolver only;
RESEND_FROM_ADDRESS/RESEND_REPLY_TO_ADDRESS newly needed on the
resolver as of this feature.
Real incidents from the production cutover (dev instance → real
Clerk production instance, custom-domain DNS, a new webhook) — all found
live and now fixed, full writeup in BACKLOG_DONE.md's "Clerk production
cutover" entry: a base64-padding bug in clerk_frontend_api_url() that
disabled Clerk site-wide, a CSS-specificity bug that left a stray nav
divider visible when signed in, and a malformed CLERK_JWT_KEY that
silently made every signed-in visitor look signed out server-side while
looking completely normal client-side. Worth reading in full before
touching Clerk env vars/DNS again.
Deliberately deferred, not yet verified end to end: the
user.deleted webhook → SavedItem purge (the right-to-deletion
cascade) has unit test coverage but has never actually been fired for
real against a deleted Clerk account. See BACKLOG.md for the user's
explicit call on this.
Two independent pieces of plumbing, easy to conflate but separate concerns:
Outbound (sending) — Resend (archive/utils/email.py, and
app/main.py's own _resend_send() + branded templates as of the
accounts/lifecycle-emails feature — previously the resolver only ever
upserted Resend audience contacts, never sent transactional mail).
RESEND_API_KEY + RESEND_FROM_ADDRESS (currently Ryan <ryan@ally.redtaperecordings.com> — ally.redtaperecordings.com is
Resend's verified sending subdomain, its DNS untouched since setup) +
RESEND_REPLY_TO_ADDRESS (currently ryan@redtaperecordings.com, the
root domain — set on all three Render services: resolver, Archive,
worker) + RESEND_AUDIENCE_ID (the newsletter audience). See "Accounts
(Clerk)" above for which lifecycle emails are actually live.
Inbound (receiving) — redtaperecordings.com's MX records point at
ImprovMX (free forwarding: mx1/mx2.improvmx.com plus an SPF TXT
record), which forwards ryan@, ally@, and a wildcard catch-all, all
to ryan@how-to-adu.com. Set up and managed entirely in Namecheap DNS +
the ImprovMX dashboard — no code involved, nothing to configure
per-environment. One subdomain has its own separate, unrelated MX:
send.ally.redtaperecordings.com → feedback-smtp.us-east-1.amazonses.com,
Resend's own bounce-handling record for its SES backend — don't touch it
when editing DNS here even though it looks similar to the ImprovMX pair,
since it's a different subdomain serving a different purpose. Full setup
history (why Namecheap's own forwarding wizard couldn't be used
directly, both live-verification passes) is in BACKLOG_DONE.md's
"Email deliverability" section.
Site-facing addresses currently shown/used are a mix, not yet
consolidated: footer mailto: links use ryan@redtaperecordings.com
(app/templates/base.html, archive/templates/base.html), the About
page shows ryan@how-to-adu.com directly
(app/templates/about.html), and RESEND_REPLY_TO_ADDRESS is
ryan@redtaperecordings.com. Auditing and consolidating these onto
ally@redtaperecordings.com is an open BACKLOG.md item as of
2026-08-12.
Ops-only addresses, unrelated to the site and out of scope for the
above: DAILY_REPORT_EMAIL_TO and YOUTUBE_FETCH_REPORT_EMAIL (both in
scripts/, both default to ryan@how-to-adu.com) send operator
digests, not user-facing mail.
When a brand-new permanent page is created in the Archive and the
resolve behind it was genuinely high-quality, the Archive can announce it
on a Bluesky and/or Mastodon account automatically
(archive/utils/social.py, hooked into /internal/ingest). Off by
default — with no credentials set the feature is completely inert.
What counts as "high quality" mirrors app/db/outcomes.py's
success bucket (the same definition /admin/stats reports on): a real
video URL, a real transcript, no garbled-source/hallucination warning,
and English-or-undetected language — plus a minimum transcript length
this feature adds on top (SOCIAL_MIN_SEGMENTS, default 50 segments),
because a technically-successful caption stub isn't worth a public post.
Agenda-only pages, garbled transcripts (e.g. the Fountain Valley case),
and non-English detections never post.
Provenance is a separate bar, and it's absolute: nothing that came
out of generic_fallback.py's scan-any-page path is ever announced —
best_effort, or platform == "unknown" — however good its transcript
looks. Everything else on the list above asks "is this content good?",
which is a different question from "do we actually know what this is?",
and a fallback resolve can score perfectly on all of it while nothing has
verified the scraped page is a genuine government meeting. Both checks
are needed, not one: the fallback delegates to YouTube whenever it finds
an embed, and those results carry platform = "youtube", so a
platform-only check misses the most common real case (see
ResolvedMeeting.best_effort). Added 2026-08-21 — the pipeline shipped
without it, see BACKLOG_DONE.md. Note this deliberately does not
extend to noindex/sitemap/hub visibility: those pages stay indexed on
purpose, see BACKLOG.md.
When it fires: only when /internal/ingest creates the page.
Re-ingests — the resolver's push-retry sweep, stale-page rechecks,
scripts/backfill_archived_pages.py's corpus-wide re-resolve — can never
trigger a post, and a SocialPost table row per (page, network),
claimed before the network call under a unique constraint, makes
posting at-most-once even under a race. A failed post is recorded and
deliberately never retried automatically (a missed announcement is
cheap; a duplicate on a public feed isn't). Announcements are also
spaced at least SOCIAL_MIN_POST_INTERVAL_SECONDS apart (default 180s):
a burst of qualifying new pages — several lookups at once, or a bulk
seed — posts at most one per window, with the rest skipped outright
rather than queued. Known residual: a page created agenda-only that
gains a real transcript later never gets announced — see BACKLOG.md.
Setup (the human half — none of this can be automated):
- Create the account: for Bluesky this is done — the real account is
redtaperecordings.bsky.social, registered 2026-08-21 (a custom@redtaperecordings.comdomain handle can be set later without changing anything here except theBLUESKY_HANDLEvalue). For Mastodon, pick an instance and register there. - Get credentials: Bluesky — Settings → App Passwords → create one
(never use the real account password); Mastodon — Preferences →
Development → New application with the
write:statusesscope, copy the access token. - Set env vars on the Archive Render service (see
archive/.env.example):BLUESKY_HANDLE+BLUESKY_APP_PASSWORDand/orMASTODON_BASE_URL+MASTODON_ACCESS_TOKEN.PUBLIC_BASE_URLmust also be set (it already is in production) — posts link to{PUBLIC_BASE_URL}/m/{slug}.
Either network can be enabled alone; each posts and dedups independently. Post text is composed once ("Somebody looked up {title} — you can now search a transcript of that meeting and link to specific timestamps at {url}", plus a "{jurisdiction} — {date}" footer line), capped at 300 characters (Bluesky's limit, the stricter of the two). The sentence and permalink always survive whole — an over-long title is truncated with an ellipsis, and the footer is dropped before the title would be gutted. A proper Bluesky link facet makes the URL clickable.
Live status: Bluesky posting is live-verified — the first real post from
redtaperecordings.bsky.sociallanded 2026-08-21 off a real prod resolve (seeBACKLOG_DONE.md). The Mastodon client is written against the documented API but has never made a real post (no account exists yet) — treat it as best-effort until one is watched, same as any schema-verified-but-not-content-verified path in this repo; seeBACKLOG.mdfor the open residuals.
Private research intake (WO-1014). Editors can review imported research at
/context/candidates. CSV/JSON imports preserve source observations, check the
existing Archive for the exact recording, and show a next action. Editors can
save reviewed fields, follow a proposed timestamp, and open the Context editor
with the saved values. This does not create or publish entries or ingest meetings. See
the operator guide for preview/apply commands,
source mapping, statuses, and release order.
Why this exists. Ryan finds many short clips of public meetings
circulating on Instagram, TikTok, and YouTube. A clip travels on its own,
stripped of the rest of the meeting, and some draw real community
attention (comments, reposts, news pickup) — but a viewer who wants the
full context has no way to get from the clip back to the actual moment in
the actual meeting. /context is a public feed of short, editor-written
entries, each linking a real social post to the exact deep-linked second
of the meeting it came from.
What an entry shows. Every published entry always carries: an
editor-written summary (max 500 characters — never fetched or generated,
see "No server-side fetch" below); a link out to the original post
(rel="noopener nofollow ugc"); the meeting's own card image (see
"Meeting card images" above); a match label; and a "Watch the full
context" deep link, /m/{slug}?t={seconds}. An entry may also carry an
optional short title (max 120 characters, CONTEXT_TITLE_MAX,
WO-945) — the editor's own headline for the post, shown as its own
heading above the match badge when set. It's entirely optional; an entry
with none looks exactly as it did before this existed.
Match labels say how precisely the clip maps onto the meeting:
| Label | Reader-facing text | What it means |
|---|---|---|
exact |
"Exact moment" | The linked timestamp is the clip itself. Requires a t value to publish. |
approximate |
"Same meeting, moment approximate" | The right meeting, but no defensible single second. |
related |
"Related meeting" | A different, related meeting than the one the clip shows. |
Embeds exist only for three networks, and most of them are still
click-to-load. For an Instagram, TikTok, or YouTube post, the entry
also offers a "Show the post here" button. The embed itself — an
official embed.js widget for Instagram/TikTok, a youtube-nocookie.com
iframe for YouTube — normally loads only after the reader clicks it.
As of WO-945 (2026-09-21), that's no longer the whole story: on
page 1 of the public feed only, the first CONTEXT_AUTOLOAD_EMBEDS
(3) entries that have an embed load it on their own, lazily — an
IntersectionObserver with a generous rootMargin starts the load as
the entry nears the viewport, not the instant the page opens, and every
entry after those first three still requires a click, as does every
entry on page 2 onward and everything on the editor's own list
(/context/new). This is a deliberate, capped trade-off: Ryan wanted the
feed to look less empty with the players already showing, but every
embed is still a real third-party request and page weight — which is
exactly why WO-943 made embeds click-to-load in the first place — so the
autoload count stays small and page-1-only rather than growing with the
feed. One exception predates all of this, confirmed by watching the real
page's requests: when the meeting is hosted on YouTube, its card image
loads from i.ytimg.com as the page opens, autoload or not. This
loading behavior is a product choice, not a promise: the privacy page
deliberately describes third-party content in general terms (Ryan,
2026-09-21), so changing how embeds load needs no policy edit, and
tests/test_privacy_page.py keeps feature-level promises from creeping
back in. This site has no cookie-consent banner, which is exactly why the
count stays capped rather than autoloading everything on the page. If an
embed fails, the entry still stands on its own — the link-out and the
deep link both still work. The "Couldn't load it here" fallback fires the
same way whether the embed was click-triggered or autoloaded, when the
network's script is blocked outright. For a deleted post, TikTok's script
still builds its frame and shows its own "unavailable" message inside it
(confirmed live). Every other network (Facebook, X, Threads, Bluesky,
Reddit, LinkedIn, anything else) is link-out only; there is no embed
widget for those, so it never autoloads or shows the button.
No server-side fetch of any social post, ever. Instagram in particular is login-walled to a non-browser client — checked live 2026-09-21 against two real posts opened signed-out: one showed only the account name and a truncated caption, the other was age-restricted and showed nothing at all. So every entry's text is written by hand by an editor, not extracted from the post.
Who can post. A web form at /context/new, open only to a signed-in
Clerk user (see "Accounts (Clerk)" above) whose Clerk user id appears in
the env var CONTEXT_EDITOR_CLERK_IDS (comma-separated) on the
Archive service. Everyone else — signed in or not — gets a 404, not
a "you don't have access" page. This is the first "this signed-in user
has a role" mechanism in the repo; before it, admin access meant only a
shared bearer token used with curl, and there were no admin HTML forms at
all.
Silent-failure risk.
CONTEXT_EDITOR_CLERK_IDSmust hold the production Clerk user id (Clerk dashboard → Users), not a development-instance id. A wrong value gives no error anywhere — the form just keeps 404ing. The var fails closed: leaving it unset means nobody is an editor, not everybody.
Drafts and publishing. An entry with no matched meeting is a private
draft — only an editor can see it, on /context/new's own list.
Publishing requires a meeting, a match label, and a summary; exact
additionally requires a timestamp. Statuses: draft, published,
hidden (a published entry pulled back out of public view without
losing its content or its original publish order).
Editor workflow. Find the moment on the meeting page, click "Share
video at M:SS" to copy its link, then paste that link into the form
together with the social post's URL and a short summary. The server
parses the slug and t out of the pasted link itself. line/version
params on a pasted link are ignored for now (see BACKLOG.md).
Architecture. The Archive renders the public pages and owns the writes; the resolver holds the public domain and proxies four GET routes plus two public write endpoints, the same two-hop pattern save-meeting/save-search already use (see "Accounts (Clerk)" above). There is no catch-all proxy — each route is explicit.
| Route | Service | Purpose |
|---|---|---|
GET /context |
resolver → proxies to Archive, forwarding the Cookie header |
the public feed page |
GET /context/feed.xml |
resolver → proxies to Archive, no cookie forwarded | RSS feed |
GET /context/new |
resolver → proxies to Archive, forwarding the Cookie header |
the editor form + draft list |
GET /context/{id} or /context/{id}-{slug} |
resolver → proxies to Archive, forwarding the Cookie header |
one entry's own permalink page (WO-945 follow-up, slugged in WO-946); anything but the current canonical form 301s |
POST /api/context/save |
resolver, public | verifies the Clerk session, forwards the verified user id to Archive |
POST /api/context/set-status |
resolver, public | same — publish/hide/draft transitions |
GET /context, /context/feed.xml, /context/new, /context/{id}[-{slug}] |
Archive | renders the page/feed/form/permalink; /context/new re-checks the editor allowlist itself, not just trusting the resolver |
POST /internal/context/save |
Archive, token-gated | the real write, re-checking the allowlist |
POST /internal/context/set-status |
Archive, token-gated | status transitions, re-checking the allowlist |
New pure modules: archive/utils/context_links.py (parses a pasted
social URL and a pasted RTR share link, and describes each network's
embed) and archive/utils/context_editors.py (the allowlist check).
Front-end: archive/static/context_embeds.js (click-to-load, plus the
page-1 autoload path — WO-945),
archive/static/context_editor.js (the form). Templates:
archive/templates/context.html, _context_entry.html,
context_new.html, context_feed.xml.jinja, context_entry_page.html
(the permalink page, WO-945 follow-up).
Data model. A new Archive table, context_entries (model
ContextEntry, archive/db/models.py), with one Alembic migration —
named to avoid the existing SocialPost table, which is this app's own
outbound Bluesky/Mastodon auto-poster (see "Social auto-posting"
above) and is otherwise unrelated. Columns: social_url (the canonical
URL) and social_url_key (the dedupe identity — instagram:{shortcode},
tiktok:{id}, youtube:{id}, or else the normalized URL, unique —
so the same post can't be entered twice), network, source_label,
summary, title (nullable, optional, added WO-945 — the editor's short
headline for the entry, capped in app code at CONTEXT_TITLE_MAX
regardless of the wider column), meeting_page_id (nullable FK,
ON DELETE SET NULL), t_seconds, match_kind, status,
created_by_clerk_user_id, published_at, created_at, updated_at.
Stores no PII — only Clerk's opaque user id as author, same
convention as SavedItem (see "Accounts (Clerk)" above). status and
created_by_clerk_user_id exist even though only allowlisted editors can
write today, so that open submissions with a review queue are a later
schema-free addition — see BACKLOG.md. On Clerk's user.deleted
webhook, an entry's author id is nulled and the entry itself is kept
(it's editorial content, not the author's personal data). When a meeting
page is deleted (delete_meeting_pages_by_slug), its published entries
are demoted back to drafts, so they reappear in the editor's own list
instead of quietly pointing at nothing.
title vs. headline — a deliberate naming split. The DB column and
the API/form field are both called title — the word Ryan uses for an
entry's own short headline. But the entry dict every reader (the public
feed, the RSS feed, the editor form) actually gets back already has a key
called title that means something else entirely: the matched
meeting's own title (MeetingPage.title, joined in at read time). To
avoid that collision, archive/db/crud.py reads the entry's own title
back under a different dict key, headline, and leaves title alone
for the meeting's. So: write title, read headline for the entry's own
one, read title for the meeting's. See that file's "Full Context feed"
section (_entry_row_to_dict()/_context_entry_dict()) for where the
rename happens.
SEO. /context is noindex and left out of the sitemap until it has
at least CONTEXT_MIN_INDEXABLE (5) published entries — the same
thin-page reasoning as the /state/*//j/* hub pages (see
STATE_HUB_PAGES.md). Pages after the first are always noindex.
The feed uses a descriptive public-meeting-clips title and description.
Its indexable first page includes CollectionPage / ItemList JSON-LD
with the visible entries in display order, linking to their canonical
permalinks. This describes the collection without claiming a rich result.
Permalink pages (/context/{id} or /context/{id}-{slug}, WO-945
follow-up, slugged and given real SEO substance in WO-946). The feed's
own per-entry anchor (#context-{id}) isn't a stable link — an entry
slides to page 2 as newer ones publish. Every entry also gets a real URL
that 404s for anything that isn't a published entry with a real,
still-existing meeting (a draft, a hidden entry, an orphan whose meeting
was deleted, or an unknown id) — even for the signed-in editor, who
previews a draft on /context/new instead. When an entry has a title,
that title is the link to its permalink everywhere it appears (the feed,
the editor list); the permalink page itself renders that same title as
its own <h1> (or, absent one, the matched meeting's title). An entry
with no title is still linkable — a small "Link to this post" text link
on the feed, a "View post" action in the editor list. The permalink
page's own embed always autoloads (unlike the feed's capped, page-1-only
autoload — see above), since a reader who opened one specific post wants
to see it. Indexing follows the feed's own threshold
(CONTEXT_MIN_INDEXABLE), not a per-entry decision.
The URL is slugged, but the id is still the real identity. The
canonical form is /context/{id}-{slug} — context_links.context_ permalink() builds the slug from the entry's own title, or, when it has
none, from the matched meeting's jurisdiction + title, the same two facts
a reader would use to recognize the meeting from the feed card. It's
computed fresh on every read, not stored, so editing a title changes the
slug for free with no backfill. The route (archive/main.py's
context_entry_page()) looks the entry up by id alone and never trusts
the slug for that; any request that isn't already at the entry's current
canonical permalink — a bare id, a stale slug left over from before a
title edit, a mistyped slug — gets a real 301 to the canonical one,
query string preserved. The resolver's proxy (app/main.py) widens the
same way and passes allow_redirects=False so that 301 reaches the
browser instead of being followed and silently served as a 200 — the
same real bug (and the same fix) /m/{slug} and /j/{hub_slug} already
had to be protected against; see those routes' own comments. An entry
with no title and no meeting yet (a fresh draft) gets a bare /context/ {id}, same as before.
The entry page isn't thin anymore (WO-946). STATE_HUB_PAGES.md
diagnosed Google declining this site's hub pages for being thin and
templated; a permalink page with 2-3 sentences of editor summary was
exactly that same shape. When the entry has a timestamp and the matched
meeting has a real, non-garbled transcript, the page now also shows a
short excerpt of what was actually said at that moment —
crud.get_context_transcript_excerpt() starts at the segment containing
the clip's t_seconds and grows the window to about 90 seconds or 900
characters of real transcript text (whichever comes first, with a floor
of 2 segments), reusing the exact same "is this actually a good
transcript" check (_has_real_warning_free_transcript()) every other
quality-gated reader in archive/db/crud.py already uses — a garbled,
likely-hallucinated, or truncated transcript never gets excerpted. The
chosen line must also sit within 60 seconds of the linked moment. Without
that, a transcript that stops early handed back its last line for a
moment an hour later, and unrelated speech presented as context is worse
than no excerpt. The page shows paragraphs, not raw caption lines:
government captions arrive a few words at a time (the real Jacksonville
excerpt was 34 lines such as "[12:34] yourself."), so
crud._excerpt_paragraphs() groups them, breaking at a speaker change
(the >> captioners use) or at a sentence end once a paragraph is long
enough. Each paragraph links to the moment it opens with (?t=&line=seg-N& version=, the same convention archive/templates/meeting_page.html
and shared_static/deep_link.js already agree on), and the section
carries the same honesty note about automatic captions/transcripts the
meeting page itself shows. Skipped entirely — no section at all, not an
empty one — when there's no timestamp, no segments, or the transcript
fails that quality check. Only ever loaded on the entry page, never the
feed: a transcript's segments JSON is six-figure bytes per meeting, and
the feed can show 20 entries at once.
Structured data (WO-946). The entry page's <title> names the
government when the headline doesn't already. If the line would run past
about 65 characters, the " | Red Tape Recordings" suffix is dropped. The
headline and the government never are: the place is what people search
for. A BlogPosting JSON-LD block (headline, description,
datePublished/dateModified, image when there's a card, isBasedOn
the meeting page, citation the social post) sits alongside a
BreadcrumbList (Home → Full Context → this entry) — every value goes
through Jinja's |tojson, same as meeting_page.html's own JSON-LD, so
an editor-written headline containing </script> can't break out of the
block. A visible <time> under the headline and a matching
article:published_time meta tag carry the publish date.
Sitemap and RSS. Once the feed clears CONTEXT_MIN_INDEXABLE, every
published entry's canonical permalink is also listed in sitemap.xml
(capped at the 500 most recently updated, crud.CONTEXT_SITEMAP_MAX_ ENTRIES) — the sitemap's own /context line already worked this way,
and now individual entries do too. The RSS feed's <item><link> now
points at the entry's own permalink too (WO-1001, 2026-09-21), not the
meeting — "the URL for this item," to a subscriber, is the curated post,
not the recording it cites. The old target — the meeting deep link —
didn't disappear; it moved into <description>, as a plain sentence
("Watch the full meeting from M:SS: {url}") alongside the existing
"Original post:" line, so a reader who only sees the feed still gets to
the recording. <guid> is untouched on purpose — it's what a reader uses
to dedupe already-seen items, and changing it would re-deliver every
existing item as new.
Entries also appear on their government's hub and their state page
(WO-947). Asked by Ryan: "a page like this that is related to
Indianapolis would also appear under the Indianapolis hub, right? And
Indiana?" /j/{hub_slug} and /state/{state_slug} each show a "Seen on
social media" section listing the published entries whose meeting
belongs to them — up to 3 on a hub, 6 on a state — each linking to its
own permalink page, which had almost no internal links pointing at it
before this. Membership is decided the exact same way the page decides
which meetings it shows (never by matching the entry's stored
jurisdiction text) — see STATE_HUB_PAGES.md's "Seen on social media"
section for why that distinction is load-bearing here. The section is
absent entirely (not shown empty) when there's nothing to list, and only
shows on the bare page, not under a ?topic= filter. The link runs both
ways: an entry's own permalink page names its government (already
linking to its hub since WO-943) and now also its state, next to it —
crud.effective_state_abbr(), a pure function over data the entry
already carries, not a query.
A published entry can also make a thin hub indexable (WO-1003). A
/j/{hub_slug} page is normally noindex'd below JURISDICTION_HUB_MIN_ INDEXABLE (2) meetings, but a hub with just one meeting and a real,
hand-written Full Context entry is no longer thin/near-duplicate content,
so it's indexable either way now — see STATE_HUB_PAGES.md's "A hub is
also indexable via a published entry" section for the rule and where it's
applied (the page's own noindex tag and sitemap.xml's /j/ list, kept
in agreement). State pages have no such meeting-count gate to extend.
And the meeting page itself, the one surface an entry cites without
linking back until now (WO-1002). /m/{slug} shows a "This moment on
social media" (or, when no shown entry has a real timestamp, "This
meeting on social media") block for the published entries that cite that
exact meeting — up to MEETING_CONTEXT_ENTRIES (5), newest first,
reusing the same shared _context_mentions.html partial the hub/state
pages use (a meeting_view flag drops the redundant meeting title/date
line and turns each entry's timestamp into its own "at M:SS" link to the
clipped second). crud.list_context_entries_for_meeting() is a thin
wrapper over list_context_entries_for_pages() with MeetingPage.id == meeting_page_id as the page condition, run through the same isolated-
session helper the hub/state lookups use so a failure here can never take
the meeting page down with it. Absent entirely — no heading, no wrapper —
when nothing cites this meeting, which is most meetings.
Card-image caveat — a real limitation. /m/{slug}/card.jpg redirects
a YouTube-backed meeting to YouTube's own standard thumbnail, regardless
of t (see "Meeting card images" above). Only a non-YouTube meeting gets
a true frame extracted at the timestamp. Many of the meetings that get
clipped on social media are on YouTube, so many Full Context entries will
show a generic video thumbnail rather than the actual clipped moment.
A second card limit: three frames per meeting. A non-YouTube meeting
stores at most MAX_FRAMES_PER_PAGE (3) extracted frames. Once a meeting
has three, a new entry's exact moment cannot be stored, and the card
route serves the meeting's default frame instead. If there is no
default frame either, the route returns a 404, so the feed shows no
image for that entry rather than a broken one. The feed decides this
with the card route's own rule (exact frame, else default, else nothing)
— see crud._ContextFrames. "The page has some stored frame" is not
enough for an arbitrary timestamp; that shortcut rendered broken images
in the WO-945 browser check.
Deploy order. Deploy the Archive first — it carries the migration
and the new routes — then the resolver, or the nav link 404s in between.
The form stays a 404 until CONTEXT_EDITOR_CLERK_IDS is set on the
Archive service in the Render dashboard (see "Deploys are manual" in
CLAUDE.md).
The lookup box, and below it what the archive already holds (WO-50, 2026-08-24): topic chips, a national recent-moments feed of real transcript quotes each deep-linked to the second it was said, and a browse-by-state row. Before this the page explained a tool and showed nothing of the archive behind it, so a visitor arriving without a meeting URL to paste had nothing to do — on the highest-value page on the domain for indexing, which carried no unique text of its own.
It is rendered by the resolver but the data is the Archive's, so the
resolver fetches it server-side from /internal/home-highlights via
app/archive_client.py's home_highlights() — deliberately not by
reading the shared Postgres, which would let an Archive migration break
the resolver. Both sides cache (300s), the resolver caches failures
too (30s), and HOME_TIMEOUT is a tighter 2s than the 5s lookup budget
because what the wait buys is one optional section.
It degrades to nothing. With the Archive unset, down, cold or slow,
home_highlights() returns None and the page renders the lookup box
alone — the resolver treats the Archive as optional everywhere else and
this page is not an exception. That is the acceptance test for the
feature, not an afterthought.
The chips and cards are the same components the /state/ and /j/ pages
render, shared as one copy rather than duplicated:
shared_templates/_topic_chips.html and _featured_meetings.html (both
Jinja loaders mount the repo-root shared_templates/), styled by
shared_static/highlights.css (both services already mount
/shared-static). The shared partials use no Jinja filters on
purpose — only the Archive's environment registers
jurisdiction_display/meeting_date_html, and depending on them 500'd
the home page while the whole test suite stayed green; display strings
are pre-rendered in crud._featured_entry() instead, which also lets
them survive the JSON hop.
?topic= filters the feed, the same way it does on the state and hub
pages; every variant canonicalizes to bare /.
archive/topics.py is a hand-curated list, deliberately (unsupervised
topic discovery is a declined idea — see BACKLOG.md's Standing
decisions). What it lacked was a way to decide what to add.
GET /internal/topic-candidates (token-gated, read-only, WO-51) ranks
phrases people actually typed into /meetings — out of the identity-free
search_queries table — by frequency and zero-result rate, excluding
anything the curated set already matches. ?phrase=... previews what
adding one would surface: a search_corpus count plus real sample
meetings.
A reader-facing "suggest a topic" form was considered and rejected:
search_queries already collects the same signal passively, at far
higher volume, and records what people looked for rather than what they
say they want. Zero-result searches — the single best source of candidate
topics — are captured there automatically. Curation stays human; this
endpoint ranks and previews, it never adds a topic.
Expect thin output for a while: the table only started filling 2026-08-23.
A /m/{slug} page's description is a real quote from its own
transcript — the stored meeting_highlights row, trimmed on a word
boundary by meta_description() (WO-49, 2026-08-24). Before that, all
~2,400 pages shared one sentence shape ("A public meeting in X on Y,
with video and transcript."), which is exactly the templated-thin
content the state/hub rebuild was diagnosed with, on the page type
Google already indexes best. The <title> carries
meeting/jurisdiction/date, so the description spends its whole budget on
the line unique to the page. Plain text by construction — meta content
cannot carry markup, so this goes through display_text(), never
highlight_html(). A meeting with nothing quotable keeps the generic
sentence; that fallback is load-bearing, not theoretical (808 of 2,362
pages had no usable highlight at the 2026-08-23 backfill).
Every meeting page needs one image: search engines require a
VideoObject.thumbnailUrl before a video is eligible for a rich result
at all, and Bluesky/Mastodon (see "Social auto-posting" above) build
their link cards from og:image. YouTube-backed pages had one from
2026-08-14 (a free, predictable i.ytimg.com URL). Every other page —
direct mp4/m3u8, the majority of the Archive — got a real one on
2026-08-21: a frame pulled out of the meeting's own video with ffmpeg.
Which frame (archive/utils/video_thumbnail.py's
target_offset_seconds()), in priority order:
- The shared moment, plus 20 seconds. When the URL carries
?t=N— someone shared that exact moment — the card shows the frame atN + 20s, landing inside the relevant content rather than on the transition into it. Verified against the real San Carlos meeting:?t=982produces a frame whose on-screen overlay reads "7. CONSENT CALENDAR", the item that timestamp points at. - Otherwise, 300 seconds before the end. Meetings routinely open
with several minutes of dead air behind a static "meeting will begin
shortly" placeholder, so an early-offset frame is frequently a literal
blank slate; near the end there is reliably a real chamber with real
people in it. Needs a duration, which comes from
ffprobe(probe_duration()). - Otherwise halfway through, when the video is too short for (2) to
mean anything — or a fixed 600s when
ffprobecouldn't report a duration at all, since "halfway" isn't computable without one.
Where the bytes live: a meeting_page_thumbnails table, keyed
(meeting_page_id, offset_seconds), one row per extracted frame with the
JPEG stored directly (archive/db/models.py). This repo has no object
storage, no persistent disk and no image library, and ~1200 pages × one
~30-55KB frame is not enough volume to justify adding a vendor. The
unique constraint doubles as the cache: a per-timestamp card is extracted
once and every later fetch is one indexed row read. MAX_FRAMES_PER_PAGE
(12) bounds what a crawl over many distinct ?t= values can store, and
timestamps are quantized into 20s buckets before extraction for the same
reason.
Serving: GET /m/{slug}/card.jpg?t=N (proxied like /m/*) with
Cache-Control and a stored ETag, so a conditional refetch — which
Googlebot and every social scraper do — costs a 304 without ever loading
the image bytes. YouTube-backed pages 302 to their i.ytimg.com
thumbnail, so one card URL shape covers every page.
Extraction never touches the render path. A page render only queues
work (FastAPI BackgroundTasks) — new pages warm at ingest, older ones
the first time anyone loads them. The card route degrades in order: the
exact per-timestamp frame → the page's stored default frame → 404, and a
miss queues the precise extraction so the next fetch is exact. A page
only advertises og:image/thumbnailUrl once a frame is actually stored
(crud.has_thumbnail()) — a card URL that would 404 is worse than none,
since validators and scrapers fetch it. At most 2 extractions run at
once, a failing source is retried at most once every 6 hours, and none of
that can delay a response.
Warming the pages nobody has viewed. New pages warm at ingest and old
ones warm on first view — which leaves every page nobody has loaded since
2026-08-21 imageless indefinitely. Real dry runs that day measured that
backlog at ~1700 pages. scripts/backfill_meeting_cards.py sweeps them:
python scripts/backfill_meeting_cards.py # read-only survey: how many, which hosts, how long
python scripts/backfill_meeting_cards.py --apply # the real sweep, resumable
python scripts/backfill_meeting_cards.py --apply --max-batches 3 # a bounded first runIt is paced to stay out of the transcription workers' way, and that is
what sets its wall clock. 996 of those pages (60%) are on
archive-stream.granicus.com — the same host the workers pull from
continuously and already hit ffmpeg timed out after 120s (source likely slow or rate-limited) against. So the script does two things instead of
walking the queue newest-first:
- Proportional interleaving. The backlog is planned up front and
reordered so each host's pages are spread evenly across the whole
run (host
hwithnofNpages gets itsi-th page at(i + 0.5) * N / n). Plain round-robin would drain the small hosts first and leave the tail as one unbroken Granicus run — exactly the pattern this avoids. - Per-host cooldowns —
--host-cooldown(default 10s) and--granicus-cooldown(default 30s). Enforced when a batch is composed, since the endpoint extracts a batch back-to-back with no pacing of its own: at most one page per host per batch, and only once that host's cooldown has elapsed. The pacing key is the registrable domain, so ~100 per-city*.cablecast.tvsubdomains share one lane and the two*.granicus.commedia hosts share one budget.
At 996 Granicus pages and one per 30s that is a ~8h20m run, with the
other ~670 pages fitting inside the same window. Deliberate: a slower
sweep that leaves the workers alone beats a fast one that fights them.
Raise --granicus-cooldown to go gentler still.
The script drives POST /internal/thumbnails/backfill, naming the exact
pages it wants via that endpoint's slugs parameter (which is what makes
client-side ordering possible at all), and is safe to Ctrl-C and re-run.
Two different things make resumption work:
- A page that gets a frame leaves the queue
(
crud.list_pages_missing_default_thumbnail()filters on "has no default thumbnail"), so a restart's survey already excludes finished work. - A page whose extraction fails leaves no trace in the database, so
the script remembers those locally
(
scripts/meeting_card_backfill_state.json, gitignored) and leaves them out of the next plan. That matters for cost, not tidiness:extract_and_store()runsprobe_duration()before it checks its in-process failure cooldown, so re-attempting a dead source costs a freshffprobeevery time. Delete that file to give every stuck page another chance — a CDN timeout is often transient.
Per batch it reports attempted/stored/failed/skipped, which lanes it touched, each failing slug with its media host and ffmpeg's own reason for the miss, and an ETA recomputed from observed throughput. The closing summary groups the stuck pages twice — by media host, and by reason (with one real example per group) — because those answer different questions: the host says which CDN is unhappy, the reason says whether it is a rate limit, a dead link, or an offset past the end of the video. Reasons are bucketed rather than counted raw, since each carries a 300-character tail of ffmpeg stderr including the media URL.
A result marked skipped is kept strictly apart from a failure: it means
the Archive attempted nothing at all (the frame was already in flight,
inside its 6h failure cooldown, or the extraction queue was full), so
those slugs are never written to the local stuck-page file and get
retried on the next run. The raw endpoint is still there for a one-off,
and its non-dry-run results carry offset_seconds, reason and
skipped per slug:
# Warm default frames for already-archived pages (dry_run defaults true)
curl -X POST -H "Authorization: Bearer $ARCHIVE_INGEST_TOKEN" \
"$ARCHIVE_BASE_URL/internal/thumbnails/backfill?limit=10&dry_run=false"
# ...or name exact pages (ignores limit/offset, max 50 per call)
curl -X POST -H "Authorization: Bearer $ARCHIVE_INGEST_TOKEN" \
"$ARCHIVE_BASE_URL/internal/thumbnails/backfill?slugs=some-slug&slugs=another&dry_run=false"ffmpeg availability — answered, 2026-08-21: ffmpeg 5.1.9 and
ffprobe are both really present on the Archive service, confirmed by a
live GET /api/health against production. The resolver-side extraction
fallback WO-28 documented in case they weren't is therefore not needed
and was never built. Present is not the same as sufficient, though
(WO-45, 2026-08-23): 5.1.9 cannot read a Cablecast fMP4 VOD playlist
through an input-side seek at any offset, which is why 107 archived
pages still have no card — see BACKLOG.md's [HUMAN] entry. The
services also run different ffmpeg versions and always have: the Archive
is runtime: python (Render's buildpack, 5.1.9) while the transcription
workers are runtime: docker off a now-pinned python:3.12-slim-trixie
(7.1.5). Check media_tools for the service you actually mean. GET /api/health reports media_tools (the real
ffmpeg/ffprobe versions on PATH, or null).
It's informational and never fails the check — Render gates deploys on
this endpoint, and a service that serves every page but can't generate
new thumbnails is healthy; those pages simply carry no og:image, which
is exactly where they were before this feature.
Non-meeting pages get a card too (WO-90, 2026-08-31). Everything above
is about /m/{slug} — a page backed by one specific meeting's own video.
The home page, /meetings (including every search result —
/meetings?q="Darth Vader" unfurls the same as the bare page), /coverage,
/state/{slug}, /state/all-50 and /j/{slug} have no single video of
their own, and until this shipped they had no og:image at all —
confirmed live: a link to any of them unfurled with no thumbnail on
X/Bluesky.
- Home and
/meetings//coverage(no per-page pool of real meetings to draw from) pointog:image/twitter:cardat one fixed, shared frame —templates.env.globals["generic_card_image_url"], set in botharchive/main.pyandapp/main.pynext topublic_base_url— a wide horseshoe-dais shot with the city seal visible, picked because it reads clearly as "a government meeting" at thumbnail size even out of context./meetings' ownog:title/og:descriptiondeliberately stay generic rather than reflectingq—meeting_list.html's<link rel="canonical">already collapses every filtered variant (q,jurisdiction,date_from, ...) to the bare/meetingsURL, so a query-specific tag would describe a URL other than the one it's attached to. /state/{slug},/state/all-50and/j/{slug}already compute afeaturedlist of real meetings from that state/jurisdiction for their moments feed and JSON-LDItemList(_featured_entry()'scard_url, filled in by_attach_thumbnails()from a bulkpages_with_thumbnails()check — the same real, already-extracted frame/m/{slug}itself would advertise). The page-levelog:imagereuses that: the first featured entry with a stored card, so a link to/state/californiaunfurls with an actual California meeting, not an unrelated stock frame. Falls back to the same sharedgeneric_card_image_urlonly when nothing in that page's own featured pool has a card yet (a brand-new state/hub with no warmed thumbnails).
Either way, every og:image on the site is a real archived frame, never
stock art, and the shared fallback needs no new static asset.
Every og:image also carries og:image:alt/twitter:image:alt
(WO-91, 2026-08-31). Before this, no image on the site — the per-meeting
frame on /m/{slug}, its <img> on the search-results/featured-meeting
cards, or any of the WO-90 page-level images above — had alt text
anywhere. The two on-page <img> uses stay alt="" on purpose (the
enclosing link is aria-hidden/tabindex="-1", and the real title +
quote already sit right next to the image as text, so a screen reader
skipping the redundant decorative image is correct, not a gap) — this is
specifically about the meta tags social platforms read for a shared
link, which had nothing at all. Same "real content, not a caption of the
photo" shape as the description text:
/m/{slug}(meeting_page.html):"A photo of a meeting in {jurisdiction} where people are talking about {quote}", built fromhighlight_description— the same real transcript excerpt the meta description already uses, never the generic fallback sentence, since a templated page description isn't something "people are talking about." Degrades to jurisdiction-only or a plain sentence when either fact is missing./state/{slug},/state/all-50,/j/{slug}: the same shape, built from whichever featured entry supplied the dynamic image (its ownjurisdiction_display/snippet_text) — truncated to 200 chars first, sincesnippet_textalone can run up tohighlights.MAX_CHARS(420) and the prefix would then risk pushing the combined alt text overtwitter:image:alt's 420-char limit. Falls back to the fixed generic description below when the image itself did.- The shared generic image (home,
/meetings,/coverage, and any state/hub page with no card of its own yet): a fixed, honest description of the photo itself —templates.env.globals["generic_card_image_alt"], next togeneric_card_image_url— deliberately not tied to the page's own jurisdiction or quote, since the image doesn't depict either.
dedupe_rollup_cues() (see "Caption format handling" below) shipped
2026-08-21 and runs on every fresh resolve, but nothing re-checks a
stored TranscriptVersion — so every page ingested before that date
still serves the duplicated roll-up text it was archived with.
scripts/dedupe_rollup_transcripts.py closes that half:
python scripts/dedupe_rollup_transcripts.py # read-only: which pages, with before/after excerpts
python scripts/dedupe_rollup_transcripts.py --slug some-slug # spot-check one page
python scripts/dedupe_rollup_transcripts.py --apply --from-report scripts/rollup_dedupe_report.jsonIt never scans the corpus. No SQL, and no call that aggregates over
segments server-side. GET /internal/pages/all-urls (one query over
meeting_pages only, no join to transcripts) gives the page list; that
list is narrowed client-side to the four platforms WO-34 confirmed serve
roll-up — Granicus, CivicClerk, eScribe, YouTube, which also covers
Legistar/CivicPlus/PrimeGov/Chicago-ELMS pages since MeetingPage.platform
stores the delegated finder's name — and to pages archived before
2026-08-21. Each survivor is then probed one at a time via its own public
GET /m/{slug}/transcript.srt export and scored with the same
_looks_like_rollup() detector the fix gates on.
The dry run against production (2026-08-22) takes ~7 minutes:
2,389 pages → 1,163 candidates → 25 still holding roll-up duplication,
976 clean, 162 with no transcript, 0 probe failures. Those 25 hold 13.8M
stored characters that become 2.5M — 81.9% of the text on them is
duplication. Eleven are Granicus, ten YouTube, two CivicClerk, two eScribe.
Jacksonville FL (jaxcityc.granicus.com, CLAUDE.md's named negative
control) came back clean on all eight of its archived pages.
An earlier run the same day reported 1,377 candidates and 26 findings.
The difference is not drift: crud.list_all_page_urls() only started
returning created_at when that change deployed, so until then the
--created-before bound was inert and 214 pages archived after WO-34
shipped were probed and cleared rather than excluded up front. The
smaller numbers are the accurate ones.
Writes go through the normal ingest path, with a promote after: a
fresh push does not become the page's default when the current default
already has segments and a language (crud._is_real_improvement()), which
is exactly a roll-up page, so ingest alone would file the fixed transcript
where nobody sees it. Four gates sit between a detection and a write —
the stored transcript must flag, the fresh resolve must not flag, the
result must be smaller than the stored one but above --min-retained
(default 0.05, against a real measured minimum of 0.066 across those
pages), and the page's own export is re-read afterwards and must come back
clean. Nothing is deleted: the old version stays reachable at
/m/{slug}?version=<old id>, and the rewritten one is stored with
source="deduped" rather than the ingest default "scraped", so the
version picker can tell them apart — "English (de-duplicated)" against
"English (sourced)", which language and source alone could not do for two
versions of one meeting. That value is a real provenance flag as well as
a label: only "transcribed" means AI-generated, and every other source
gets meeting_page.html's third-party disclaimer through a {% else %}
fallback (see TranscriptVersion.source's own docstring).
The dry run splits findings by the detector's score. WO-34's 18-file
calibration measured real roll-up tracks at ≥ 0.401 and real non-roll-up
ones at ≤ 0.048 with nothing between; at corpus scale that gap is not
empty, and what sits in it is a coherent cluster rather than noise — 10 of
the 25 score 0.202–0.244 and every one is a YouTube auto-caption track
behind a CivicWeb or Municode portal emitting each speaker-change line as
both >> and », retaining ~0.80 of its characters against ~0.07–0.12
for the Granicus ticker shape. Both bands are rewritten by default —
--min-ratio 0.40 restricts a run to the confident one.
Most local governments don't build their own video/meeting-minutes website — they buy one from a handful of vendors, so a fairly small number of platforms cover a huge number of cities and counties. This app supports a platform once, and every city on that platform works.
One AssetFinder per platform, not per city — cities on the same
platform share the same page/API structure. Detection lives in
detect_platform(); adapters are registered in app/main.py.
| Platform | File | How video is found | How captions/agenda are found |
|---|---|---|---|
| Granicus | granicus.py |
Regex-scan the page HTML for .m3u8/.mp4 URLs (shared media_scan.py helper) |
Guessed /videos/{id}/captions.vtt path + scanned .vtt URLs; language verified from actual cue content (not the untrustworthy srclang label); RSS channel title (ViewPublisherRSS.php) used for reliable jurisdiction/title. Agenda items (AgendaViewer.php's chapter markers) are fetched independently of transcript availability into their own agenda_items field, when that customer has Granicus's native agenda index turned on (not universal — some customers redirect it to their own site instead, surfaced as a plain link instead). Date fallback chain: page text (excluding a "previous meeting" reference, a real confirmed false-positive trap) → RSS item date → Granicus's own published Minutes document (MinutesViewer.php, plain HTTP-fetchable, real date at the top) → document-link-filename guess |
| CivicClerk | civicclerk.py |
Public REST API (<subdomain>.api.civicclerk.com) — the portal page itself is a client-rendered SPA with nothing to scrape. A bare portal URL with no /event/<id> (what a government page such as vallejo.gov iframes) returns a calendar_page pick-list of the newest past meetings that have a recording, from GET /v1/Events |
closedCaptionTracks/closedCaptionUrl when populated — real format is SRT, not VTT (confirmed live); language verified from actual cue content, same distrust-the-label approach as Granicus. The API's eventBookmarks (agenda-item timestamps) are fetched independently into agenda_items |
| Swagit | swagit.py |
jwplayer JSON blob embedded in the page (shares Granicus's CDN infra, but a different page shape) | .playerControl[data-ts] agenda-item markers fetched independently into agenda_items |
| eScribe | escribe.py |
<div id="isi_player" data-client_id data-stream_name> when present — video integration varies entirely by city, "no video" is a normal outcome here |
iSiLIVE captions, keyed by language suffix in the filename ({file}.vtt, {file}.fr.vtt, ...). Real per-item agenda timestamps from an embedded video.Bookmarks JS array, when present — not every item gets one, so items without a match are omitted rather than guessed. jurisdiction falls back to the pub-{city}.escribemeetings.com subdomain when the page body has no "City of X" phrase — validated against the same Census/StatsCan tables Granicus uses (jurisdiction_enrich.validated_label_extract()) before being accepted, declining rather than guessing when nothing validates (gated 2026-08-18; previously an ungated wordninja-split guess, see BACKLOG_DONE.md) |
| California Legislature | ca_legislature.py |
Self-hosted (stream.{assembly,senate}.ca.gov), not a vendor platform |
Self-hosted .vtt at a matching filename; genuinely high quality when present |
| Legistar | legistar.py |
Doesn't host video — finds the embedded/redirected link to a platform above (usually Granicus) and delegates via resolve_via_platform(). If its own a.videolink pattern finds nothing, falls back to a broader link scan (base.find_platform_link(), shared with the generic fallback below) before giving up — confirmed live on Baltimore's instance, whose real recording is a plain attachments-table link to YouTube, not the usual pattern. If that finds nothing either, a last-resort city-YouTube-channel match (youtube_channel.py, WO-30) covers the four confirmed instances where the page genuinely has no video to parse — Phoenix, Philadelphia, Baltimore and Albuquerque all publish recordings only to their own YouTube channel, unlinked from Legistar. It enumerates that city's known channel (a curated, human-verified netloc→channel-id dict, never a YouTube search) and matches the meeting's own body name and exact date against the video titles, declining on any ambiguity; a match carries a video_warning naming the channel and video it came from, since the meeting page never linked it. Last resort: a same-tenant Granicus-listing match (granicus_channel.py, added 2026-08-29 for Kansas City, MO, second tenant Yonkers, NY added 2026-09-10 WO-145) covers a different real shape — the recording genuinely exists on Granicus, the same platform the tenant's video already comes from when it's linked at all, just under a meeting id this page doesn't reference (Kansas City) or with no video mechanism wired up for a past meeting at all (Yonkers, whose a.videolink only gets a real link client-side for a live event). Matches the tenant's own public ViewPublisherRSS.php listing by body name and exact date, same decline-on-ambiguity posture as the YouTube-channel case. Curated per-tenant — see that module's docstring for the evidence and WO-145's read-only sweep of ~30 other Legistar tenants testing whether this generalizes |
Whatever the delegated platform provides — for a channel match, YouTube's own captions |
| CivicPlus | civicplus.py |
Same delegation pattern as Legistar, from AgendaCenter listing rows. Detected either by a *.civicplus.com netloc or, since WO-275 (2026-09-12), by the /AgendaCenter path alone — most real tenants are white-labeled onto the government's own domain and never touch civicplus.com, the same reason Hyland's own path-only check below exists |
Whatever the delegated platform provides |
| Destiny AgendaQuick | destinyhosted.py |
Confirmed live 2026-08-21 across 61 real destinyhosted.com tenants (enumerated via Wayback CDX — see BACKLOG_DONE.md) to be a pure agenda/minutes CMS, not a video host — delegates to GenericFallbackAssetFinder's own tiers unchanged (a thin wrapper, not a redundant parser), only claiming the "destinyhosted" platform identity itself when nothing deeper resolves. Registered as its own platform (rather than left as "unknown") specifically so base.find_platform_link() follows a destinyhosted.com link found on some other wrapper's page as a real one-more-hop delegation target — confirmed necessary live: Roswell, NM runs CivicPlus's AgendaCenter self-hosted on its own domain rather than *.civicplus.com and links straight to a destinyhosted.com backend. (Until WO-275, 2026-09-12, detect_platform()'s civicplus check only fired on a *.civicplus.com netloc, so a self-hosted AgendaCenter URL reaching it directly — not via civicplus.py's own delegation — returned "unknown" rather than "civicplus"; it now also matches on the /AgendaCenter path shape, the same way the Hyland row below matches on /Meetings/ViewMeeting alone. This delegation from inside a civicplus.py resolve is unaffected either way.) 18/61 sampled tenants have a real, confirmed Swagit video link — via a formal built-in AgendaQuick↔Swagit integration, onclick="swagitPlay('https://...')" rather than a plain href — some others use YouTube/Granicus/Cablecast-Castus/other vendors, several have none this month (a real per-tenant negative, not a gap) |
Whatever the delegated platform provides |
| PrimeGov | primegov.py |
Doesn't host video itself. Prefers a plain JS variable (var videoUrl = "...") directly in the page HTML when present, delegating to YouTube — but some tenants' real video is actually hosted on Swagit or Granicus instead, with no trace of it in the page HTML at all (confirmed live 2026-08-19, see BACKLOG_DONE.md). When no YouTube id is on the page, falls back to the tenant's own GET /api/v2/PublicPortal/ListArchivedMeetings?year={YYYY} API, matching the page's meetingTemplateId against documentList[].templateId to find that meeting's real videoUrl, then delegates to whichever adapter matches. Every delegation path preserves the original PrimeGov URL as source_url (unlike the Legistar/CivicPlus delegation pattern) |
Whatever the delegated platform (YouTube, Swagit, or Granicus) provides |
| YouTube | youtube.py |
No direct video file URL exists (unlike every platform above) — playback is an embedded iframe + the YouTube IFrame Player API, not the native <video>/hls.js pathway. Handles a direct youtube.com/youtu.be URL too, not just PrimeGov delegation |
yt-dlp (plain HTTP requests to YouTube's caption endpoints are blocked — see BACKLOG.md); prefers a manual/CC track over auto-generated only when its coverage is comparable, since a manual track can start well into the video and skip pre-meeting dead air |
| Viebit | viebit.py |
Reached via Legistar delegation (confirmed so far only under NYC Council's instance) — a var pageConfig = {...} JS object embedded in plain HTML gives a real HLS master.m3u8 URL, no JS execution needed for discovery. Playback is an iframe embed (/embed/vod?v={id}&t={seconds}), not the native <video>/hls.js pathway every other platform above uses — confirmed live 2026-08-12 that the raw master.m3u8 403s from a CDN-level Referer/Origin check, and that no postMessage-reachable seek API exists in Viebit's own player bundle (lgx-videojs-plugins-*.js/vod-embedded-*.js, pulled and read directly). t= is read by the iframe only at load time, so "seeking" after playback has started means reloading the iframe with a new t= — wireSharedControls(adapter, { liveTracking: false }) degrades live-position-dependent UI (playhead tracking, "currently playing" highlight, play/pause control) honestly instead of faking it, a deliberate, user-confirmed tradeoff |
Real, populated VTT captions from the same pageConfig; two-line rolling-caption shape, collapsed by dedupe_rollup_cues() (see "Caption format handling" below — that function was rewritten in WO-34 to handle four real roll-up shapes; this track's output is byte-for-byte unchanged by that rewrite apart from one segment it now de-duplicates correctly) |
| Minneapolis LIMS | lims.py |
The one adapter that isn't plain aiohttp — both the agenda page and its /MeetingYoutubeVideo/{id} JSON endpoint return a genuine Cloudflare JS challenge to a normal HTTP request, so this uses headless_browser.py's real (headless) Chromium fetch instead. Delegates to YouTube for the video itself |
Whatever YouTube provides, via delegation. Real per-agenda-item timestamps from the JSON endpoint's SerializedVideoTimestamps tree — genuinely richer than most platforms above, since most don't have real per-item start times at all |
| Salt Lake City meeting recaps | slc.py |
Also Cloudflare-gated (same headless_browser.py fetch as LIMS above) — scoped to slc.gov/council/*-meeting-recap/ pages specifically. Not multiple distinct videos per page (confirmed live across four real pages, see BACKLOG_DONE.md) — one video, several manually-curated t= timestamp links into it, turned into agenda_items the same way LIMS's structured data is |
Whatever YouTube provides, via delegation |
| Aurora, CO (auroratv.org) | aurora.py |
Confirmed live 2026-08-12. Parses the page's own embedded Drupal <script data-drupal-selector="drupal-settings-json"> blob for jw_data's real mp4_url — already-supported video_format="mp4", no frontend changes needed. A real early mistake worth flagging: the blob's top-level video_caption field looks like the caption URL but is actually a server filesystem path (/home/atowntv/public_html/...); only found by resolving and seeing 0 segments despite the real file curling fine directly, not by reading the schema |
jw_data.caption_file_path, a real fetchable URL unlike the sibling field above |
| CivicWeb (iCompass/Diligent) | civicweb.py |
Also covers Diligent's newer portal hosts (*.diligent.community, *.community.diligentoneplatform.com); on 10 hand-approved 2026-09-26 meetings across CivicWeb, Diligent and BoardDocs, every one resolved to a YouTube video. Confirmed live 2026-08-12 to be a YouTube-delegating platform, not a video host of its own — its "Video" tab embeds a plain YouTube iframe. Delegates straight to YouTubeAssetFinder.resolve_video_id() (the PrimeGov pattern: original CivicWeb URL preserved as source_url, platform stays "youtube"). A second, genuinely different real URL shape (found 2026-08-30 via a Wayback Machine CDX search, 108 real tenants confirmed): a direct /document/{id}/ splitscreen-widget link, whose real meeting id lives only in the page's own inline JS config, not the URL — resolved via a second, richer API (/api/geteventwithindexpoints/) rather than the Id=-shape's /api/videolink/ |
Whatever YouTube provides. One real gotcha: /api/videolink/{id} is double-JSON-encoded — a first .json() parse yields a Python str, not the list it looks like; _fetch_json() re-parses if that happens. The /document/{id}/ shape also gets real per-agenda-item deep-link timestamps (agenda_items) — the richer API's LocalIndexPoints field, matched against real <a name="AgendaHeadingN">/<a name="AgendaItemN"> anchors already embedded in the document's own body HTML (see civicweb.py's own API reference for the full field mapping) |
| Cablecast | cablecast.py |
Confirmed live 2026-08-12 on Detroit's Cablecast portal, then confirmed 2026-08-13 that Charlotte, NC's real /internetchannel/show/{id} pages use the exact same Remix.js SSR template (all data, including a ~35-item "related shows" carousel, embedded in one window.__remixContext JSON blob) — one adapter, two confirmed real customers at build time. Grown well past that since: the /coverage page's own live jurisdiction data showed 17 real Cablecast tenants as of 2026-08-26 (charlotte/concordca/coralvision/glendoraca/hctv/leonvalleytx/cityofpontiac/rialtoca/riverviewmi/salem/cityofsantee/city-slp-mn/virginiabeach/wilmington-nc/winchendon.cablecast.tv among them, spot-checked live the same day), split across both the old /internetchannel/show/{id} path and the newer bare /show/{id} path this adapter already handles (_SHOW_ID_RE/_SHOW_ID_SHORT_RE) — still deliberately not a general *.cablecast.tv rule, since an unconfirmed customer could still use a different template, but the two-template split has held across a much bigger real sample than the original two. Jurisdiction is derived per-customer (see _extract_jurisdiction()), not hardcoded. Real quirk: Detroit's portal HTTPS hangs indefinitely for the whole domain (15s+ confirmed via direct curl) — resolve() always fetches over plain HTTP regardless of the scheme pasted, matching how Detroit's own city site (detroitmi.gov) actually links it. The real video is a direct, unauthenticated .m3u8 on a separate reflect-detroit-vod.cablecast.tv subdomain (HTTPS works fine there), already fully supported by the existing hls.js pathway |
vodTranscripts was empty on every one of 36 Detroit shows checked, but a real, populated, fetchable entry was found on a Charlotte show (show/2451) — a plain-text transcript file, parsed directly (_parse_transcript()), not through the shared VTT dispatch |
| CHAMP/ChampDS | champds.py |
Confirmed live 2026-08-13 against 6 independent real customers via a plain, unauthenticated JSON API (playapi.champds.com/{customer}/event/{id}). Only the direct-MP4 MediaInfo.DownloadURL case is wired up to play (2 of 6 customers). The majority-case VOD2 HLS stream answers only requests with Referer: https://play.champds.com/, which a reader's browser can't send, so it never goes in video_url. Since WO-1045 it is returned as server_media_url instead: the transcription paths read it (our server can send that Referer), and, since A9 (2026-10-17), the reader gets ChampDS's own event page in an iframe as the player (video_format="champds"): it sends ChampDS's Referer itself, and transcript clicks reload it at /event/{id}/s/{seconds}. The reader presses play inside the frame, and the transcript highlight does not follow the video. Since WO-1052 the stream is set even when a DownloadURL exists, and our own server-side reads (transcription, the queue probe) prefer it: many of these MP4s keep their index at the end of the file, which ChampDS's server takes minutes to reach |
Real VTT captions from MediaInfo.Captions, fetched from play.champds.com/CAPTION/{customer}{MediaPath} (WO-1045; confirmed on El Paso County CO and Atlanta GA). Plus Agenda.Attachments as agenda_link, preferring a .pdf-shaped one |
| IQM2 | iqm2.py |
Confirmed live 2026-08-13 against Atlanta, GA — doesn't host video itself; a past meeting's real "Video" link carries a plain static onclick="OpenWindow('/Citizens/SplitView.aspx?Mode=Video&MeetingID={id}...')" (a future/no-recording meeting's link stays a bare unpopulated placeholder, the signal used to tell the two apart), and that page's raw static HTML carries a literal <!-- MEDIA URL: ... --> HTML comment with a direct Granicus HLS URL — no JS execution needed, a plain curl sees it, though the stream URL itself needs a real (non-default) User-Agent. Santa Clara County, CA is a second real confirmed customer — smaller commissions/committees checked there often have no video link populated, but its flagship Board of Supervisors meetings do (confirmed live 2026-08-14), so this is a real body-type-dependent gap on that instance, not a limitation of this adapter |
Real per-item timestamps: the same per-meeting page, requested with Target=Detail&CssClass=AgendaOutline&Mode=Video&Frame=Nothing, renders every agenda item as a real SetPosition({seconds}) onclick alongside the actual item text (procedural entries and full ordinance/resolution text alike) |
| ClerkBase (clerkshq.com) | clerkbase.py |
Confirmed live 2026-08-14 against one real customer (Yellow Springs, OH) — doesn't host video itself. The landing page's static HTML (no JS execution needed) embeds the real agenda-document URL and title as plain JS variables (window.autoOpenDocUrl/autoOpenDocTitle); that document (a raw MS Word HTML export, also directly linkable on its own) embeds the video as an opengovideo.com wrapper link straight to a YouTube embed — delegates to YouTubeAssetFinder, platform stays "youtube" (same PrimeGov/CivicWeb pattern). Jurisdiction comes from the URL's own {Name}-{ST} client-site slug (a ClerkBase product convention), not page text |
Whatever YouTube provides. Grown to 6 real, live, pushed customers as of 2026-08-26 (Cumberland/East Providence/Pawtucket/Scituate/West Warwick, RI, plus South Kingstown School Committee, RI) — the landing-page/document-page shapes have held across all of them, see BACKLOG_DONE.md |
| TelVue | telvue.py |
Confirmed live 2026-08-16 against a real Ashland, OR Planning Commission meeting on videoplayer.telvue.com (also reachable via a peg.tv shortlink, a plain HTTP redirect to the same page — no separate platform). Everything needed is a plain JSON Player.setupData['playlist'] array embedded in the static HTML, no JS execution needed — a real file: HLS URL plus a tracks: list |
Real captions confirmed present and high-quality on the one sample checked — WebVTT with <v Speaker N> voice tags. parse_vtt() strips them from the text and keeps each label as the segment's speaker (WO-1967). A separate chapters.vtt track gives real start/end agenda-item ranges |
| Seattle Channel (seattlechannel.org) | seattlechannel.py |
Confirmed live 2026-08-14 against two independent real meetings, scoped narrowly to the /videos?videoid={id} URL shape (the older feed-style index page and a bare /videos with no id are deliberately left to generic_fallback.py). The primary video's JW Player instance is always the fixed element id vidPlayer — HTML is sliced to that specific block, bounded by the following .on('complete', ...) call, so an unrelated "related video" embedded further down the same page is never picked up by mistake |
Real SRT captions, plus real per-item data-seek timestamps from <a class="seekItem"> elements, turned into agenda_items |
| Hyland "OnBase Agenda Online" | hyland.py |
Confirmed live 2026-08-16 across 26 real customer domains, spanning two distinct real UI versions of the same vendor product. Version A's separate /Meetings/ViewMeetingAgenda endpoint and Version B's /Documents/ViewAgenda endpoint (a converted-Word-document render) are both plain server-rendered HTML, no JS execution needed — resolve() tries Version A first and falls back to Version B only when that yields nothing. Falls back to YouTubeAssetFinder delegation when no direct JW Player media file is found (one real customer's page uses a YouTube embed instead) |
Real timestamped agenda_items on customers with video: each version's own agenda outline embeds a loadAgendaItem({id}) link per item, joined against the main page's inline itemEventPoints video-seek-offset map on that same id. No caption/transcript track of any kind has been found on any JW-Player-backed customer — only the YouTube-delegated one has a real transcript |
| Town Hall Streams (townhallstreams.com) | townhallstreams.py |
Confirmed live 2026-08-20 against 7 real towns — a plain jwplayer(...).setup({file: "..."}) call in static HTML embeds a directly-fetchable HLS URL (no Referer/Origin gating, confirmed via direct curl), no JS execution needed. The page itself carries no jurisdiction/title/date text at all — everything comes from the video URL's own path, mp4:{town_slug}/{date}_{numeric_id}_{Meeting_Title}.mp4. Jurisdiction is routed through the shared, Census-validated jurisdiction_enrich.validated_label_extract() (never a by-eye slug decode — 2 of 7 real slugs carry zero state information at all, see BACKLOG_DONE.md), with a wordninja-split check of the slug's own trailing token for a real state abbreviation, falling back to a Census unambiguous-name lookup |
A real get_transcriptions AJAX endpoint exists but returned empty on all 7 real meetings checked — treated as best-effort/unconfirmed per this repo's convention: a genuinely non-empty response is surfaced as a transcript_warnings entry (not silently dropped, and not guess-parsed) since no real positive example has been found yet to build an actual parser against |
12milesout.com ({tenant}.12milesout.com) |
twelvemilesout.py |
Confirmed live 2026-09-24 (WO-1047) across 6 real San Diego/Inland Empire, CA tenants (Escondido, Coronado, Colton, Covina, Big Bear Lake, Solana Beach) — one meeting per {tenant}.12milesout.com/.../{id} page, tenancy per-subdomain. A direct, unauthenticated CloudFront MP4 download link on every meeting page (<a href="...mp4" download>), not the JW Player/HLS smil: URL also present (its CDN host has no live DNS). Real #agenda-list chapter markers (data-start/data-duration) present on every tenant regardless of captions. Jurisdiction from a small curated tenant map (subdomains don't self-describe) |
A hidden _su field (base64, pipe-delimited: server URL/file id/aspect ratio/duration/video id/thumbnail/permalink) plus the <video> tag's own data-captions="{codes}" attribute (comma-separated ISO 639-1 codes, e.g. "en,es") drive a plain, unauthenticated GET to /api/captions/{fileId}/{code}/0/{durationMs} — confirmed live returning real WebVTT (1,874 cues, Coronado). Empty/absent data-captions (confirmed on Covina/Escondido) means no captions, not an error |
spectrumstream.com (spectrumstream.com/streaming/{tenant}/) |
spectrumstream.py |
Confirmed live 2026-09-24 (WO-1047) across 5 real LA-area CA tenants (Alhambra, Arcadia, South Pasadena [+committees], Glendale USD, Burbank-Glendale-Pasadena Airport Authority) — tenancy is PATH-based on one DNS-wildcarded domain, unlike 12milesout.com above. Every tenant shares one legacy ColdFusion template: a jwplayer("myElement").setup({file: "https://spectrum_streaming.s3.amazonaws.com/{tenant}/..."}) inline script (a direct S3 mp4) and a <a onclick="jwplayer().seek({s})"> agenda list. A tenant's bare root is a truncated fragment carrying only the newest meeting's filename (no real listing) — the listing walker bootstraps from it, or from a known-real seed page when even that's absent (South Pasadena's root was replaced by a live-only Castr embed, its own archive pages unaffected) |
When present, the same jwplayer config's tracks[].file is a real .vtt URL (confirmed live, Alhambra: real English captions). Two real "no captions" shapes, both non-errors: tracks present with an empty file string (Glendale USD), and no tracks key at all (South Pasadena, BGPAA) |
open.media ({tenant}.open.media) |
openmedia.py |
Confirmed live 2026-08-21 across 7 real tenants (Goodyear AZ, Eugene OR, Cortez CO, Santa Barbara CA, Surprise AZ, Georgetown CO, Pitkin County CO) — a YouTube-delegating platform, doesn't host video itself. Requires a modern desktop Chrome User-Agent (a bare/default UA 403s — same class of bot-check gap as generic_fallback.py's own cityofsebastopol.gov fix). The visible player iframe is injected client-side and invisible to a plain fetch, but every tenant checked also carries a real <meta property="og:video"> tag in the raw, un-rendered HTML pointing at the same video (youtube.com/v/{id} or youtube.com/live/{id}) — already a shape YouTubeAssetFinder's own regex recognizes, so no headless-browser fetch is needed. Delegates to YouTubeAssetFinder.resolve_video_id(), original open.media URL preserved as source_url (the PrimeGov/CivicWeb pattern) |
Whatever YouTube provides. Title comes from the page's own og:title (plain meeting title, no date suffix, independent of whether yt-dlp itself is blocked) rather than YouTube's metadata. Jurisdiction comes from the pre-pipe half of <title> ("{Jurisdiction} | {Meeting title}" — the reverse order from generic_fallback.py's own CRRMA-shaped title parsing, so that shared helper isn't reused here), falling back to the tenant subdomain itself (run through the same Census-validated jurisdiction_enrich.validated_subdomain_extract() eScribe/Granicus use) when the tenant has never customized its <title> away from the vendor's own default — confirmed real and live on Cortez, CO. Agenda comes from a <iframe id="document"> present on every tenant checked — either a direct link to another already-registered platform (Goodyear links straight to a destinyhosted.com AgendaQuick page) or this tenant's own pdf.js viewer wrapping a direct S3-hosted PDF as its ?file= query param (Eugene/Cortez/Santa Barbara/Surprise), unwrapped to the raw PDF URL |
| Castus (cloud.castus.tv) | castus.py |
Confirmed live 2026-08-21 (WO-19) against one real customer (Billings, MT's "comm7tv" channel). The static page is a pure JS-redirect shell into a client-rendered React SPA — no headless browser needed, though: the SPA's own webpack bundles were fetched once and read directly, surfacing a plain, unauthenticated POST .../upload/info {"file": videoId} API that returns everything needed for one video (title/duration/readiness flags/an embedded agenda array) from a global endpoint shared by every tenant, plus a global CloudFront CDN (outputs/{id}/Default/HLS/out.m3u8) for the HLS video itself. Also solves the tenant-slug → internal-channel-id mapping this platform's first investigation pass had flagged as unsolved (see castus.py's own module docstring for the endpoint, unused by this adapter but documented for a future per-tenant feature) |
Real, populated AWS-Transcribe-style VTT (captions/{videoId}.vtt on the same CDN) with per-word confidence/speaker inline tags — parse_vtt()'s existing generic tag-stripping already handles this shape with no Castus-specific code. Real per-item agenda_items come from /upload/info's own embedded agenda array (a separate api.castus.tv/ccs/v1/agenda/{id} endpoint returns the same data independently but isn't called, to save a request). Jurisdiction is cross-checked by fetching whichever destinyhosted.com page an agenda item's own hyperlinks point at (Destiny Software's AgendaQuick — real "City of Billings" text confirmed live), with a small known-tenant map filling in the state where the page's own ZIP isn't a real Census-covered ZCTA; falls back to a best-effort tenant-slug parse otherwise (unconfirmed against any real second example) |
| SuiteOne Media (suiteonemedia.com) | suiteone.py |
Confirmed live 2026-08-21 across 6 real tenants (pacificgroveca, lorainoh from a prior investigation, plus tuscaloosaal, camaswa, holladayut, stmarysga newly confirmed live this session; 5 other CDX-derived leads — mcallentx, southbendin, prescottaz, richlandwa, laytonut — 404 and aren't registered). A plain, static-HTML JW Player setup embeds a direct, unauthenticated S3 mp4 (var src = '...' — genuinely empty when a meeting has no recording yet, confirmed on a real St Marys, GA event, not a parse failure). Jurisdiction reuses jurisdiction_enrich.validated_label_extract_with_state() directly plus the same wordninja-last-token state check townhallstreams.py established. The 2 tenants that originally couldn't resolve at all (stmarysga, camaswa — wordninja mis-splits their trailing state letters into a non-word chunk) were fixed 2026-08-21 in the shared module itself, not here: its new tier-5 strip removes a trailing state/province code from the raw label before wordninja sees it, and this adapter asks that module for the code it stripped rather than re-deriving it (the raw trailing letters aren't trustworthy on their own — "tacoma" ends in a real state code too). Both now resolve correctly ("St Marys, GA", "Camas, WA") — see suiteone.py's own module docstring and BACKLOG_DONE.md |
Real, populated WebVTT at a JW Player tracks[].file URL (/Event/GetCaptions/?eventId={id}, no file extension — parsed directly as VTT rather than through the usual extension-sniffed dispatch) — confirmed on 2 of the 6 tenants (Pacific Grove CA, Holladay UT); omitted entirely (not an empty array) when a meeting has none. A separate /event/GetAgendaFile/Agenda?aid={N} PDF embed, confirmed on 3 of 6 sampled events, is surfaced as agenda_link |
| Vimeo | vimeo.py |
Not a civic-video vendor — a general-purpose host that a real, confirmed set of small local governments use directly, built 2026-08-21 (WO-29) against 8 named jurisdictions. Playback is an iframe + Vimeo's own Player SDK, not the native <video>/hls.js pathway: no direct media file URL exists (see captions column). Unlike Viebit's iframe, this one gets the full wireSharedControls() treatment with live playhead tracking, because Vimeo's postMessage API was verified live against a real showcase embed before any code was written (ready/getDuration/setCurrentTime/getCurrentTime/play/timeupdate all real) rather than assumed. Metadata comes from Vimeo's public, unauthenticated oEmbed endpoint (vimeo.com/api/oembed.json), which a plain aiohttp GET can reach; it rejects showcase URLs, so every shape is normalized to vimeo.com/{id} first. Six real URL shapes handled, including the privacy-hashed vimeo.com/{id}/{hash} (the hash must ride along on the embed URL or the player refuses). detect_platform() deliberately does not claim the bare domain — a city's vimeo.com/cityname footer link is a real false-positive class — only shapes with a real video id plus the two listing shapes |
No captions, and that's a wall, not an omission. Real populated English WebVTT does exist on some meetings (confirmed via a real browser on Salisbury NC) but only behind a signed captions.vimeo.com/...?expires=&sig= URL that appears solely inside player.vimeo.com/video/{id}/config — which returns 403 to every non-browser client. The same signed response holds the only real media file, so on-demand Whisper audio is blocked by the identical wall, not a second one (see BACKLOG.md). A transcript_warnings line says so and points at the player's own CC button. /showcase/{id} and /channels/{name} listing pages return a real calendar_page pick-list instead, parsed from a server-rendered JSON-LD ItemList (a bare vimeo.com/{username} page genuinely is client-rendered and is left to generic_fallback.py; Meeting Finder still lists the account behind a Vimeo video through Vimeo's public v2 API instead — vimeo.list_owner_videos(), 2026-10-02, see docs/MEETING_FINDER.md's List section). Jurisdiction comes from the oEmbed account name run through the shared Census-validated check, which usually and correctly declines |
| Zoom cloud recording | zoom.py |
A government's shared Zoom recording (*.zoom.us/rec/share/...), A7, 2026-10-18. Plain HTTP, four requests: the share page (one-time meetingId, session cookies), play/share-info, the play page (fileId), play/info (topic, date, length, signed MP4, transcript address). No player on our page: the MP4 answers only with Zoom's session cookies, which a reader's browser does not have, so it is server_media_url for our transcription, and media_probe sends that resolve's cookies with it. Only links a government published; Zoom's robots.txt disallows /rec/*, see the tier-3 standing decision in BACKLOG.md. |
Zoom's own WebVTT transcript (transcriptUrl) when the host made one (2 of 8 in the pilot); otherwise tier 3 |
| Chicago City Clerk ELMS | chicago_elms.py |
Confirmed live 2026-08-21 (WO-29) against 5 real meetings spanning 2020–2026. A single-city platform, not a vendor product. The meeting page's raw HTML has zero mention of its video — it's injected client-side — so everything comes from the public, unauthenticated api.chicityclerkelms.chicago.gov/meeting-agenda/{meetingId} JSON API (meetingId is a GUID off the portal URL). videoLink is a real Vimeo URL in both showcase styles; delegates to vimeo.py while keeping the ELMS URL as source_url and re-asserting its own platform (the PrimeGov/CivicWeb pattern, plus an attribution fix so /coverage shows a real Chicago row) |
Agenda link only. transcriptLink is in the API schema but empty on all 5 real samples; a populated one would be surfaced as a warning, never guess-parsed. agenda.groups[].items[] is rich (473 real items on one sample: matter title, action taken, vote type) but carries no timestamps at all, so agenda_items is deliberately left empty rather than fabricating offsets — Chicago pages have a working agenda PDF link but no clickable agenda items, unlike LIMS/Hyland/IQM2. Tracked as a real gap in BACKLOG.md |
| Tampa, FL City Council transcripts | tampa.py |
Confirmed live 2026-08-30 (WO-73) across 4 real meetings spanning 2022–2026. A single-city platform (apps.tampagov.net's own "CTTV" webapp), not a vendor product. Each transcript detail page (Agenda.aspx?pkey={N}) embeds its own paired YouTube video directly in a sidebar card — no cross-referencing against the 2,611-row paginated listing grid needed (that grid's own pagination is real-postback-only ASP.NET WebForms, confirmed walkable with plain aiohttp POSTs if ever needed, but wasn't). Delegates to YouTubeAssetFinder for playback (the PrimeGov/CivicWeb pattern: original apps.tampagov.net URL kept as source_url, platform reset back to "tampa" afterward, the ChicagoElms variant). A bare listing-page URL (no ?pkey=) returns the 50 most recent meetings as CalendarPageError candidates, same shape as Legistar's Calendar.aspx |
Real, per-utterance HH:MM:SS-timestamped closed captioning parsed directly from the transcript page itself — genuinely higher quality than YouTube's own auto-captions for the same video (confirmed live: YouTube's auto-caption track for one sample was almost entirely [Music]/filler noise, Tampa's own transcript had clean real dialogue with speaker names), so segments always prefers this page's own transcript over whatever YouTube's delegated resolve found. Two real timestamp-id shapes handled (24-hour t090416, 12-hour t50414PM) |
CivicLive ({tenant}.hosted[2].civiclive.com) |
civiclive.py |
Confirmed live 2026-09-01 (WO-92) — a real, 1000+-customer municipal CMS (Intrafinity, formerly West Corp) with no video product of its own, same delegation shape as CivicPlus/Destiny AgendaQuick. Both hosted/hosted2 subdomains are genuinely Cloudflare-wildcarded at DNS/TLS (unlike civicplus.com/primegov.com), so tenant enumeration goes through a Wayback CDX domain-wide scan rather than subdomain guessing — see civiclive.py's own module docstring. Real per-meeting agenda/calendar content turned out to be client-rendered on every tenant checked (Auburn WA, Escalon CA) and invisible to a plain fetch, so this doesn't attempt CivicPlus-style row scraping; instead it (1) fetches with redirects and re-checks the final URL's host — Auburn's own "Agendas & Minutes" page is a genuine plain-HTTP 302 straight to auburnwa.portal.civicclerk.com, delegated via resolve_via_platform() — and (2) otherwise delegates wholesale to GenericFallbackAssetFinder (the destinyhosted.py thin-wrapper pattern), which already excludes a bare YouTube channel/handle link (confirmed real and common — Auburn/Escalon both only publish a channel-level link, never a per-meeting one) from being mistaken for a specific video |
Whatever the delegated platform provides |
Utah Public Notice Website (utah.gov/pmn) |
utah_pmn.py |
Confirmed live 2026-09-08/09 via a full statewide enumeration pilot (scripts/pmn_utah_pilot.py, ENUMERATION_METHODS.md §105) — Utah's statutory Open and Public Meetings Act notice board, where every Utah government type (state agencies, counties, municipalities, special districts, school districts, and more) is legally required to post meeting notices with self-reported Government Type/Entity/Public Body fields. Two real media shapes: a notice's "Audio File Location" field most often links OFF utah.gov (YouTube/Vimeo/CivicClerk/etc) — delegates via resolve_via_platform(), keeping the PMN notice page as source_url and overriding jurisdiction/meeting_body/title/date with PMN's own ground-truth fields rather than trusting the delegate's own guess; or the notice's own "Download Attachments" table has a bare .m4a/.mp3/.mp4 file hosted directly on utah.gov with no platform wrapper at all — a shape no other adapter here has needed before, resolved by setting video_url directly to the file with video_format inferred from its extension, no new transcription-pipeline work needed |
Never resolved directly by this adapter — either whatever the delegated platform provides, or (the bare-file case) real segments only after the on-demand Whisper pipeline transcribes it, same as any other audio-only source (see media_probe.py's probe_has_video_stream()) |
Wistia ({account}.wistia.com) |
wistia.py |
Confirmed live 2026-09-10 (WO-161) against RegionalWebTV/Advanced Media Solutions of Virginia's shared amsva.wistia.com account — real, independent Virginia governments Warrenton (town), Manassas (city), Fredericksburg (city), and Stafford County. A general-purpose video host (Vimeo's shape), not a civic-video vendor — detect_platform() only claims a real /medias/{id} or /channel/{id} URL on a per-account subdomain, never the bare wistia.com marketing site. A direct media/channel URL resolves straight from Wistia's own public, unauthenticated JSON API (fast.wistia.com/embed/medias|channel/{id}.json); a delegating government page (any URL, not just RegionalWebTV's) is scanned for four real embed shapes (fast.wistia.com/embed/medias/{id}.jsonp, fast.wistia.net/embed/iframe/{id}, class="wistia_embed|wistia_channel wistia_async_{id}", a plain <a href="{account}.wistia.com/medias/{id}">), keeping the government page as source_url. RegionalWebTV's own current-year pages embed the whole channel via a Wix custom-HTML widget invisible to a plain fetch — a headless-browser render (same Playwright path Vimeo's captions use) finds the real nested custom-HTML document, then a plain fetch reads it, no further JS needed. A dead/pruned id answers HTTP 200 with {"error": true} (confirmed on every 2019/2021 id checked from RegionalWebTV's own archive pages) — raised as a real not-found, never a silent empty page; the sweep target is this year's meetings, not the historical archive. A channel URL/embed returns a real calendar_page pick-list, same shape as Vimeo's showcase/channel listings |
Real, unauthenticated WebVTT with real timestamps (fast.wistia.com/embed/captions/{id}.vtt?language=eng) — unlike Vimeo, no headless browser or signed URL needed for captions at all. video_url is a real, public, byte-range-capable MP4 delivery URL straight from the media JSON (also unlike Vimeo, whose real media file is walled off) |
BoxCast (boxcast.tv) |
boxcast.py |
Confirmed live 2026-09-11 (WO-227) against four independent governments — Wilmington OH, St. Louis County (Clayton MO), Hondo TX, Atlantic City NJ. A general-purpose government video vendor, reached via a real, public, unauthenticated REST API (rest.boxcast.com), not page scraping. Three real URL shapes: boxcast.tv/channel/{id} (a government's whole channel — auto-picks the newest past broadcast whose title looks like a real meeting, via a shared phrase-list hand-check plus a governing-body keyword gate, since a channel's own recent broadcasts often include ceremonies/press-conferences/ribbon-cuttings mixed in with real meetings) and boxcast.tv/view/{slug}/boxcast.tv/view-embed/{slug} (one specific broadcast, resolved directly with no filtering). video_url is a real, signed, expiring HLS master playlist fetched fresh at resolve time (re-resolved fresh by the worker before transcribing, same as every other signed-URL platform here). The Archive never trusts that stored URL past ingest, either (WO-229, 2026-09-11): a BoxCast page's own stored video_url expires a couple of days after ingest, so the player is pointed at /m/{slug}/video, which resolves a fresh signed playlist at view time (cached briefly) and 302s there, falling back to the stored URL if BoxCast can't be reached — see archive/utils/video_refresh.py. Jurisdiction and the stable per-government external_id (used for tenant_overrides.csv pins, since boxcast.tv is a shared multi-government host) both come from the broadcast's own BoxCast account, never guessed from a title |
Real, server-fetchable WebVTT captions — on some tenants. A real, paid, per-account BoxCast add-on (confirmed: Atlantic City and St. Louis County have it, Wilmington and Hondo don't) delivered as a genuine #EXT-X-MEDIA:TYPE=SUBTITLES HLS track, itself split into hundreds of ~12-second signed WebVTT segments (533 for one real 107-minute meeting) fetched with bounded concurrency and concatenated — each segment's own cue timestamps are already correct absolute positions in the recording, confirmed live rather than assumed. No caption track at all is a normal, honest outcome on the other tenants |
BoardDocs (go.boarddocs.com) |
boarddocs.py |
Built WO-365, 2026-09-14, from the mechanism spec'd in docs/investigations/boarddocs_video_adapter.md. Mostly a school-board agenda platform (969 of 972 research rows mentioning it are school districts) with no video at all — but where a tenant's own bd.videoservice flag is on (YouTube/Vimeo), a fixed, unauthenticated per-meeting mechanism exposes the real video id: a tenant Public page for the flag and org name, BD-GETMeetingsListForSEO for the meeting list (newest first, includes future meetings), then one POST .../VIDEO-GetAgenda per candidate meeting (its own real Unique id) until one comes back with a video, capped at 40 — same delegation pattern as PrimeGov (YouTubeAssetFinder/VimeoAssetFinder.resolve_video_id() called directly, original BoardDocs meeting URL kept as source_url, platform stays "youtube"/"vimeo"). On-demand only, never a sweep: go.boarddocs.com/robots.txt disallows every agent with Crawl-delay: 1000, so this adapter reads exactly the one tenant/meeting a resolve asks for — no script in this repo may loop it over tenants. Confirmed live on two real tenants (Tallahassee FL's City Commission, 6 of 6 recent meetings have video; Colorado City AZ schools, 5 of 40) and four negative controls (Austin ISD, Northside ISD, St. Charles R-VI MO, Vacaville USD — flag on, zero real video ids on the 40 newest, degrades cleanly to "no video" rather than erroring) |
Whatever the delegated platform provides — no confirmed BoardDocs-native caption track exists (unclaimed, not silently assumed). The VIDEO-GetAgenda fragment's own agenda outline is real and richer than the spec first suggested: each timed <li class="item" data-videohours/minutes/seconds> is a genuine per-item offset into the video, turned into agenda_items (a video-less meeting still returns its category headings, just with no timed item rows at all) |
Sliq Harmony (sg001-harmony.sliq.net/{tenant}/Harmony/...) |
sliq_harmony.py |
Built WO-921, 2026-09-20, against real pages from all seven state-legislature tenants that use it: Arkansas 00284, Colorado 00327, Delaware 00329, Kansas 00287, New Mexico 00293, Oklahoma House 00283, West Virginia Senate 00289 (82 recent events resolved live through the adapter with zero errors; 10 saved real event pages and 1 listing are the test fixtures). One shared host, the government is a five-digit tenant path segment, so identity is pinned per tenant path (tenant_overrides.csv, us:state:NN, source=wo921), never by host; the chamber or committee is meeting_body (D1). The event page is not JavaScript-only: the server writes a dataModel block into the HTML, so ONE plain GET returns the HLS playlist.m3u8 (open CORS, no cookie, plays in hls.js), the closed captions, a timed agenda and the meeting times. A listing URL (.../View/, .../View/RecentEnded/) raises a pick-list (CalendarPageError). No video (not yet held, or a stream flagged Enabled: false with a novod.com or empty placeholder) resolves as a real meeting with a plain video warning |
Real, timed captions where the legislature publishes them: ccItems carries absolute wall-clock Begin/End, turned into offsets from mediaStartTime. Coverage is uneven and reported as it is: Colorado, Oklahoma, West Virginia, Arkansas, Delaware and New Mexico publish captions on most meetings; Kansas published none on any of the 12 recent meetings read (video-only, so its pages go to the tier-3 transcription queue); every tenant has some captionless meetings. The agenda (AgendaTree, nested, timed) becomes agenda_items; WV/AR Speakers turns are left out on purpose |
Invintus (player.invintus.com) |
invintus.py |
Public Event/getDetailed API: the direct MP4 download link, else the HLS stream (streamingURIs.main) |
Two ways. (1) A WebVTT caption file in captionPath. (2) Since WO-1065 (2026-09-25): CEA-608 captions carried inside the HLS video itself, with no caption file. When captionPath is empty, resolve() reads up to 3 six-second pieces spread through the smallest (160p) stream and runs ffmpeg's caption reader on each (embedded_captions.py). If real words come out, the page says captions are embedded and not yet text. The cloud worker then reads the whole stream (about 210 MB per hour of meeting) and stores the captions as the page's transcript. Caption markers alone prove nothing: they sit on every frame, even in silence. |
| Direct downloads (any host) | direct_file.py |
A bare video or audio file, recognized by the URL's own extension (.mp4, .mov, .m4v, .webm, .mp3, .m4a, .wav) on any host, plus Google Drive single-file links (drive.google.com/file/d/<id> and drive.usercontent.google.com/download?id=<id>), Dropbox share links, Laserfiche WebLink file downloads, and iSiLIVE files and player pages (`video.isilive.ca/download |
play/{client}/{file}, video.isilive.ca/{client}/{file}(.html), cdn1.isilive.ca/vod/definst/mp4:{client}/{file}/playlist.m3u8, WO-1155). Checked last in detect_platform()`, after every vendor platform. Built WO-303 (2026-09-12); audio added WO-317. See "Direct downloads" below for the hosts confirmed in real use. |
| CivicMedia (CivicPlus video) | civicmedia.py |
CivicPlus's own first-party video widget, backed by TikiLive's API. Built WO-341 (2026-09-13) for Hobart, IN. | Its captions, when the tenant has them. |
| ProudCity (WordPress government CMS) | proudcity.py |
A curated set of known ProudCity domains (tenants are white-labelled, so there is no shared host to detect). Video is a plain YouTube embed; the adapter adds agenda bookmarks (data-youtube-seek), a reliable jurisdiction from og:site_name, and the agenda PDFs. Built 2026-08-26. |
YouTube captions; data-youtube-seek agenda items. |
Municode Meetings ({tenant}.municodemeetings.com) |
municode_meetings.py |
An agenda CMS, not a video host: follows the meeting row's video link (absolute to a platform, or one hop to a detail page's iframe#mcc_agenda_video) and delegates. Confirmed 2026-09-01. |
Whatever the delegated platform provides. |
Arizona Legislature (azleg.gov/videoplayer) |
az_legislature.py |
A wrapper page around Invintus: extracts the clientID/eventID from the inline Invintus.launch(...) call and delegates to invintus.py. Found 2026-09-09. |
From Invintus. |
TVW (tvw.org, Washington State) |
tvw.py |
A wrapper page around Invintus: reads the eventID/clientID <meta> tags and delegates to invintus.py. Confirmed 2026-09-22 (WO-1010). |
From Invintus. |
CVTV (cvtv.org, Clark/Vancouver WA) |
cvtv.py |
A wrapper page around Invintus (WordPress plugin): takes the eventID from the end of the link (or the page's data-eventid), uses CVTV's one clientID (2917038973), and delegates to invintus.py. Confirmed 2026-10-11 (WO-1189). |
From Invintus (none on the one meeting checked). |
AV Capture All (media.avcaptureall.cloud) |
avcaptureall.py |
Office-Mac only, like Vimeo; untested from Render. One viewer site for many small governments (a meeting is /meeting/{guid}, the address does not name the customer). The meeting record comes from public-api.avcaptureall.cloud, which refuses a plain client and wants an AWS WAF token that only a real browser page can mint, so the adapter loads the meeting page in the headless browser and reads the reply the page asks for (headless_browser.fetch_api_reply_via_browser()); it never builds a header or token itself. With no headless browser it raises HeadlessBrowserUnavailable, the way Vimeo does. The video is a direct .mp4 (or .mp3) on download.avcaptureall.cloud (Range works, no login). The local date comes from the file name (the record's own dates are UTC). video_channel = avcapture:{customerGuid}, pinned per customer in tenant_overrides.csv (no pins for court calendars, City of Monroe WA, or customers with no gov_id). Confirmed live 2026-10-11 on 4 meetings (WO-1190) |
WebVTT from the record's videoSubtitlesUrl when present (tier 1; about 1 meeting in 5, 11 of 36 target customers); no key means no captions and the page goes to tier 3. Auto-generated style, verbatim, English. No per-item times, so agenda_items stays empty |
Internet Archive item (archive.org/details/{item_id}) |
archiveorg.py |
Shared host: any government's items. Claims only the item page (also http, www, trailing slash, query or hash); a /download/... media link is not claimed and keeps resolving through direct_file.py. One request to archive.org/metadata/{item_id}, then at most one for the caption file; no YouTube request, no media download. Title as given; date from metadata.date, else a date in the title or file name, else blank (with a note). Video is the .mp4 (h.264 first), else the .mp3, at archive.org/download/{item_id}/{name}; length from the file's length (seconds or HH:MM:SS). Government fields stay blank (creator is the station). youtube-id becomes external_id youtube:{id} (never fetched) with the note "Copy of a YouTube video held by the Internet Archive." archive.org is in MULTI_GOV_HOSTS: callers pass an explicit gov_id, there are no per-item pins. Confirmed live 2026-10-14 on an own-station upload (Foxborough MA) and an ORCA Media (VT) YouTube mirror (WO-1962) |
*.en.vtt, then any .vtt, then .en.asr.srt/.srt (tier 1). A caption file named .asr. (or an item that lists an .asr.srt) gets the note "Captions are archive.org's own speech recognition." No caption file: no segments, "No captions found for this video.", tier 3. About half of the YouTube mirrors carry captions; own-station uploads usually carry none |
| Granicus "Video on Demand" channel matching | granicus_channel.py |
Not a new platform: for a Legistar meeting whose own video link is empty (Kansas City, MO's structural norm, confirmed 2026-08-29), matches it to the right item in the tenant's Granicus VOD RSS listing. | From Granicus. |
direct_file.py handles a meeting recording that is just a file, wherever it lives. It needs no per-host adapter: a URL whose path ends in a video or audio extension is enough, and the file's own Content-Type confirms it. So a new file host usually needs nothing built.
Hosts confirmed carrying real meeting recordings in use (hand-checked Meeting Finder finds, 2026-09-25/26; live HEAD checks 2026-09-27):
| Host | What it is | URL shape |
|---|---|---|
| Google Drive | file sharing | drive.google.com/file/d/<id>/view, drive.usercontent.google.com/download?id=<id> |
| Dropbox | file sharing | dropbox.com/scl/fi/<id>/<name>.mp4 (and older /s/ links) |
| ParentSquare | school website host | files.smartsites.parentsquare.com/<n>/<name>.mp3 / .mp4 |
| Finalsite | school website host | resources.finalsite.net/videos/.../<name>.mp4 |
| Edlio | school website host | <n>.files.edl.io/.../<name>.mp4 / .m4a |
| Thrillshare (Apptegy) | school website host | files-backend.assets.thrillshare.com/documents/asset/uploaded_file/.../<name>.mp3 |
| SchoolCMS / scschoolfiles (S3) | school website host | s3.amazonaws.com/scschoolfiles/<n>/<name>.mp4 |
| Podbean | podcast host | mcdn.podbean.com/mf/web/<id>/<name>.mp3 / .mp4 |
| Utah Public Notice Website | state notice site | utah.gov/pmn/files/<n>.MP3 |
| Laserfiche WebLink | records repository | ElectronicFile.aspx?... and /WebLink/<n>/edoc/<docid>/<name>.mp3 |
| CMS Max, Municipal Websites (CA), Weebly, WordPress | government website builders | the site's own /uploads/ or /wp-content/uploads/ file paths |
| The government's own domain or CDN | anything else | any path ending in a media extension, e.g. Zoom exports (GMT2026..._Recording.mp4) |
Limits worth knowing: a Google Drive or Dropbox folder is not a single meeting and is not handled; a Facebook fbcdn.net file URL is signed and expires, so it cannot be stored; a file host may be public but slow, which the tier-3 probe reports honestly. The hand-check rules for whether a bare file is a meeting at all (length, file name, the page that links it) live in docs/MEETING_FINDER.md, not here.
Every URL detect_platform() doesn't recognize goes to
generic_fallback.py's GenericFallbackAssetFinder, registered under
platform_name = "unknown" — rebuilt 2026-08-14 (see BACKLOG_DONE.md
for the full build, backtested against every real coverage-gap example
in BACKLOG.md) as a diagnostic router: figure out what the page
needs, then hand off to machinery that already exists. Before any of the
tiers below run at all, the fetch itself is checked (WO-166,
2026-09-10): when the URL resolves straight to a direct video/audio file
rather than a page — a Zoom export, a bare .mp4/.mp3 — the response's
Content-Type/Content-Disposition say so before the page-text read
ever happens, and the result is a plain playable video_url with no
HTML parsing attempted at all (this is what fixed unsupported-platform- no-adapter on a real direct-media file, which used to trip the 10MB
page-text size cap instead). Video tiers, in
order: (1–2) an embedded YouTube video in any confirmed shape — a
URL-shaped id (raw or HTML-entity-escaped, youtube-nocookie included),
or a bare videoId = '...' JS assignment gated on the page actually
loading the IFrame Player API (Tarrant County's real shape) — delegated
to YouTubeAssetFinder; (3) a link to any OTHER platform this app fully
supports, delegated to that adapter's own resolve() — either a plain
<a href>/<iframe src> (Austin, TX → Swagit, confirmed live
2026-08-10) or an onclick="someFunc('https://...')" JS-modal link with
no real href at all (base.find_platform_link() checks both — see
destinyhosted.py's table row below for the confirmed real example);
(4) a directly playable media URL
via media_scan.py's shared scanner (which handles query-stringed
m3u8s, HTML-entity-escaped URLs, JW Player file: config keys, and
protocol-relative/relative paths — the Sacramento/Seattle shapes);
(5) when nothing playable exists, a video pointer —
ResolvedMeeting.video_link — "we think the video is here: <link>",
with two confidence tiers of copy: a curated known video host (Vimeo
video/showcase links) gets "we recognize {host} as a regular video
host", a looser video-shaped guess (a "Video"-texted anchor, a non-junk
third-party player iframe) gets "we don't recognize {host}... so
proceed with caution". Note that as of 2026-08-21 (WO-29) a Vimeo link
usually never reaches this tier at all — Vimeo is a fully supported
platform now, so tier 3 above delegates to its real adapter and produces
an actually playable video instead (confirmed live on Sebastopol, CA).
The curated tier survives for the Vimeo shapes detect_platform()
deliberately doesn't claim, and as the pattern for whatever unsupported
video host shows up next. Since WO-933 (2026-09-21), tiers 3 and 4 skip a
link the shared gate (app/utils/video_hand_check.py) can already tell is
page furniture rather than a meeting: a widget's own animation file
(a0.muscache.com), a hero-embed query string (background=1, loop=1),
a link that is not a video at all, or a video in a looping <video> with no
player controls (the homepage banner shape). A skipped link is treated as if
it were never on the page, so the honest "no video found" or pointer outcome
applies instead. Filename words like "welcome" are deliberately NOT used
here, because this adapter serves a page the user chose. Captions come from
their own candidate chain
(<track> elements, plain caption-file <a href>s, JW tracks:
entries, scan results); metadata from a breadth of confirmed-real
shapes (title-tag separators, og:title, h1 assembly, video_date
meta, <time datetime>, heading dates, URL-slug humanization as last
resort) — every extractor only fills still-empty fields. The agenda
link finder is unchanged: a single best-effort <a>, never fabricated
agenda_items.
Blocked fetches escalate to the real headless-browser fetch
(GENERIC_FALLBACK_HEADLESS=1 — enabled in production since
2026-08-14, the same day playwright-on-Render was finally verified
working for real; see render.yaml's env-var comment for the
evidence): a block-family status (Wayne County MI's real Akamai 403),
a small challenge-interstitial body, or an empty-evidence resolve of a
client-rendered shell triggers at most one Chromium retry, whose
rendered HTML re-runs the same diagnosis — end-to-end confirmed in
prod against the previously fully-Akamai-blocked Wayne County page,
which now resolves with real video/title/jurisdiction/date/agenda. Dedicated adapters can also
opt into the same page-analysis tiers as a backstop when their own
extraction found no video (scan_page_for_video_evidence() — eScribe
is wired in; opt-in per adapter only, since a blind second pass on a
page carrying other meetings' videos could attach the wrong one).
Every fetch this adapter makes — the initial page load, a caption URL
found on that page, and the headless-browser escalation above (including
every redirect and sub-resource request the browser itself makes) —
passes through app/utils/url_guard.py's SSRF guard first: an
http/https scheme allowlist, rejection of private/loopback/link-local/
reserved destinations re-checked on every redirect hop (not just the
entry URL), and a response-size cap. /api/resolve itself rejects a
blocked URL immediately, before any adapter — including this one — ever
runs. See that module's docstring for the real gap this closes.
Every result from this adapter (including when it delegates to
YouTubeAssetFinder, whose own platform field stays "youtube") sets
ResolvedMeeting.best_effort = True, which drives a dedicated, openly
tentative UI on the meeting page (app/static/player.js): a full-width
"we're trying our best" banner, plain "we think the video/agenda is
here: <link>" lines instead of a declarative warning box, and a manual
timestamp-entry box in place of the live playhead-tracking reader other
platforms get (deep-link reliability isn't confirmed here, so there's no
adapter-driven "current time" to honestly display). The video line
specifically is only rendered when there's no playable video to show
(renderBestEffortVideoPointer(), WO-43, 2026-08-22): a real
video_url means the embedded player right below it is already the
answer, and that URL is always a machine artifact rather than a page a
human would open — a Vimeo/YouTube embed shell or a raw CDN media file
(all three confirmed live). The link a reader actually wants, the city's
own page, is the "View original source" line above it, which
_try_delegate_to_known_platform() deliberately resets back to the city
page after delegating. The two informative variants stay: the tier-5
video_link pointer, and a plain "[No video found]" when nothing turned
up at all.
Since 2026-08-21 that flag also survives the push to the Archive
(meeting_pages.best_effort), where it does two things: it disqualifies
the page from social auto-posting entirely, and it lists the page in
GET /internal/low-trust-pages for review. It deliberately does not
affect indexing, the sitemap, or hub listings — see BACKLOG.md. Note
the flag, not platform, is the signal to check anywhere this matters:
a fallback resolve that delegates to YouTubeAssetFinder reports
platform = "youtube", and that's the most common real case.
BoardDocs (mostly a school-board agenda platform, but where a
tenant turns its video service on, a fixed unauthenticated mechanism
exposes a per-meeting YouTube/Vimeo id) is built — see the "Supported
platforms" table above and docs/investigations/boarddocs_video_adapter.md
for the full mechanism and house rule.
Caption format handling is centralized in
app/utils/vtt_parser.py's parse_captions_by_extension(), used by
Granicus/CA Legislature/Swagit/CivicClerk/Aurora/Seattle Channel and the
generic fallback instead of each reimplementing its own format detection.
It's also where roll-up ("scrolling ticker") caption reconstruction
runs, via dedupe_rollup_cues() — roll-up captioning is a property of how
a city captions its meetings, not of which platform hosts the video, so
wiring it per-adapter (as it was until WO-34, 2026-08-21) is what left
Granicus/CivicClerk/eScribe serving visibly duplicated transcripts while
YouTube and Viebit were fine. Four real cue shapes are handled, each
confirmed against a live-fetched file and kept as a fixture; a track that
doesn't look roll-up is returned untouched, which is what makes it safe to
run from a shared dispatch. eScribe parses VTT directly rather than
through this dispatch, so it calls dedupe_rollup_cues() itself.
That fix is resolve-time only, so already-archived pages need a
sweep. Nothing re-checks a stored TranscriptVersion on its own, so
every page ingested before 2026-08-21 still serves the text it was stored
with. scripts/dedupe_rollup_transcripts.py finds those pages and
rewrites them — see "Rewriting pre-WO-34 transcripts" above.
VTT and SRT are real, structurally-parsed formats, confirmed
against real samples on multiple platforms. TTML/DFXP/ITT also get a real
structured parser, but — unlike VTT/SRT — that's verified against the
W3C spec only, not any real captured sample (see BACKLOG.md). SBV/SUB/
SMI/SAMI/plain-.txt get a generic best-effort text extraction with no
per-line timing (t= deep-linking to the video never depended on
transcript timing anyway). SCC/STL (binary/encoded broadcast formats) are
detected but link out rather than attempting to display content, since
nothing can be extracted without real codec-level decoding.
Platform coverage research: this app supports a platform once and
every city on it works, so the actual bottleneck to growing coverage is
finding real, live samples of platforms it doesn't handle yet — per
this file's own "never build an adapter from assumption" convention.
A 2026-08-11 survey of the largest US cities/counties by population
(cross-checked against what's already live in the Archive) produced a
filterable results table
of every jurisdiction checked — platform, rough population, and sample
URLs, split into "real coverage gap" vs. "already-supported pattern,
just not yet resolved." It's hosted as a private Claude artifact, not
version-controlled with the rest of this repo, so treat it as a
point-in-time research snapshot rather than a durable source of truth —
the concrete, actionable findings that came out of it (Phoenix's
Legistar-never-has-video pattern — closed 2026-08-21 by WO-30's
city-YouTube-channel fallback, see the Legistar row above, along with
the same pattern on Philadelphia, Baltimore and Albuquerque —
Chicago ELMS's now-unblocked API
samples, new unsupported vendors like Cablecast/IQM2/CivicWeb, etc.)
were folded into BACKLOG.md's "Platform & jurisdiction coverage" and
"Dormant" sections at the same time, which are the durable record to check
before starting adapter work. Four of that survey's items (Cablecast/
Detroit, Aurora, CivicWeb, Viebit) were picked up and actually shipped
2026-08-12 — see the "Supported platforms" table above for what's real
in code today, and the artifact's own "Shipped since this survey"
section (added the same day) for two real corrections the
implementation work turned up in the original research (Cablecast isn't
one uniform template across cities; Detroit's own portal domain hangs
over HTTPS but works over plain HTTP).
What you can actually do once a meeting page has loaded:
- Video player: hls.js for
.m3u8(Safari falls back to native HLS), locked to a 16:9 box so it never collapses to a tiny default size, with a large overlay play button and a warm-up trick (muted play-then-pause onloadedmetadata) that pre-buffers so the user's real first play starts instantly. YouTube videos use an embedded iframe + the YouTube IFrame Player API instead (no direct video file URL exists for YouTube) — transcript click-to-seek, "Copy link to current time", "Go to time", and deep-link-on-load all work identically either way, since both are wrapped behind the same{currentTime, play, pause, addEventListener}adapter shape (createNativeAdapter/createYouTubeAdapter). When a channel has disabled embedding elsewhere,renderYouTubeEmbedFallback()swaps the dead iframe for a "Watch on YouTube" link instead (see "Permanent pages (the Archive)" above for the full mechanism, shared witharchive/static/meeting_page.js). - Playback speed: a chip in the top-right of the video frame opens
0.75x-3x. It is drawn by this app rather than delegated to the player's
own control bar because a native
<video>bar is browser shadow DOM and cannot be extended — which is also why the browsers' own speed settings are buried in an overflow menu or a right-click and nobody finds them. The offered rates come from the adapter, not a fixed list, because the ceilings genuinely differ: a native<video>accepts the whole ladder (pitch preserved, so voices stay intelligible), YouTube caps hard at 2x and ignores any value off its owngetAvailablePlaybackRates()list, Vimeo's SDK documents 0.5-2x, and Viebit has no cross-frame API at all so it renders no control rather than a dead one. The choice is remembered across meetings (localStorage), clamping down when a player can't reach it without overwriting the stored preference. Lives inshared_static/playback_speed.js+.css— one module used by both the resolver and the Archive, so the two surfaces cannot drift. - Agenda: a dedicated section (
renderAgenda()), shown above the Transcript section whenever agenda/chapter-marker data was found, independent of whether a real transcript also exists. Reuses the transcript's timestamp/link-icon/text markup for visual consistency and click-to-seek + copy-link, but doesn't participate in the transcript's "currently playing" highlighting or theline=deep-link param — agenda items are seek-only viat=. - Transcript: click a line to seek + highlight; a chain-link icon per line (visible on hover, or ambiently on the current line while paused) copies a link to that line without disturbing playback.
- Find bar: mirrors browser Ctrl+F over the agenda and the transcript —
highlights every match, "N/M" count, cycles with prev/next or
Enter/Shift+Enter, Esc clears. On a narrow screen it sticks to the top of
the screen, since phones have no easy Ctrl+F. Same code on both meeting
pages (
shared_static/meeting_tools.js). - Auto-scroll is off by default (2026-10-16). The toggle turns on "follow the playhead"; a deep link or "Go to time" scrolls either way.
- Agenda document viewer loads on click only (Archive page,
2026-10-16). It used to frame
agenda_linkon page load, and some links made the browser download the file. Now a "Show the agenda here" button builds the frame; it is hidden on touch screens, where the plain link opens the document instead. - Theater mode and player resize (desktop, both pages): a "Theater
mode" button puts the video full width with the transcript below it, and
a drag handle between the columns sets the player's width (double-click
resets). Both are remembered in
localStorage. - Manual timestamp entry: a "Go to time" box in the toolbar (accepts
H:MM:SS,M:SS, or plain seconds) — works even with no transcript, since deep-linking is the point even when there's nothing to click. - Sticky toolbar: stays reachable at the top of the viewport when scrolling, so auto-scroll never strands you away from the controls.
- Language mismatch handling: if the best available caption track isn't in the target language, it's used anyway but flagged with a warning rather than silently presented as if correct.
- Transcript export: "Text"/"SRT" download buttons above the
transcript. Built client-side from the in-memory
segmentsarray (no server round-trip — this page has no persistence to download from); the Archive's permanent pages get a real server-side equivalent instead (GET /m/{slug}/transcript.{txt,srt}, since that data actually persists there). - Report a problem: a small form (wrong video, bad transcript, wrong
metadata, or other) that POSTs to
/api/report-problem, visible on both a successful resolve and a failed one. Same control exists on the Archive's permanent pages. Reports land in the resolver's DB, viewable via the token-gatedGET /admin/problem-reports.
A map of the codebase, for orientation — two independent apps
(app/, the resolver, and archive/, permanent pages), each with its
own routes, database, and frontend.
app/
main.py FastAPI app: routes, adapter registration,
/api/resolve's cache-check + logging (rate
limited via slowapi), /api/report-problem,
/admin/*, /robots.txt, the /m/*,
/archive-static/*, /meetings, /account/saved,
/coverage, /coverage/detail, /state/*,
/api/jurisdictions, /sitemap.xml, /feed.xml
Archive proxy routes, /context, /context/new,
/context/feed.xml (see "Full Context feed"
above),
/api/newsletter/signup, /unsubscribe, the
accounts routes (/api/account/*,
/api/clerk/webhook) and their three
lifecycle-triggered emails -- see "Accounts
(Clerk)" above, and /api/context/save,
/api/context/set-status (see "Full Context
feed" above)
archive_client.py lookup()/push() to the Archive + proxy_get()
(cookie-forwarding for auth-aware pages) +
the /internal/account/* wrappers +
the /internal/context/* wrappers
agenda_text.py agenda/packet text: which links to read
(CivicPlus, CivicClerk), fetch, extract,
guards (WO-1966; used by the Archive's ingest)
agenda_extract_worker.py one PDF parse in a child process under a
memory ceiling
db/
engine.py DATABASE_URL (falls back to local SQLite) +
async engine/session
models.py MeetingResolution (the cache/log table) +
ProblemReport (viewer-submitted issue reports)
crud.py get_cached_resolution, log_resolution,
get_stats, list_resolutions,
log_problem_report, list_problem_reports
outcomes.py classify_outcome() — content-quality bucketing
platforms/
base.py detect_platform(), AssetFinder ABC, the
adapter registry, CalendarPageError,
resolve_via_platform()
models.py ResolvedMeeting / TranscriptSegment
media_scan.py shared regex-based media-URL scanner
(Granicus + Swagit + CA Legislature), including
caption-file detection across VTT/SRT/TTML/DFXP/
ITT/SCC/STL/SBV/SUB/SMI/SAMI/keyword-gated XML/TXT
granicus.py, civicclerk.py, swagit.py, escribe.py,
ca_legislature.py, legistar.py, civicplus.py,
primegov.py, youtube.py
one AssetFinder per platform
utils/vtt_parser.py WebVTT/SRT/TTML/DFXP/ITT parsers, a generic
best-effort text fallback for other caption
formats, parse_captions_by_extension() --
the single dispatch point every caption-
fetching adapter goes through (see "Supported
platforms"'s "Caption format handling" note) --
dedupe_rollup_cues() for roll-up captions, and
detect_language_from_texts(), which votes
across the whole transcript rather than
sampling its opening
utils/gov_registry/ one namespaced, deterministic `gov_id` per
GOVERNMENT, and the resolver that assigns it
(WO-98/WO-99). resolver.py is the seven-rung
ladder; classify.py decides the TYPE before
any place lookup, which is what stops LADWP
resolving as Los Angeles; tables.py holds the
Census/StatCan lookups; display.py generates
the display name and the /j/ slug from the
registry row; registry.py reads the committed
CSVs. Imports the standard library, its own
data files and jurisdiction_enrich — nothing
else from this repo, so it can be lifted into
its own distribution and imported by
rtr-discovery and rtr-upcoming (decision D5).
A test enforces that boundary.
utils/jurisdiction_data/ the national tables plus the registry's own
three committed, hand-editable files:
governments.csv (generated for national rows,
reviewed for minted ones),
curated_governments.csv (hand-written, and
the ONLY source of lookup aliases) and
tenant_overrides.csv (the "hard-code a public
name" pins),
and consolidated_governments.csv (the ~40 US
city-counties the Census keeps two rows for;
says which row IS the government, applied at
the resolver's single table-hit choke point)
utils/video_hand_check.py
the ONE shared gate for "is this really a
meeting video?" (WO-933): assess_video_candidate()
returns pass / reject / cannot_tell. Holds the
title allow/block lists, the risky-platform set
(HIGH_RISK_TITLE_PLATFORMS), the decorative-video
checks and the wrong-body phrase list; used by
verify_hub(), generic_fallback.py and the sweep
and ingest scripts. See its module docstring.
utils/url_normalize.py normalize_url() — the cache/log dedup key
utils/clerk_auth.py get_clerk_user_id()/clerk_frontend_api_url() --
see "Accounts (Clerk)" above; deliberately
duplicated in archive/utils/clerk_auth.py
templates/base.html shared layout (nav, brand, manifest/favicon
link, Clerk sign-in link/nav slot)
templates/index.html URL input page
templates/meeting.html video + transcript page shell
templates/about.html about page
templates/privacy.html privacy policy (/privacy) -- linked from
the footer of BOTH services; the route
lives here because the resolver holds the
public origin
static/player.js all client-side behavior
static/style.css
static/manifest.json PWA manifest (Add to Home Screen)
static/icon.svg app icon referenced by the manifest + favicon
archive/ is a second, independent FastAPI app (own requirements.txt,
own deploy) for permanent pages — see
Permanent pages (the Archive) above.
archive/
main.py FastAPI app: /internal/lookup, /internal/ingest,
/internal/transcription/* (all token-gated),
/m/{slug}, /m/{slug}/transcript.{txt,srt},
/m/{slug}/card.jpg (see "Meeting card images"
above),
/meetings, /coverage, /coverage/detail,
/state/{slug}, /state/all-50, /j/{slug},
/api/jurisdictions (public, backs /coverage's
search box), /sitemap.xml, /feed.xml,
/api/health, /account/saved, and the token-gated
/internal/account/* routes -- see "Accounts
(Clerk)" above, plus /context, /context/new,
/context/feed.xml and the token-gated
/internal/context/* routes -- see "Full
Context feed" above
db/
engine.py own DATABASE_URL resolution + local SQLite
fallback (archive_dev.db -- never shares the
resolver's dev.db)
models.py MeetingPage, TranscriptVersion,
MeetingPageUrlAlias, TranscriptionJob (see "On-
demand transcription" above), SavedItem (see
"Accounts (Clerk)" above),
MeetingPageThumbnail (extracted video frames,
see "Meeting card images" above), ContextEntry
(see "Full Context feed" above),
AgendaVersion (agenda/packet text, WO-1966)
agenda_versions.py store/list AgendaVersion rows, per page only
crud.py identity matching/dedup, slug generation,
content-hash version dedup, list_pages()
(paginated + filtered, backs /meetings),
get_platform_coverage()/get_jurisdiction_coverage()
(back /coverage's sections),
get_full_jurisdiction_coverage() (backs the
split-out /coverage/detail),
search_jurisdictions() (backs /api/jurisdictions),
get_state_coverage_index()/get_state_page_data()
(back /state/{slug} + /coverage's state links),
get_national_government_list()/
get_all50_page_data() (back /state/all-50 --
see "/state/{slug}" above for why these are NOT
get_state_page_data() with its state filter
dropped),
get_jurisdiction_hub_data()/
list_indexable_hub_entries() (back /j/{slug} +
its sitemap entries),
list_all_page_slugs() (backs /sitemap.xml),
list_recent_pages_for_feed() (backs /feed.xml),
the TranscriptionJob lifecycle (create/claim/
report/confirm/finalize),
promote_transcript_version(), and the saved-
items functions (save/unsave meeting/search,
list_saved_items, delete_account_data -- the
right-to-deletion cascade), plus the
ContextEntry save/status/list functions
backing /context, /context/feed.xml and
/context/new -- see "Full Context feed" above
data/
hub_slug_aliases.csv retired /j/ slugs -> the hub they 301 to,
generated by scripts/score_gov_registry.py --
see archive/utils/hub_aliases.py for why this
is a committed file and not a table
utils/
agenda_ingest.py the ingest-time agenda/packet read (WO-1966),
see "Agenda and packet text" above
gov_groups.py gov_type -> the /state/* heading it lands
under. Replaced gov_classify.py (WO-99), a
regex over the display name that filed
"Broward County Public Schools" under
counties and "Minnesota Senate" under cities
hub_aliases.py the 301 lookup behind /j/{old-slug}
slugify.py slug generation
search.py keyword matching for list_pages() -- exact
(substring) and fuzzy (bounded edit-distance
per word) search, see "Search" above
transcript_export.py to_txt()/to_srt() formatters, backs
/m/{slug}/transcript.{txt,srt}
url_normalize.py deliberate duplicate of
app/utils/url_normalize.py -- kept in sync
manually so the two services stay
deploy-independent
language.py same deliberate-duplicate pattern, of
app/utils/vtt_parser.py's
detect_language_from_texts() -- used to detect
a transcribed version's language (see "On-
demand transcription" above)
email.py Resend integration: transactional sends
(confirmation, transcription-complete,
transcription-failed) and an
audience-membership check -- see "On-demand
transcription" and "Accounts (Clerk)" above
clerk_auth.py deliberate duplicate of
app/utils/clerk_auth.py -- see "Accounts
(Clerk)" above
video_thumbnail.py which frame a meeting card shows and the
background extraction that produces it, plus
the free i.ytimg.com URL for YouTube-backed
pages -- see "Meeting card images" above
clips.py schema.org Clip entries out of stored
agenda_items, including the endOffset
resolution for a run of items sharing one
source timestamp
context_links.py parses a pasted social-post URL into
network/id/social_url_key, parses a pasted
RTR share link into slug/t, and describes each
network's embed (or lack of one) -- see "Full
Context feed" above
context_editors.py CONTEXT_EDITOR_CLERK_IDS allowlist check --
see "Full Context feed" above
templates/
meeting_page.html SSR permanent page + transcript-version
picker (real content on first byte, for
crawlability -- not client-fetched JSON like
app/templates/meeting.html); also carries the
schema.org VideoObject JSON-LD block, the
"Report a problem" form, and the "Save this
meeting" button/bookmark icon
meeting_list.html paginated index + search/filter form, an
RSS autodiscovery link, and the "Save this
search" button
saved_items.html "My Saved Items" page -- see "Accounts
(Clerk)" above
coverage.html per-platform table + real example page
links, the alphabetical "receipts" table, and
the jurisdiction search box, backs /coverage
coverage_detail.html the sortable/filterable full jurisdiction
detail table, split out 2026-08-26, noindex'd,
backs /coverage/detail
state_page.html per-state hub, backs /state/{slug}
state_all50.html national (50-US-states-only) hub, added
2026-08-26, backs /state/all-50
_government_groups.html shared grouped-government-list partial,
included by both state_page.html and
state_all50.html
sitemap.xml.jinja sitemap.xml template
feed.xml.jinja feed.xml (RSS) template
context.html the public /context feed -- see "Full
Context feed" above
_context_entry.html one feed entry, included by context.html,
context_new.html and context_entry_page.html
context_new.html the editor form + draft list, backs
/context/new
context_feed.xml.jinja /context/feed.xml (RSS)
context_entry_page.html one entry's own permalink page, backs
/context/{id}[-{slug}] -- WO-945 follow-up,
slugged + given real SEO substance in WO-946
static/style.css duplicated from app/static/style.css
static/meeting_page.js trimmed port of player.js's seek/highlight
logic, wired onto already-rendered DOM, plus
the Save-this-meeting toggle
static/saved_items.js unsave-button handlers on the saved-items page
static/context_embeds.js Instagram/TikTok/YouTube embeds on /context --
click-to-load, plus lazy autoload for the
first few on page 1 (WO-945) -- see "Full
Context feed" above
static/context_editor.js the /context/new form
shared_static/ holds the handful of JS files identical between the
resolver and Archive — both services mount it at /shared-static and
serve the exact same files, rather than each keeping its own copy:
shared_static/
deep_link.js the t=/line=/version= deep-link contract both
app/static/player.js and
archive/static/meeting_page.js depend on --
see "Running tests" above for its own JS suite
clerk_nav.js loads ClerkJS, mounts the nav sign-in
link/user avatar, exposes window.RTRClerk --
see "Accounts (Clerk)" above
worker/ is a third, independent service (own requirements.txt, own
Docker-based deploy — see "On-demand transcription" above) for processing
transcription jobs — it can run as more than one Render service from this
same codebase (rtr-transcription-worker / rtr-transcription-worker-2
in render.yaml) sharing one job queue, see that file's own comment.
Unlike the resolver/Archive split, it deliberately imports from both of
the other two: archive.db/archive.utils.email directly (it is
Archive backend logic, just in a process shape the Archive's own web dyno
can't offer) and app.platforms (read-only, to re-resolve a fresh media
URL before each chunk).
worker/
main.py the poll-claim-process loop; loads the
transcription model once at startup, reused
for every job/chunk after that
transcription_engine.py TranscriptionEngine interface + the real v1
implementation, self-hosted faster-whisper
media_probe.py -- lives in app/platforms/, not here, so
app/main.py's feasibility check can use it too
without app/ depending on worker/; see that
file's own docstring
segment_utils.py pure, dependency-free chunk math + timestamp
shifting (shift_segments()) -- kept apart from
the two files above specifically so it stays
trivially unit-testable
requirements.txt app/'s adapter-fetch deps + archive/'s DB
deps + faster-whisper -- no fastapi/uvicorn/
jinja2/slowapi, this process never serves HTTP
Dockerfile installs ffmpeg (a system binary, not a pip
package) explicitly -- see render.yaml's
comment on the resolver service for why plain
`runtime: python` is a real, unverified risk
there too
Governments that publish meeting video only on a channel, playlist or
video feed, with no link from the meeting page, are not reached by any
adapter; the join that would recover them is documented and shelved in
docs/VIDEO_TO_CALENDAR_JOIN.md.
Nothing here is finished — this is a fast-moving project with real,
tracked gaps rather than silently-assumed correctness. See BACKLOG.md
for the full, up-to-date list of open issues (completed
fixes and their verification history have moved to BACKLOG_DONE.md,
linked from there) — a few caption paths are shape-verified but not
content-verified pending a real example, some metadata (Alexandria VA
dates) can't be extracted at all, there's no UI yet to pick between
multiple caption language tracks when more than one exists, and the
account-deletion webhook's SavedItem-purge cascade has unit coverage
but has never been fired against a real deleted Clerk account (see
"Accounts (Clerk)" above).