From 82228128c15ddeb6565b13c9fc8051d61fe015f5 Mon Sep 17 00:00:00 2001 From: Aswin C Date: Wed, 7 Oct 2026 23:35:38 +0530 Subject: [PATCH] docs: update repository guidance --- .coderabbit.yaml | 2 +- .github/PULL_REQUEST_TEMPLATE.md | 10 +-- .gitignore | 16 ++-- .worktreeinclude | 4 + AGENTS.md | 140 ++++++++----------------------- ARCHITECTURE.md | 2 +- CONTRIBUTING.md | 47 +++++++++++ docs/TESTING.md | 34 ++++++-- scripts/README.md | 88 ++++++++++--------- 9 files changed, 174 insertions(+), 169 deletions(-) create mode 100644 .worktreeinclude diff --git a/.coderabbit.yaml b/.coderabbit.yaml index f579d5fbf..9f9f60312 100644 --- a/.coderabbit.yaml +++ b/.coderabbit.yaml @@ -1,6 +1,6 @@ # CodeRabbit configuration for boxlore (Android / Kotlin / Jetpack Compose). # Docs: https://docs.coderabbit.ai/reference/configuration -# Aligns with AGENTS.md, ARCHITECTURE.md, and .cursor/rules/*.mdc +# Aligns with AGENTS.md, ARCHITECTURE.md, and docs/TESTING.md. language: en-US diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 8e898f63f..279a2a6ae 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -12,25 +12,23 @@ Do **not** use sentence-case titles without a type prefix (e.g. avoid `Polish th ## Merge gate (required before merge) -Unit tests, detekt, ktlint, and the Kover coverage gate run on **every PR push** (a new commit cancels the previous in-progress unit run; plus optional Actions → Run workflow). There is **no merge queue**. +The unit workflow runs architecture checks, detekt, JVM tests, Kover coverage, dependency guards, Python release-tooling tests, and Android lint on PR pushes unless a safe docs/chore PR uses `[skip unit]`. A new commit cancels the previous in-progress run; Actions → Run workflow runs the full suite. Run `./gradlew ktlintCheck` locally for Kotlin changes; ktlint is not part of the current PR workflow. There is **no merge queue**. Master is protected by a branch ruleset. Required checks before merge: 1. **`testDebugUnitTest`** — PR pushes (new commits cancel the prior run; `[skip unit]` in the title no-ops for safe docs/chore only) 2. **`coderabbit-threads-resolved`** — every non-outdated CodeRabbit review thread is marked Resolved -Also on PRs (not ruleset-required): SonarCloud App, CodeRabbit App, Gitleaks. +Also on PRs (not ruleset-required): SonarCloud App, CodeRabbit App, Gitleaks. The PR must have **zero Sonar new-code issues**. Flow: 1. Open the PR and iterate (unit suite cancels prior runs). -2. Address **every** CodeRabbit finding and mark every CodeRabbit thread **Resolved**; wait for unit + **`coderabbit-threads-resolved`**. -3. If review decision is **`CHANGES_REQUESTED`**, do not agent-merge — ask a human to merge (or dismiss) manually. +2. Address **every** CodeRabbit finding and mark every CodeRabbit thread **Resolved**; fix Sonar new-code issues and wait for unit + **`coderabbit-threads-resolved`**. The bare CodeRabbit status only confirms that its review completed. +3. If review decision is **`CHANGES_REQUESTED`**, stop automated merging. Do not dismiss the review or force-merge; ask a maintainer to merge (or dismiss) manually. 4. Otherwise squash-merge when required checks are green. 5. Optional: Actions → Run workflow (`Unit Tests`) for a manual full gate. -Scheduled bots push to `master` via the **boxlore-master-pusher** GitHub App (ruleset Integration bypass). - ## Summary diff --git a/.gitignore b/.gitignore index ac5567c98..38ea77be6 100644 --- a/.gitignore +++ b/.gitignore @@ -38,15 +38,14 @@ bench_node.js *.log *_logcat.txt -# Standalone Proxy — lives in github.com/ashwkun/Proxy +# Local standalone backend checkout proxy/ # Sub-projects (local only) admin-panel/ telemetry-worker/ -# Hosting lives in aswin.cx / ashwkun/aswin-cx — ignore any leftover local docs/web. -# Only the web site — not the rest of docs/. +# Local website checkout; keep the rest of docs/ tracked. docs/web/ # Transformers model cache @@ -75,8 +74,11 @@ posthog-setup-report.md # Editor / agent settings .vscode/ .claude/ -# Local agent/editor noise (AGENTS.md + .cursor/rules are committed) +# Local agent/editor cache .agents/**/cache/ -.cursor/debug-*.log -.cursor/plans/ -.cursor/settings.json +/.cursor/ + +# Maintainer-only instructions and Codex hooks +/AGENTS.internal.md +/.codex/hooks.json +/.codex/hooks/ diff --git a/.worktreeinclude b/.worktreeinclude new file mode 100644 index 000000000..3e7f15a34 --- /dev/null +++ b/.worktreeinclude @@ -0,0 +1,4 @@ +# Copy private guidance into local Codex-managed worktrees; keep it out of Git. +AGENTS.internal.md +.codex/hooks.json +.codex/hooks/ diff --git a/AGENTS.md b/AGENTS.md index 16bd1bad6..eb7312a03 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,121 +1,47 @@ -# AGENTS.md — Boxlore agent contract +# Working on boxlore -Short entrypoint for Cursor / Codex / cloud agents. Prefer this over long essays. +Before planning or editing, check for `AGENTS.internal.md` in the repository root and load it when present. Report missing internal guidance. Do not perform production deployments or repository-administration operations without the relevant internal instructions. In a configured maintainer environment, missing internal guidance restricts work to read-only investigation until restored. -## Non-negotiables +When installed and trusted, local Codex hooks load the internal document at session/subagent start and after compaction. Verify loading rather than assuming that a file's presence enables it. The tracked `.worktreeinclude` copies ignored local instructions and hooks into new local Codex-managed worktrees; ordinary Git worktrees and cloud environments require separate provisioning. -- Read [`ARCHITECTURE.md`](ARCHITECTURE.md) + the touched module `README.md` before editing; **ARCHITECTURE wins** on conflicts. -- Before editing `scripts/sync/` (catalog pipeline): read [`scripts/README.md`](scripts/README.md) **and** the **Catalog sync** section below. Sync **does not** run on GitHub Actions — only on the Netcup VPS. A `git push` alone does **not** update the live runner. -- No feature→feature deps/imports; no PostHog in features (use `:core:analytics`); no Hilt/Koin/MockK. -- **Decoupling principle**: If a domain, flow, or subsystem is significant enough and makes architectural sense to decouple (e.g. settings, history, auth), keep it separate in its own focused module rather than bloating general-purpose modules. Module extractions and decoupling can **strictly only be performed post explicit confirmation from the user**. -- Never break identity/storage contracts (`applicationId`, DataStore `user_preferences`, Room names, `rss:` IDs, single `PlaybackRepository`, smart-queue refill ownership). See ARCHITECTURE identity table. -- Update the touched module README in the same change (template: [`docs/MODULE_README_TEMPLATE.md`](docs/MODULE_README_TEMPLATE.md)). -- Extend JVM `src/test` for touched logic; **bug fix ⇒ regression test** (same failure mode app-wide when shared). No Compose `androidTest` / emulator CI. -- Commit / push / open a PR **only when the user asks**. Conventional Commits titles. -- Every PR needs **exactly one** user-impact label (`user-impact-critical|high|medium|low` or `no-user-impact`); optional `backend-change`. Changelog / README upcoming workflows depend on these — see [`.cursor/rules/pr-impact-labels.mdc`](.cursor/rules/pr-impact-labels.mdc). For `user-impact-critical`, fill the PR **Release copy** regions (CHANGELOG + README); those are pasted verbatim and must not be Groq-rewritten. -- Merge when required checks are green (squash). Required checks: **`testDebugUnitTest`** + **`coderabbit-threads-resolved`**. SonarCloud / Gitleaks / CodeRabbit apps still run on PRs (fix Sonar issues). Unit suite cancels prior in-progress runs on new commits; `[skip unit]` / `[skip changelog]` only when appropriate. No merge queue / `merge-ci`. -- CodeRabbit (mandatory for agents): - - Address every CodeRabbit finding and mark **every** CodeRabbit review thread **Resolved** before merge. Do not rely on the bare `CodeRabbit` status (that only means the review job finished). The hard gate is **`coderabbit-threads-resolved`**. - - If the PR review decision is **`CHANGES_REQUESTED`** (CodeRabbit or anyone with write access): **stop**. Do **not** dismiss the review, do **not** force-merge / queue merge. Tell the user the PR is blocked on requested changes and ask them to merge (or dismiss) manually. -- SonarCloud: **0 new-code issues** on the PR (App quality gate). Fix Sonar findings; do not treat a missing Sonar ruleset requirement as permission to ignore them. -- Never commit secrets (`local.properties`, `.env`, keystores, `google-services.json`). -- Do **not** hand-edit `CHANGELOG.md` or README Upcoming / What's New regions (`` / ``) — `changelog-on-merge` owns those. Write the exact bullets in the PR **Release copy** markers instead; the merge/release scripts paste them as-is and must not Groq-rewrite filled regions. Hand-edits of CHANGELOG/README are OK only for intentional release-note rewrites with matching script contracts. -- **boxlore-only:** do not change other `boxcreate` repos or org-wide bot settings unless asked. Keep proxy/backend internals out of public Android PR text. -- Product name in user-facing copy is **boxlore** (all lowercase), not “Boxlore” / “BoxLore”. -- **Mandatory UX writing skill:** Before drafting or changing any text shown in the app, read and apply the available `ux-writing-content-design` skill's `SKILL.md` and relevant references, even when the user does not explicitly invoke it. This includes titles, labels, buttons, instructions, dialogs, errors, notifications, empty/loading/success states, and accessibility descriptions. Review wording in the surrounding UI and verify that it accurately describes the app's behavior. -- Cards / panels: solid Material 3 surfaces only — no glassmorphism / translucent card backgrounds. +## Project context -## Source of truth (priority order) +boxlore is a Kotlin Android app with `:app`, `:core:*`, and `:feature:*` modules. `AppContainer` owns dependency wiring. The external backend is not part of this repository; build and launch setup is described in [CONTRIBUTING.md](CONTRIBUTING.md). -1. Latest user message (explicit overrides win) -2. This file + [`.cursor/rules/*.mdc`](.cursor/rules/) -3. [`ARCHITECTURE.md`](ARCHITECTURE.md) -4. Touched module `README.md` -5. [`docs/TESTING.md`](docs/TESTING.md) and [`.github/PULL_REQUEST_TEMPLATE.md`](.github/PULL_REQUEST_TEMPLATE.md) +## Engineering constraints -## Where to look +- Read [ARCHITECTURE.md](ARCHITECTURE.md) and the affected module README before changing code. Architecture owns module boundaries and identity/storage contracts; module READMEs describe local behavior. +- Features must not depend on or import other features. Use `:core:analytics` instead of PostHog directly in features. Do not add Hilt, Koin, Dagger, or MockK. +- Keep cohesive capabilities in focused modules. Obtain explicit maintainer approval before module extraction or architectural decoupling. +- Preserve application/storage names, legacy worker and service identities, deep links, `rss:`/negative IDs, and cache-key contracts. Keep one UI-scoped `PlaybackRepository` and service-owned Smart Queue refill. +- Update the affected module README in the same change using [the module template](docs/MODULE_README_TEMPLATE.md). +- Add or extend hermetic JVM `src/test` coverage for changed logic. Bug fixes require a regression test for the failure mode, including shared behavior where applicable. Do not add Compose instrumentation or emulator CI. +- Before changing catalog scripts, read [scripts/README.md](scripts/README.md). Catalog sync runs outside GitHub Actions; pushing code does not deploy it. Production operations require the internal runbook. +- Keep work scoped to this repository. Preserve unrelated edits and avoid destructive cleanup of developer checkouts. -| Topic | Doc | -| :--- | :--- | -| Module graph, DI, identity | [`ARCHITECTURE.md`](ARCHITECTURE.md) | -| Unit / Kover / Konsist / CI | [`docs/TESTING.md`](docs/TESTING.md) | -| Catalog sync pipeline (VPS, not GHA) | [`scripts/README.md`](scripts/README.md) | -| Always-on agent rules | [`.cursor/rules/`](.cursor/rules/) | -| PR body / merge checklist | [`.github/PULL_REQUEST_TEMPLATE.md`](.github/PULL_REQUEST_TEMPLATE.md) | -| Impact labels + merge gate | [`.cursor/rules/pr-impact-labels.mdc`](.cursor/rules/pr-impact-labels.mdc) | - -## Catalog sync (VPS — not GitHub Actions) - -**Hard stop before editing `scripts/sync/`.** Full detail: [`scripts/README.md`](scripts/README.md). - -### Sync does not run on GitHub +## Product and data safeguards -- The old GHA workflow **`sync-pi-data` is sunset / removed**. -- There is **no** GitHub cron for charts, PI import, episode sync, or vectorization. -- Do **not** re-add a GitHub sync workflow unless the user explicitly asks. -- Do **not** assume `git push` to `master` updates the live pipeline by itself. +- Use **boxlore** in user-facing copy. Keep wording concise, consistent with surrounding UI, and accurate about the resulting behavior. Apply available UX writing guidance; maintainer skill requirements are in the internal document. +- Use solid Material 3 surfaces for cards and panels. +- Keep secrets and local configuration out of Git, including `.env`, `local.properties`, keystores, and `google-services.json`. +- Release workflows own `CHANGELOG.md` and the root README's generated Upcoming/What's New regions. Put exact release wording in the PR template's release-copy regions. Intentional release-note rewrites require matching script contracts. +- Keep private operational details out of public documentation and Android PR descriptions. -### Sync runs on the Netcup VPS +## Validation and review -| What | Where | -| :--- | :--- | -| Live runner root | `/opt/boxlore-sync/` | -| **Code the cron executes** | `/opt/boxlore-sync/repo/scripts/sync/` (`run-sync.sh` → `cd $REPO`) | -| Orchestrator | `/opt/boxlore-sync/run-sync.sh` (systemd timers; panel install from `netcup-panel`) | -| Secrets / budgets | `/opt/boxlore-sync/.env` (never commit) | -| Run logs | `/opt/boxlore-sync/logs/runs/` | -| Local Turso / Qdrant | `/opt/boxlore-stack` | +Use [docs/TESTING.md](docs/TESTING.md) to select relevant checks and distinguish local verification from remote CI or device verification. Use the Gradle wrapper. -**Deploy rule:** after changing `scripts/sync/` or `scripts/package.json`, **redeploy into `/opt/boxlore-sync/repo`** (rsync/pull) or the live job keeps old code. GitHub is deploy source of truth; **`repo` is the runner**. Ignore `/opt/boxlore-sync/boxlore-src` for cron — only `repo` matters. +Follow [the PR template](.github/PULL_REQUEST_TEMPLATE.md) for Conventional Commit titles, exactly one user-impact label, release copy, and review requirements. Before an authorized squash merge, required checks must be green, every CodeRabbit finding addressed and thread resolved, and SonarCloud must have zero new-code issues. Agents must not dismiss requested-change reviews or bypass merge checks. -### Tagged files +## Documentation map -| Path | Role | +| Topic | Source | | :--- | :--- | -| `scripts/sync/lib/config.js` | Countries, tiers, check cadence, embed provider, budgets | -| `scripts/sync/01-…` … `07-…` | Staged pipeline | -| `scripts/sync/lib/turso.js` + `turso-page.js` | Page large SELECTs (`RESPONSE_TOO_LARGE`) | -| `scripts/sync/lib/staleness.js` / `episode-caps.js` | Core vs relaxed checks; per-storefront caps | -| `scripts/sync/lib/embedder.js` / `scalars.js` / `podcast-index.js` | `bge` vs `qwen`; scrub non-scalars; PI rate-limit failsafes | -| `scripts/package.json` | Sync Node deps + `npm run test:sync` | - -### Checklist - -1. Read [`scripts/README.md`](scripts/README.md) + current `config.js` country list. -2. Run `npm run test:sync` from `scripts/` when touching sync lib logic. -3. Keep large Turso reads on `fetchAllPaged` / country×category pages. -4. When asked to ship: commit/push **and** deploy to `/opt/boxlore-sync/repo`, then verify on the VPS. -5. Never commit sync `.env` / PI / Turso / Telegram secrets. - -## Large refactors / P1 batches (hard stop) - -For large refactors or multi-issue P1 batches: **propose in plain English → wait for explicit user OK → then branch and code**. Do not start implementation on approval-shaped silence. Details: [`.cursor/rules/p1-workflow.mdc`](.cursor/rules/p1-workflow.mdc). - -## Default local loop - -After UI / app-behavior changes: `./gradlew installDebug` on a connected device when available. Do not run device automation (taps, screenshots, layout dumps) unless the user asks. Gradle must use the real `GRADLE_USER_HOME` — see [`.cursor/rules/gradle-no-sandbox.mdc`](.cursor/rules/gradle-no-sandbox.mdc). - -## Cursor Cloud specific instructions - -boxlore is a single-product **Android app** (Kotlin, multi-module Gradle: `:app`, `:core:*`, `:feature:*`). There is no server to run — "running" the product means building the debug APK and launching it on an Android emulator. The "smart" backend (search/recommendations/briefing) is a private external service and is not in this repo; without it the app still works as a standard podcast client (offline library, RSS, OPML). - -### Environment (already provisioned in the snapshot) -- JDK 17 at `/usr/lib/jvm/java-17-openjdk-amd64` (the repo requires 17; the base image also ships JDK 21 — do not let Gradle pick 21). -- Android SDK at `~/Android/Sdk` with `platform-tools`, `build-tools;36.0.0`, `platforms;android-36`, `emulator`, and system images `android-34;google_apis;x86_64` and `android-34;default;x86_64`. -- `~/.bashrc` exports `JAVA_HOME`, `ANDROID_HOME`, `ANDROID_SDK_ROOT`, and `PATH`. New non-login shells may not source it — if `java -version` shows 21 or `sdkmanager` is missing, `source ~/.bashrc` first. -- The update script runs `scripts/ci/write-cloud-agent-local-config.sh`, which writes `local.properties` (`sdk.dir`) and a non-secret stub `app/google-services.json`. Both are gitignored and are NOT secrets. - -### Build / test / lint (no device needed) -Standard commands, all via the Gradle wrapper (see also `.github/workflows/unit-tests.yml`): -- Build debug APK: `./gradlew assembleDebug` (first run downloads Gradle 9.6.1 + deps, ~4–5 min). -- Unit tests: `./gradlew testDebugUnitTest --continue` -- Lint: `./gradlew detekt ktlintCheck lintDebug` -- Coverage floor / dep guard: `./gradlew :koverVerifyMerged :app:dependencyGuard` -- Optional local screenshot goldens (not CI-gated): `./gradlew :feature:home:recordRoborazziDebug` → PNGs under `screenshots/baselines/`. - -### Running the app on the emulator (non-obvious gotchas) -- There is **no `/dev/kvm`** here, so the emulator runs with software CPU emulation (`-no-accel -gpu swiftshader_indirect -no-window`). It is usable but very slow: boot takes several minutes and the starved CPU triggers frequent system-wide "System UI / Process system isn't responding" ANR dialogs. These are environment slowness, not app bugs — dismiss with "Wait" and give screens 60–90s to settle. -- Prefer the lighter **AOSP image** (`system-images;android-34;default;x86_64`, AVD `boxlore_aosp`) over `google_apis`: Play Services background work on the google_apis image makes ANRs much worse. -- After install, run `adb shell cmd package compile -m speed -f cx.aswin.boxlore` to AOT-compile — this removes the runtime class-verification overhead that otherwise causes a playback-service ANR on the slow CPU. -- The launcher activity is `cx.aswin.boxlore/.MainActivity`. -- **The app requires a syntactically valid `BOXLORE_API_BASE_URL` to launch.** `PodcastRepository` eagerly builds a Retrofit client from it, so an empty value (the default in the stub config) crashes at startup with `IllegalArgumentException: Expected URL scheme 'http' or 'https'`. To launch the UI offline, add to `local.properties`: `BOXLORE_API_BASE_URL=https://api.boxlore.example` (and optionally `BOXLORE_PUBLIC_KEY=demo-placeholder-key`), then rebuild. With a placeholder URL, backend-dependent screens (Explore search, Lore/curiosity, briefing) show graceful "failed to load" states; offline features (onboarding, Library/Downloads, RSS/OPML) work normally. For real backend functionality, set the private `BOXLORE_API_BASE_URL`/`BOXLORE_PUBLIC_KEY` as secrets. +| Local setup and build versus runtime configuration | [CONTRIBUTING.md](CONTRIBUTING.md) | +| Architecture, dependency direction, stable identities | [ARCHITECTURE.md](ARCHITECTURE.md) | +| Commands, test selection, coverage, CI | [docs/TESTING.md](docs/TESTING.md) | +| Script responsibilities and development checks | [scripts/README.md](scripts/README.md) | +| Module documentation format | [docs/MODULE_README_TEMPLATE.md](docs/MODULE_README_TEMPLATE.md) | +| PR labels, release copy, review and merge process | [.github/PULL_REQUEST_TEMPLATE.md](.github/PULL_REQUEST_TEMPLATE.md) | + +`AGENTS.internal.md` supplements these public contracts with local authorization, skills, environment setup, and operations. It is intentionally absent from public clones and must remain untracked. diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 62a88b69f..421016428 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -10,7 +10,7 @@ The graph is layered so playback and features depend inward on catalog and lower ### Decoupling and module granularity -When a domain, presentation flow, or capability is significant enough to form a cohesive domain (such as the settings hub, listening history timeline, or authentication SDK), **keep it decoupled in its own standalone module**. Avoid co-locating unrelated subsystems inside general-purpose modules (e.g., settings inside `:feature:home`, history inside `:feature:library`, or auth credentials inside `:core:network`). If something is significant enough and makes architectural sense to decouple, keep it separate. Any module extractions or architectural decoupling must strictly be executed **only post explicit confirmation and approval from the user**. +Keep cohesive domains and flows in focused modules: settings belongs in `:feature:settings`, listening history in `:feature:history`, and authentication in `:core:auth`. Avoid placing unrelated capabilities in general-purpose modules. Obtain explicit maintainer approval before extracting modules or changing architectural boundaries. ## Identity and storage diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 7893fccae..2aebccf4c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -39,3 +39,50 @@ The recommendation, search, and catalog backend proxy is tracked in a separate p ## Local Development & Personal Use In accordance with the [PolyForm Strict License 1.0.0](LICENSE), you are welcome to build and run boxlore locally for personal study, testing, and private noncommercial use. Refer to the [README.md](README.md) and [ARCHITECTURE.md](ARCHITECTURE.md) for build setup and module details. + +### Requirements + +| Tool | Requirement | +| :--- | :--- | +| Java | JDK 17; set `JAVA_HOME` and the Android Studio Gradle JDK to this installation | +| Android SDK | Platform 36, Platform-Tools, and the Build-Tools version requested by the Android Gradle plugin | +| Gradle | Use the checked-in `./gradlew` wrapper; no separate Gradle installation is needed | +| Node.js | Version 20 and npm for scripts; not needed for ordinary Android builds | +| Python | Python 3 for local configuration and release-tooling tests | +| Device | Android 12 / API 31 or newer when running the app | + +Set `ANDROID_HOME` to the SDK installation and add its `platform-tools` directory to `PATH`. Accept SDK licenses through Android Studio or `sdkmanager --licenses`. Verify that `java -version` reports 17 before building. + +### Build configuration + +From the repository root, create non-secret local configuration: + +```bash +bash scripts/ci/write-cloud-agent-local-config.sh +./gradlew assembleDebug +``` + +The configuration script writes `sdk.dir` in `local.properties` and a non-secret `app/google-services.json` stub. It preserves either file when it already exists. Verify `sdk.dir` if the script cannot find your SDK. Both files are ignored by Git; keep them untracked and never commit real service configuration, keys, or signing material. + +The stub supports building and JVM checks without production credentials. It does not provide working Firebase services or API access. The APK is written to `app/build/outputs/apk/debug/app-debug.apk`. + +### Running locally + +Building and launching have different configuration requirements. The app constructs its API client during startup, so an empty `BOXLORE_API_BASE_URL` causes startup to fail. For a local launch without backend access, add these non-secret placeholders to the existing `local.properties`, retaining `sdk.dir`: + +```properties +BOXLORE_API_BASE_URL=https://api.boxlore.example/ +BOXLORE_PUBLIC_KEY=demo-placeholder-key +``` + +Rebuild after changing these values. The placeholder has a valid URL format but supplies no backend: search, recommendations, and briefing cannot retrieve live results. Use local library, downloads, and RSS/OPML flows for checks that do not need the API. Working backend and Firebase services require separately provisioned configuration. + +Connect a device or start an emulator, then install and launch: + +```bash +adb devices +./gradlew installDebug +adb shell am start -n cx.aswin.boxlore/.MainActivity +``` + +Use the [test selection guide](docs/TESTING.md#choose-checks-for-a-change) for validation commands and [architecture guide](ARCHITECTURE.md) for module ownership. Local maintainer environments also use root `AGENTS.internal.md` for private workflow and operational instructions. diff --git a/docs/TESTING.md b/docs/TESTING.md index 08acdcd65..d2c890b84 100644 --- a/docs/TESTING.md +++ b/docs/TESTING.md @@ -15,7 +15,7 @@ How Boxlore is tested: layers, commands, coverage floors, architecture gates, an Automated coverage focused on **hermetic JVM** for product logic (queue fill, ranking, catalog, prefs, feature `logic/`). High Kover floors fail CI on drop. Architecture guards fail the unit PR job on graph drift. -**Strategy:** constructors, domain ports, shared fakes in `:core:testing`, assemblers, Turbine. No MockK/Hilt. No Application-backed Home/Info suites. Media3 service / `PlaybackRepository` stay out of the line gate; covered by policy unit tests. Maestro YAML is validated nightly (no paid Maestro Cloud device runs). +**Strategy:** constructors, domain ports, shared fakes in `:core:testing`, assemblers, Turbine. No MockK/Hilt. No Application-backed Home/Info suites. Media3 service / `PlaybackRepository` stay out of the line gate; covered by policy unit tests. Maestro YAML is validated nightly; the existing optional Cloud job runs only when its credentials are configured and is not a PR gate. ## Layers @@ -27,11 +27,30 @@ Automated coverage focused on **hermetic JVM** for product logic (queue fill, ra | Android lint | `./gradlew lintDebug` | Manifest / resource / API lint | Done | | Coverage (Kover) | `./gradlew :koverVerifyMerged` | Merged floor (ratchet toward 80%) | WIP | | Screenshots | `screenshots/baselines/` + optional Roborazzi (local) | Visual regressions | Optional | - | Maestro | `maestro/` YAML validate | Flow file presence/syntax | Done | Architecture boundaries: [`ARCHITECTURE.md`](../ARCHITECTURE.md). +## Choose checks for a change + +Run commands from the repository root. Start with the affected behavior and broaden to shared consumers when a change crosses module boundaries. See [local setup](../CONTRIBUTING.md#local-development--personal-use) for tool and configuration requirements. + +| Change | Checks | +| :--- | :--- | +| Android logic or state | Add or extend JVM regression coverage; run `./gradlew ::testDebugUnitTest` for affected modules (for example, `:core:playback:testDebugUnitTest`), then `./gradlew testDebugUnitTest --continue` for shared behavior | +| Module boundaries, dependencies, or composition | Run both `scripts/ci/check-feature-no-boxlore-database.sh` and `scripts/ci/check-feature-no-posthog.sh` with Bash; run `./gradlew :core:testing:testDebugUnitTest :app:dependencyGuard :core:catalog:dependencyGuard :core:playback:dependencyGuard` | +| Kotlin, Android resources, or manifests | Run `./gradlew detekt ktlintCheck lintDebug`; include affected JVM tests for changed behavior | +| App UI or runtime behavior | Build/install with `./gradlew installDebug` on an available device; verify the affected screen and action; use optional local Roborazzi when appropriate | +| Catalog sync logic | Run `npm ci --prefix scripts` once, then `npm run test:sync --prefix scripts`; deployment is a separate operation requiring the internal runbook | +| New-episode notification checker | Run `npm ci --prefix scripts` once, then `npm run test:check-new-episodes --prefix scripts` | +| Tracked-feed repair | Run `npm ci --prefix scripts` once, then `node --test scripts/backfill-tracked-podcast-feeds-lib.test.js` | +| Changelog or release tooling | Run `python3 -m unittest discover -s .github/scripts -p 'test_*.py' -v` | +| Documentation only | Review local links and instruction destinations; run `git diff --check` | + +For Android code changes, the full CI command is `./gradlew detekt testDebugUnitTest :koverVerifyMerged :app:dependencyGuard :core:catalog:dependencyGuard :core:playback:dependencyGuard --continue`, with `./gradlew lintDebug` in a separate job. The unit workflow also runs the architecture shell guards and Python release-tooling tests. `ktlintCheck` is a local check; it is not part of the current PR workflow. + +Report which checks completed and distinguish automated coverage from manual checks or paths that still need verification. Script tests exercise local logic; they do not verify a production deployment. + ## Stack - JUnit 5 (+ Vintage where leftovers remain) @@ -119,6 +138,7 @@ bash scripts/ci/check-feature-no-posthog.sh Detekt: `config/detekt/{detekt.yml,baseline.xml}`. ktlint: per-project baselines under `config/ktlint/`. +Use the root `detekt` task; individual modules do not all define one. These commands do not rewrite source files. ## Module × layer checklist @@ -152,7 +172,7 @@ Application-backed Home/Info suites are **not** pursued; hermetic `logic/` + ass | :--- | :--- | | Flow YAML under `maestro/` | Done | | Nightly YAML validate | Done | -| Maestro Cloud device runs | Out of scope (not subscribed) | +| Maestro Cloud device runs | Optional; requires configured service credentials | See [`maestro/README.md`](../maestro/README.md). @@ -170,12 +190,14 @@ See [`docs/screenshots/README.md`](screenshots/README.md). | Workflow | Runs | When | Status | | :--- | :--- | :--- | :--- | -| `unit-tests.yml` | Architecture + detekt + unit + Kover + lint + Dependency Guard | PR / dispatch | Done | +| `unit-tests.yml` | Architecture + detekt + unit + Kover + lint + Dependency Guard + Python release tests | PR / master push / dispatch | Done | | `coderabbit-threads-resolved.yml` | Fail unless all non-outdated CodeRabbit review threads are Resolved | PR / review | Done | | `gitleaks.yml` | Secret scan | PR / push to master | Done | -| `maestro-nightly.yml` | Validate Maestro YAML | Nightly / manual | Done | +| `maestro-nightly.yml` | Validate Maestro YAML; optional Cloud device job when configured | Nightly / manual | Done | + +**Merge gate:** master uses a branch ruleset with no merge queue. Required checks are **`testDebugUnitTest`** and **`coderabbit-threads-resolved`**. SonarCloud, CodeRabbit, and Gitleaks also run on PRs. Fix all Sonar new-code issues and address every CodeRabbit finding, marking every review thread Resolved; the bare CodeRabbit status only confirms that the review completed. -**Merge gate:** master uses a branch ruleset (no merge queue). Required checks: **`testDebugUnitTest`** and **`coderabbit-threads-resolved`**. SonarCloud / CodeRabbit / Gitleaks still run on PRs (fix Sonar new-code issues; resolve CodeRabbit threads — the bare `CodeRabbit` status only means the review finished). The unit suite cancels prior in-progress runs on each PR push (or via Actions → Run workflow). Put `[skip unit]` in the PR title to no-op that job for docs/chore-only changes (still reports green; `workflow_dispatch` always runs full). Bots push to master via **boxlore-master-pusher** (ruleset Integration bypass). +The unit suite cancels prior in-progress runs on each PR push. `[skip unit]` in the PR title no-ops that job only for docs/chore changes with no logic risk; it still reports green. Actions → Run workflow always runs the full suite. If the review decision is `CHANGES_REQUESTED`, stop automated merging and have a maintainer handle the review or merge manually. Otherwise squash-merge only after the required checks are green and review requirements are satisfied. Repository-administration procedures belong in root `AGENTS.internal.md`. Protected inputs: `app/google-services.json` is gitignored; CI writes a non-secret stub. diff --git a/scripts/README.md b/scripts/README.md index c95815873..7671a6bea 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -1,64 +1,56 @@ -# scripts/ — **READ BEFORE EDITING (agents)** +# Scripts -> **Stop.** If you are about to change anything under `scripts/sync/`, read this file first. -> Shipping sync code to GitHub alone does **not** run the catalog pipeline. +Development guidance for catalog synchronization, notification checks, feed repair, and CI helpers. Read this file and the current [`sync/lib/config.js`](sync/lib/config.js) before changing catalog sync. -## Sync does **not** run on GitHub +## Catalog sync and deployment -- The old GitHub Actions workflow **`sync-pi-data`** is **sunset / removed**. -- There is **no** GHA cron that refreshes charts, imports podcasts, syncs episodes, or vectorizes. -- Do **not** re-add a GitHub sync workflow unless the user explicitly asks. -- Do **not** assume `git push` to `master` updates the live pipeline by itself. +Catalog sync refreshes charts, imports podcasts, syncs episodes, and vectorizes content. It runs outside GitHub Actions; the former `sync-pi-data` workflow has been removed. Adding a replacement GitHub sync workflow requires explicit maintainer approval. -## Sync runs on the Netcup VPS +A Git push does not deploy this pipeline. Changes to `scripts/sync/` or `scripts/package.json` require a separate deployment and verification that the active runner uses the updated code and configuration. Read the production runbook in root `AGENTS.internal.md` before deployment. If it is unavailable, obtain it from the maintainer before performing production operations. -| What | Where | -| :--- | :--- | -| Live runner root | `/opt/boxlore-sync/` | -| **Code the cron executes** | `/opt/boxlore-sync/repo/scripts/sync/` (`run-sync.sh` → `cd $REPO`) | -| Orchestrator | `/opt/boxlore-sync/run-sync.sh` (systemd timers; panel install from `netcup-panel`) | -| Secrets / budgets | `/opt/boxlore-sync/.env` (never commit) | -| Run logs | `/opt/boxlore-sync/logs/runs/` | -| Local Turso / Qdrant | `/opt/boxlore-stack` (sqld + qdrant on the same box) | - -**Deploy rule:** after changing `scripts/sync/` (or `scripts/package.json`), you must **redeploy to the VPS `repo` tree** (rsync/pull into `/opt/boxlore-sync/repo`) or the live job keeps running the old code. GitHub is the source of truth for *what to deploy*, not the runner itself. - -A second tree `/opt/boxlore-sync/boxlore-src` may exist as a snapshot — **cron does not use it**. Only `repo` matters for the live sync. - -## Tagged surfaces (edit with care) +## Catalog components | Path | Role | | :--- | :--- | -| [`scripts/sync/lib/config.js`](sync/lib/config.js) | **Countries, tiers, check cadence, embed provider, budgets** — Phase-2 country list lives here only | -| [`scripts/sync/01-refresh-charts.js`](sync/01-refresh-charts.js) … [`07-record-stats.js`](sync/07-record-stats.js) | Staged pipeline (charts → import → episodes → **cleanup before vectors on CLEANUP=1** → stats) | +| [`scripts/sync/lib/config.js`](sync/lib/config.js) | Countries, tiers, check cadence, embedding provider, and budgets; the country list is defined here | +| [`scripts/sync/01-refresh-charts.js`](sync/01-refresh-charts.js) … [`07-record-stats.js`](sync/07-record-stats.js) | Staged pipeline: charts → import → episodes → cleanup before vectors when `CLEANUP=1` → stats | | [`scripts/sync/lib/tip-queue.js`](sync/lib/tip-queue.js) | Turso `ep_vec_tip_queue` — durable tip lane (IDs); stage 3 upsert, stage 4 delete on complete | | [`scripts/sync/lib/vectorize-lanes.js`](sync/lib/vectorize-lanes.js) | Tip-first drain order + partial-complete flag rules for stage 4 | | [`scripts/sync/lib/pi-handoff.js`](sync/lib/pi-handoff.js) | Same-run episode payloads stage 3→4 (not cross-run durable) | | [`scripts/sync/lib/turso.js`](sync/lib/turso.js) + [`turso-page.js`](sync/lib/turso-page.js) | HTTP Turso client; **always page** large SELECTs (`RESPONSE_TOO_LARGE`) | | [`scripts/sync/lib/staleness.js`](sync/lib/staleness.js) | Core vs relaxed episode-check windows | | [`scripts/sync/lib/episode-caps.js`](sync/lib/episode-caps.js) | Per-storefront episode vector caps | -| [`scripts/sync/lib/embedder.js`](sync/lib/embedder.js) | `bge` (default) vs `qwen` on VPS | +| [`scripts/sync/lib/embedder.js`](sync/lib/embedder.js) | Embedding providers: `bge` (default) and `qwen` | | [`scripts/sync/lib/podcast-index.js`](sync/lib/podcast-index.js) | PI API client — Retry-After / global cooldown failsafes | | [`scripts/sync/lib/text.js`](sync/lib/text.js) | Description cleaning + embed text (**uncapped**; `PAYLOAD_DESCRIPTION_MAX` null) | | [`scripts/sync/lib/scalars.js`](sync/lib/scalars.js) | Scrub non-scalars before Qdrant/Turso writes | | [`scripts/package.json`](package.json) | Sync Node deps + `npm run test:sync` | -## Agent checklist before sync edits +## Changing catalog sync + +- Extend hermetic tests under `sync/lib/*.test.js` and run `npm run test:sync` from `scripts/`. +- Keep large Turso reads on `fetchAllPaged` or country/category pages; avoid unbounded wide SELECTs. +- Preserve the chart index: `charts.itunes_id` is TEXT, so casting that column to INTEGER disables the index and inflates `rows_read`. Use `c.itunes_id = CAST(p.itunes_id AS TEXT)`, [`sync/lib/chart-countries.js`](sync/lib/chart-countries.js) (`loadCountriesByItunesId` plus a JavaScript filter), or page podcasts by `id` and normalize IDs in JavaScript. This applies to stages 2, 3, 4, 5, and remediation. +- Keep credentials and environment files out of Git, including `.env`, Podcast Index keys, Turso tokens, and Telegram secrets. + +Use Node.js 20 to match the script workflows. From the repository root: -1. Read this README + current [`sync/lib/config.js`](sync/lib/config.js) country list. -2. Prefer hermetic tests under `scripts/sync/lib/*.test.js` (`npm run test:sync` from `scripts/`). -3. Keep large Turso reads on `fetchAllPaged` / country×category pages — do not reintroduce unbounded wide SELECTs. -3b. **Never** join/filter charts with `CAST(c.itunes_id AS INTEGER)` (charts column is TEXT). That disables the index and can turn candidate / pending scans into huge `rows_read`. Use `c.itunes_id = CAST(p.itunes_id AS TEXT)`, [`sync/lib/chart-countries.js`](sync/lib/chart-countries.js) (`loadCountriesByItunesId` + JS filter), or page podcasts by `id` and normalize itunes ids in JS. Stages 2/4/5/remediate must follow the same rule as Stage 3. -4. After commit/push (when asked): **deploy to `/opt/boxlore-sync/repo`** and confirm the runner sees the new countries/flags (`node -e '…require config…'` on the VPS). -5. Never commit `.env`, PI keys, Turso tokens, or Telegram secrets. +```bash +npm ci --prefix scripts +npm run test:sync --prefix scripts +npm run test:check-new-episodes --prefix scripts +node --test scripts/backfill-tracked-podcast-feeds-lib.test.js +``` -## Other things under `scripts/` +These commands test local logic; running a production script requires the internal runbook and deployment authorization. The catalog suite is a local check and is not scheduled by GitHub Actions. -Non-sync helpers (CI stubs, one-offs, data files) may also live here. They are unrelated to the VPS catalog cron unless documented otherwise. When in doubt, ask before assuming GHA runs them. +## Other scripts -### Check New Episodes (GitHub Actions — not VPS catalog sync) +CI configuration helpers, maintenance tools, and data files also live here. Check the owning workflow or runbook before using a tool; its presence in this directory does not mean it runs automatically. -[`.github/workflows/new-episode-check.yml`](../.github/workflows/new-episode-check.yml) still runs on a ~30 minute cron. It is **not** the sunset catalog pipeline. +### Check New Episodes + +[`.github/workflows/new-episode-check.yml`](../.github/workflows/new-episode-check.yml) runs approximately every 30 minutes on GitHub Actions. It polls notification registrations independently of catalog sync. | What | Where | | :--- | :--- | @@ -67,12 +59,26 @@ Non-sync helpers (CI stubs, one-offs, data files) may also live here. They are u | Last-notified state | [`scripts/data/episode-tracker.json`](data/episode-tracker.json) (the Action commits this) | | Tests | `npm ci` then `npm run test:check-new-episodes` from `scripts/` (the Check New Episodes workflow runs the same script after `npm ci`) | -If a tracked row has HTTPS `feedUrl`, the checker polls that RSS/Atom feed and compares `lastRssKey` (guid, else enclosure). This selection is independent of the phone's Missing episodes? cache setting. Otherwise catalogue rows keep Podcast Index `episodes/byfeedid?max=1` vs `lastEpisodeId`. With no saved state, the first check quietly seeds a baseline. When a catalogue row already has `lastEpisodeId` but no `lastRssKey`, its first RSS check can notify if the newest feed item matches a PI episode with a different ID. For catalogue shows, RSS fetch failure falls back to Podcast Index and does not wipe `lastRssKey`. Publisher feeds may be tens of MB (The Daily ~18 MB); the checker uses the same **25 MB** hard cap as Android `RssFeedClient` and reads the complete bounded body, so oldest-first feeds can put the newest episode at the end. Interrupted, oversized or failed reads retain the last-good release baseline. RSS/Atom parsing accepts playable enclosures regardless of attribute order and ignores non-media/untitled entries, matching Android release hydration. The Action never mints negative episode ids; unmatched feed-only drops omit `episodeId` and deep-link the podcast page. The phone persists the raw GUID/enclosure hint before hydration and resolves the release in its publisher-feed Room catalog. Visible release alerts request high Android FCM priority; foreground discovery recovers missed episodes on the phone by default. Background auto-download discovery is off by default and requires separate user consent; there is no periodic phone notification worker. +Release selection and recovery: + +- A tracked HTTPS `feedUrl` selects RSS/Atom and compares `lastRssKey` (GUID, otherwise enclosure), independently of the phone's Missing episodes? cache setting. Other catalog rows use Podcast Index `episodes/byfeedid?max=1` and `lastEpisodeId`. +- The first check without saved state quietly seeds a baseline. When an existing catalog row has `lastEpisodeId` but no `lastRssKey`, its first RSS check can notify if the newest feed item matches a PI episode with a different ID. Catalog RSS failures fall back to Podcast Index without clearing `lastRssKey`. +- Read the complete body within the **25 MB** cap shared with Android `RssFeedClient`; oldest-first feeds may put the newest release at the end. Interrupted, oversized, or failed reads retain the last-good baseline. Parsing accepts playable enclosures regardless of attribute order and ignores non-media or untitled entries. +- The Action never creates negative episode IDs. Unmatched feed-only releases omit `episodeId` and open the podcast page. The phone persists the raw GUID/enclosure hint before hydration and resolves the release in its publisher-feed Room catalog. +- Visible alerts request high Android FCM priority. Foreground discovery recovers missed episodes by default. Background auto-download discovery is off by default and requires separate consent; the phone has no periodic notification worker. + +### Public RSS notification registrations + +The checker supports explicitly opted-in public `rss:` subscriptions from RTDB. Topics use `new_ep_rss__` and payloads retain `rss:`. Pure RSS failures or disabled URLs preserve the last-good state for retry and never fall back to PI. Each new URL scope quietly seeds a baseline; repeated keys remain quiet and FCM failures do not advance it. + +RSS-only Git state uses `rss:~` keys with a SHA-256 episode-key digest and episode title, excluding the feed URL, raw GUID, and enclosure. Inactive and legacy show-only scopes are retired. Feed URLs remain in RTDB and push payloads and are published by the weekly backup workflow, so this shared checker is unsuitable for private or premium feeds. These registrations do not require catalog sync changes. + +Device RTDB rows use `rss:~~`. Group only matching show IDs and exact trimmed feed URLs: different URLs under a preserved show ID have independent topics and release histories. The device journals registration and cleanup before publication, awaits RTDB and FCM acknowledgements, and retries failed cleanup through event-driven WorkManager work even when notifications are disabled. Migration removes legacy device rows and show-only topic subscriptions. ### Weekly tracked-podcast feed repair -[`backfill-tracked-podcast-feeds.yml`](../.github/workflows/backfill-tracked-podcast-feeds.yml) runs weekly and can be dispatched manually. It shares the `tracked-podcast-rtdb-maintenance` concurrency group with the new-episode checker, so the two RTDB jobs never run at the same time. Before mutation, [`backfill-tracked-podcast-feeds.js`](backfill-tracked-podcast-feeds.js) writes numeric registrations and aggregated public RSS show metadata to [`data/tracked-podcasts-backups/`](data/tracked-podcasts-backups/) and retains one file per UTC ISO week for the latest 10 weeks; the workflow must commit the changed snapshot to `master` successfully before its repair step can start. It then resolves only rows without a valid HTTPS `feedUrl` through the authenticated boxlore `/podcast` endpoint, probes HTTPS upgrades for legacy HTTP feeds, and uses an exact-title Apple directory match only when the API cannot supply a secure URL. Each result is written with a transaction to the individual `feedUrl` leaf, so a newer app write wins and `title` / `imageUrl` cannot be replaced. Unresolved rows are logged and retried on the next run. +[`backfill-tracked-podcast-feeds.yml`](../.github/workflows/backfill-tracked-podcast-feeds.yml) runs weekly and supports manual dispatch. Its `tracked-podcast-rtdb-maintenance` concurrency group prevents overlap with the notification checker. -- Check New Episodes supports explicitly opted-in public `rss:` subscriptions from RTDB. Their topics map to `new_ep_rss__`; payloads keep `rss:`. Pure RSS failures/disabled URLs never fall back to PI and preserve last-good state for retry. Each new URL scope quietly seeds a baseline, repeated keys stay quiet, and FCM failures do not advance it. RSS-only Git state uses `rss:~` keys and stores a SHA-256 episode-key digest and episode title, never the feed URL/raw GUID/enclosure. Inactive and legacy show-only scopes are retired; changing a URL starts a separate baseline. Feed URLs remain in RTDB/push payloads and are published by the weekly RTDB backup workflow, so this shared checker is unsuitable for private/premium feeds. No change to the VPS catalog sync pipeline is needed. +Before mutation, [`backfill-tracked-podcast-feeds.js`](backfill-tracked-podcast-feeds.js) saves numeric registrations and aggregated public RSS show metadata to [`data/tracked-podcasts-backups/`](data/tracked-podcasts-backups/), retaining one snapshot per UTC ISO week for the latest 10 weeks. The workflow must successfully commit a changed snapshot to `master` before repairing rows. Backups preserve every accepted public URL scope without publishing device registration IDs. -RSS tracking uses per-device `rss:~~` RTDB rows. The checker groups only matching show IDs and exact trimmed feed URLs, so different URLs under a preserved ID have independent topics and release histories. The device journals registration and cleanup before publication, awaits both RTDB and FCM acknowledgements, and retries failed cleanup with event-driven WorkManager work even when no notifications are enabled. Legacy device rows and show-only topic subscriptions are removed during migration. Weekly backups retain every accepted public URL scope without publishing device registration IDs. Feed URL repair skips RSS rows instead of searching for private/premium shows in the catalog. +Repair considers catalog rows without a valid HTTPS `feedUrl`. It uses the authenticated boxlore `/podcast` endpoint, probes HTTPS upgrades for legacy HTTP feeds, and falls back to an exact-title Apple directory match when the API cannot supply a secure URL. Transactions update only the `feedUrl` leaf, preserving newer app writes, `title`, and `imageUrl`. Unresolved rows are logged and retried next run. RSS rows are skipped rather than searched in the catalog.