Skip to content

Make object storage provider-neutral for B2 - #698

Open
Volyarosh94 wants to merge 3 commits into
mainfrom
feat/object-store-b2
Open

Volyarosh94 wants to merge 3 commits into
mainfrom
feat/object-store-b2

Conversation

@Volyarosh94

@Volyarosh94 Volyarosh94 commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

What

  • services/object_store.py replaces services/r2_client.py. It reads OBJECT_STORE_* (endpoint, region, key id, secret, bucket) and falls back to the R2_* names while none of those is set, so the droplets keep working until their env moves.
  • For B2 the region comes from the s3.<region>.backblazeb2.com endpoint. Non-AWS endpoints use path-style requests and send checksums only where an operation requires one; botocore's newer default CRC32 trailers were rejected by B2 until mid-2025.
  • scripts/object_store_check.py writes, reads, lists, multipart-uploads and deletes test objects, and prints PASS/FAIL per step.
  • Runbook section "Moving object storage to B2"; .env.example and the architecture doc name the store neutrally.

Calls I made

  • Any OBJECT_STORE_* set means only those names are read. An incomplete set turns the store off with one error naming what's missing, instead of mixing B2 and R2 settings. OBJECT_STORE_BUCKET has no default.

  • A bare endpoint host gets https://.

  • The alert r2_replication_failed is now object_store_replication_failed, and the two "r2 disabled" status strings follow. Nothing in the repo keys on them; alert rules outside it might.

  • The final pass after the switch is rclone copy --update, not rclone sync, which would delete what the app wrote to B2 since the switch.

  • The runbook has rclone read its credentials with read -rs, so no key lands in the shell history, and uses a separate temporary B2 key for the copy.

  • The test suite ignores any store env in the shell, so a local run can't write to a real bucket.

  • delete_objects still sends a CRC32 header (botocore requires one there). B2 documents support for it; the live check covers it.

  • The runbook copies through B2's S3 API rather than rclone's native b2 backend, so rclone check compares MD5 instead of sizes only.

  • The runbook raises Backblaze's daily caps first. Copying this bucket with the free-tier caps hit the Class B cap (rclone reads each upload back unless told not to) and then the storage cap.

Tested:

  • just check passes up to web, whose dashboard lint fails with the same 54 problems as main. The rest of web passes.
  • New tests cover config resolution, region, B2 request headers, presign scope and the check script.
  • scripts.object_store_check against the new B2 bucket: provider b2, 7/7 PASS.

OBJECT_STORE_* env names, with R2_* as the fallback until the droplets move.
Adds scripts/object_store_check and a runbook section for the move.
@claude

This comment has been minimized.

@claude

claude Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Claude finished @Volyarosh94's task in 10m 30s —— View job


Code review: Make object storage provider-neutral for B2

  • Gather context (PR diff, CLAUDE.md compliance)
  • Review services/object_store.py core logic
  • Review scripts/object_store_check.py
  • Review migration call sites (state_snapshot.py, archive_lifecycle.py, users_backup.py, backfill_archive_fuzz.py)
  • Review tests, config, and docs changes
  • Post final review

Summary

This is a clean, thoroughly-tested rename/generalization of r2_client.py into provider-neutral services/object_store.py. The B2-specific quirks (path-style addressing, conditional checksums, region-from-endpoint) are well documented and well covered by wire-level tests. Call-site migrations (state_snapshot.py, archive_lifecycle.py, users_backup.py, backfill_archive_fuzz.py) are mechanical 1:1 renames with no logic changes. Nothing here blocks merging, but one item is worth the author's attention before it goes live.

Worth a look before merging:

  1. backend/services/object_store.py:230-236 — the new path-style addressing and "when_required" checksum settings are gated on if config.endpoint_url:, which is true for any non-AWS provider — including R2, today's live production provider, not just B2. The old r2_client.py never set addressing_style, request_checksum_calculation, or response_checksum_validation, so R2 has been running on botocore's defaults (virtual-hosted addressing, default CRC32 trailers) in production. This PR silently changes that for R2 too, the moment it merges and deploys — before any B2 migration happens. The module comment and the entire wire-level test suite (test_the_b2_client_pins_path_style_and_checksums_when_required, test_b2_put_object_is_path_style_and_sends_no_checksum_headers, etc.) are scoped only to a b2_client/B2_ENV fixture; there's no equivalent assertion that R2 behaves the same under the new settings. In practice this is plausibly benign or even a fix (R2 has had its own reported issues with botocore's newer default checksum trailers), but as written the rationale ("botocore >= 1.36 otherwise sends CRC32 trailers that B2 has rejected") doesn't acknowledge it now also applies to R2/MinIO. Consider either scoping the branch to config.provider == "b2", or keeping it provider-neutral but adding a comment + an R2 wire test confirming the broadened behavior is intentional and safe.
    Fix this →

Minor, non-blocking:

  1. The signing region for a custom endpoint override that isn't B2/R2 (e.g. a local MinIO R2_ENDPOINT_URL/OBJECT_STORE_ENDPOINT_URL) changes from "auto" (old r2_client.py, any non-empty endpoint) to "us-east-1" (new, per the module docstring's "AWS, MinIO and other S3-compatibles" rule). This is intentional and tested, and matches MinIO's actual default region, so it reads as a fix rather than a regression — flagging only because it's an unannounced behavior change for anyone relying on the old default with a non-default MinIO region.
  2. delete_key/delete_objects still require a CRC32 checksum header per the module comment (object_store.py:362-392), but the wire-level unit tests only pin checksum headers for put_object, get_object, and multipart upload, not delete_objects. Covered instead by the live object_store_check.py run per the PR description — a reasonable but not equivalent substitute for a unit assertion.
  3. The dual OBJECT_STORE_*/legacy R2_* env scheme has a clear removal path for the .env lines (runbook step 6), but no tracked follow-up for removing the R2_* fallback code itself once every droplet has moved. Worth a dated TODO or linked issue so the "legacy" branch doesn't quietly become permanent.
  4. (Simplification, optional) _config()/_get_client()'s hand-rolled double-checked locking (object_store.py:202-267) could be functools.lru_cache(maxsize=1) as the old r2_client.py used, trimming ~20 lines and the module's only threading.Lock use. Not a problem as written, just more machinery than needed.

Verified against repo conventions (CLAUDE.md):

  • No route under /v1/nodes, no model, and nothing in backend/services/node_config.py changed, so the contracts/nodes-v1.openapi.yaml regeneration requirement doesn't apply.
  • backend/pyproject.toml/backend/uv.lock are untouched (boto3/moto were already pinned before this PR), so no relock is needed.
  • backend/.env.example was updated alongside the new OBJECT_STORE_* keys, with the legacy R2_* block kept and dated for removal post-migration.
  • No credentials, droplet addresses, or account IDs in the diff — the runbook's new B2 migration section uses placeholders (<b2-bucket>, <account-id>, <region>) and the existing ssh retina-prod alias throughout.

Things worth calling out as careful work:

  • backend/tests/conftest.py blanks all ten store env names so a developer's real backend/.env (loaded via load_dotenv() in main.py, which doesn't override existing vars) can't leak into the test suite and write to a real bucket.
  • resolve_config cleanly partitions OBJECT_STORE_* vs. legacy R2_*: any OBJECT_STORE_* value present disables R2_* fallback entirely, and a partial OBJECT_STORE_* set fails loudly naming the missing keys.
  • _endpoint_url's validation is exhaustively tested against 13 malformed-endpoint cases (whitespace, userinfo, bad ports, https:/ typos, IPv6, underscores) in test_object_store.py.
  • The runbook's B2 migration section correctly uses rclone copy --update (not sync, which would delete what the app wrote to B2 since the switch), read -rs so credentials never land in shell history, and a separate temporary B2 key for the migration.

Testing: Per the PR description, just check passes except the pre-existing web dashboard lint findings shared with main, and the live check against the new B2 bucket reports 7/7 PASS. I reviewed the diff and tests statically rather than re-running the suite myself.
· branch feat/object-store-b2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant