Skip to content

README.md

Latest commit

 

History

History
736 lines (519 loc) · 30.5 KB

README.md

File metadata and controls

736 lines (519 loc) · 30.5 KB
Raw

ARMOR Scripts

This directory contains operational, testing, and monitoring scripts for ARMOR deployments.

definition-of-done.sh

Local verification gate. scripts/definition-of-done.sh --fast runs the toolchain parity gate (scripts/toolchain-parity.sh), the compose.yaml ↔ VERSION parity gate (scripts/compose-version-parity.sh), the prohibited deployment constructs gate (scripts/prohibited-constructs-gate.sh), the Dockerfile image contract gate (scripts/image-contract-gate.sh), the documentation status gate (scripts/documentation-status-gate.sh), the executable CLI contract gate (scripts/cli-contract-gate.sh), go build ./..., go vet ./..., and the Python scripts test suite (tests/test_drift_check.py, tests/test_toolchain_parity.py, tests/test_compose_version_parity.py, tests/test_image_contract_gate.py, tests/test_cut_release.py, tests/test_restore_verifier_inventory.py, tests/test_restore_verifier_scope_validation.py, tests/test_prohibited_constructs.py, tests/test_gate_inventory.py, tests/test_documentation_status.py, tests/test_publish_release.py); without --fast it additionally runs go test ./... -short and the Agentation browser mount smoke (scripts/verify-agentation-mount.sh), which loads every web UI entry point in a headless Chromium and requires #agentation-root in the rendered DOM — failing, not skipping, when no browser is present or esm.sh is unreachable. Exits non-zero if any leg fails. tests/test_gate_inventory.py pins the Python list — and the copies in AGENTS.md, the script's own header, and the tests/README.md table — to the invocation in the script, so the documented suites cannot drift from the ones executed.

release-gate.sh

The test gate CI and the Dockerfile run before building an image: the toolchain parity check, the compose.yaml ↔ VERSION parity check, the prohibited deployment constructs gate, the Dockerfile image contract gate, the executable CLI contract gate (scripts/cli-contract-gate.sh), the publisher contract tests (tests/test_publish_release.py — network-free, fake endpoints; needs python3 + pytest, which the Dockerfile builder stage and CI's go-test step install for this leg), crypto, backend, restore-verifier, canary, config, cmd/armor and the server handlers, plus an integration-suite compile. ARMOR_RELEASE_RACE=1 adds -race to the packages that support it (CI sets it).

toolchain-parity.sh

go.mod ↔ Dockerfile toolchain parity gate. Parses go.mod's toolchain directive (the single source of the Go version) and every golang:<tag> base image in Dockerfile and Dockerfile.test, and exits 1 on any drift — a different version, an untagged (implicit :latest) or digest-pinned golang base, or a floating minor tag — and 2 on a structural break (no directive, a missing Dockerfile, a Dockerfile with no golang base). Warn-only make toolchain-check remains for the local go, where a newer version is fine. Wired into the definition of done, the release gate (so the Dockerfile builder stage gates itself), and make docker. Tests: python3 -m pytest tests/test_toolchain_parity.py -q.

compose-version-parity.sh

compose.yaml ↔ VERSION image-pin parity gate. Every ${ARMOR_VERSION:-...} default in the tracked compose.yaml (demo and production profiles) must equal the repository's VERSION file, which is the tag the README Quick Start points readers at. cut-release.sh rewrites the defaults in the release commit; this gate exits 1 on any other drift — a default pinning a different, floating or non-semver value — and 2 on a structural break (missing or non-semver VERSION, missing compose.yaml, no default left to check). Wired into the definition of done, the release gate (so no image is built from a tree whose compose pin lags VERSION), and make docker. Tests: python3 -m pytest tests/test_compose_version_parity.py -q.

image-contract-gate.sh

Dockerfile default-image contract gate. An untargeted docker build -f Dockerfile publishes the LAST stage as ronaldraygun/armor, so that stage must be the armor server — ENTRYPOINT ["/armor"] in exec form (a shell-form entrypoint cannot execute on FROM scratch) and no CMD — and the companion images' --target stages (restore-verifier-runtime, armor-fleet-runtime) must stay present under those names with their own entrypoints. Dockerfile.test is held to the same default-image rule. The final stage's name is deliberately not pinned: renaming it changes nothing an untargeted build publishes. Exit 1 on any contract violation, 2 on a structural break (missing Dockerfile, no stages). Wired into the definition of done, the release gate — so the Dockerfile builder stage refuses to build an image whose own contract is broken — and make docker. Tests: python3 -m pytest tests/test_image_contract_gate.py -q.

cut-release.sh

Cuts a release commit. scripts/cut-release.sh <MAJOR.MINOR.PATCH> (or make release V=...) writes VERSION, rewrites every compose.yaml ${ARMOR_VERSION:-...} default to the new version, prepends a CHANGELOG.md entry generated from the commit subjects since the previous v* tag (bead-checkpoint and release commits excluded), commits release: armor <version> with exactly those three files, and pushes to origin main. CI does everything after that (images, tag, Forgejo and GitHub releases); see docs/release-process.md.

Options: --dry-run prints the entry and changes nothing; --no-push commits without pushing.

Refuses to run off main, with staged changes in the index, or with a version that is not newer than the current one. Tests: python3 -m pytest tests/test_cut_release.py -q — isolated git fixtures seeded with a previous v* tag, covering the exactly-three-files release commit, compose parity, changelog generation, version validation, the state gates, --dry-run purity, and re-cut refusal.

publish_release.py

The idempotent release publisher the armor-build workflow runs after the image-existence gate, also usable by hand for backfills: creates the annotated v<version> tag at --commit through the Forgejo API (never moves an existing tag; exit 3 on conflict), creates or refreshes the Forgejo release with the image digests, waits for the push mirror to carry the tag to GitHub, then creates or refreshes the GitHub release. Credentials come only from FORGEJO_TOKEN and GITHUB_TOKEN in the environment; DOCKER_CONFIG_JSON optionally points at a Docker config whose Docker Hub auth lets it resolve the private image digests.

FORGEJO_TOKEN=... GITHUB_TOKEN=... python3 scripts/publish_release.py --version 0.1.1969 --commit <full sha> --dry-run

Options: --dry-run, --tag-only, --no-github, --workflow NAME (credited in the body), --forgejo-api, --github-api, --github-wait-seconds. Exit codes: 0 ok, 2 usage, 3 tag conflict, 4 GitHub failure, 5 Forgejo failure. Tests: python3 -m pytest tests/test_publish_release.py -q — run by the definition of done and the release gate (armor-562d57c9).

Version Drift Monitoring

Automated version drift monitoring across ARMOR deployments.

Full documentation: docs/drift-check.md and the continuously running Deployment runbook

drift_check.py

The GitOps-managed armor-drift-checker Deployment passes the cluster API configuration and fails closed when a Deployment or its live pods cannot be verified. It writes both the JSON report and a Prometheus text snapshot; use the cluster-api and require-live options when reproducing that mode manually.

Fleet drift check with live verification and deduplicated alerting.

Purpose: Enumerates ARMOR deployments, compares each against the approved latest release, and classifies every deployment as current, stale, mismatched, or unavailable. Optionally probes running versions via each cluster's /version endpoint (or the Server: ARMOR/<version> header) and files ONE deduplicated alert bead (bead create --unique-ref drift-check:<fingerprint>) when anything is non-current.

Inputs: declarative-config checkout, release tags via github-release-fetcher.py (or --releases-file), optional per-cluster probe URLs

Options:

  • --config FILE - Configuration file (default: config/drift-config.json)
  • --manifests DIR - declarative-config checkout (default from config)
  • --releases-file FILE - Releases JSON (default: run github-release-fetcher.py)
  • --latest-tag TAG - Treat TAG (e.g. v0.1.1970) as the latest release even if the release source does not list it yet
  • --probe-url CLUSTER=URL - Live /version endpoint for a cluster (repeatable)
  • --expected-cluster NAME - Cluster that must have an ARMOR manifest (repeatable)
  • --releases-threshold N / --days-threshold N - Stale thresholds (default: 50 / 30)
  • --json - Machine-readable JSON output
  • --emit-bead - File the deduplicated alert bead when anything is non-current
  • --dry-run - With --emit-bead, print the bead command instead of running it

Examples:

# Classify the fleet against the live release list
python3 scripts/drift_check.py --json

# Probe running versions, then file a deduped alert bead if anything drifted
python3 scripts/drift_check.py --emit-bead

# Preview the alert without filing
python3 scripts/drift_check.py --emit-bead --dry-run

Exit codes: 0 (all current), 1 (stale/mismatched/unavailable present), 2 (error)

check-version-drift.sh

Main orchestrator script for version drift monitoring.

Purpose: Scans ARMOR deployments across all clusters, compares against GitHub releases, and flags deployments needing attention.

Inputs: None (reads from declarative-config and GitHub)

Options:

  • --releases N - Flag deployments N or more releases behind (default: 3)
  • --days N - Flag deployments N or more days behind (default: 30)
  • --json - Output machine-readable JSON
  • --output FILE - Write report to file
  • --sort-by FIELD - Sort by: cluster, releases, days, correctness (default: correctness)
  • --config FILE - Use configuration file (JSON)

Examples:

# Basic check with default thresholds
./scripts/check-version-drift.sh

# Custom thresholds
./scripts/check-version-drift.sh --releases 5 --days 60

# JSON output for automation
./scripts/check-version-drift.sh --json --output report.json

# Sort by cluster name
./scripts/check-version-drift.sh --sort-by cluster

Exit codes: 0 (all OK), 1 (drift detected), 2 (error)

version-drift-check.py

Python pipeline that wires together all version drift components.

Purpose: Integrates github-release-fetcher, find-armor-deployments, and compare-version-drift into a single pipeline.

Inputs: Command-line arguments for thresholds and endpoints

Examples:

# Run with default settings
python3 scripts/version-drift-check.py

# Custom thresholds
python3 scripts/version-drift-check.py --releases-threshold 5 --days-threshold 60

find-armor-deployments.py

Discovers ARMOR deployments in declarative-config.

Purpose: Scans jedarden/declarative-config for ARMOR deployment files and extracts image tags.

Inputs: Path to declarative-config (defaults to ../declarative-config)

Outputs: JSON with deployment metadata per cluster

Examples:

# Scan default location
python3 scripts/find-armor-deployments.py

# Scan specific path
python3 scripts/find-armor-deployments.py --path /path/to/declarative-config

validate-restore-verifier-scopes.py

Validates declarative-config against the ADR-004 restore-verifier scope contract.

Purpose: Every restore-verifier Deployment must carry exactly one ARMOR_BUCKET and one ARMOR_MEK as valueFrom references (never literals, never envFrom), must not straddle scopes (a second bucket or MEK anywhere in the pod — init containers included — is a violation), and no two verifier Deployments may resolve to the same scope identity (cluster, namespace, bucket source, MEK source, B2 key sources). Proxy-declared scopes with no verifier are reported because ADR-004 makes verifier rollout a per-scope decision — a warning, or an error under --strict-coverage. Discovery is delegated to find-armor-deployments.py, so its classification rules (argo-workflows exclusion, .disabled files, placeholder tags) apply here too. Tests: python3 -m pytest tests/test_restore_verifier_scope_validation.py -q.

Inputs: Path to a declarative-config checkout (default ~/declarative-config)

Outputs: Per-verifier scope lines, a per-scope coverage table, and ERROR/WARNING findings; --json for the machine-readable report

Examples:

# Validate the live tree; exit 0 with uncovered-scope warnings
python3 scripts/validate-restore-verifier-scopes.py

# Fail on uncovered scopes too (once the known gaps are closed)
python3 scripts/validate-restore-verifier-scopes.py --strict-coverage

# Machine-readable report against a specific checkout
python3 scripts/validate-restore-verifier-scopes.py /path/to/declarative-config --json

Exit codes: 0 valid (warnings allowed), 1 validation errors, 2 structural break (no k8s/ directory, or zero verifier Deployments discovered).

compare-version-drift.py

Compares deployments against releases to detect drift.

Purpose: Takes deployment and release JSON, calculates drift metrics, and flags outdated deployments.

Inputs:

  • --deployments FILE - Output from find-armor-deployments.py
  • --releases FILE - Output from github-release-fetcher.py
  • --releases-threshold N - Flag N+ releases behind (default: 50)
  • --days-threshold N - Flag N+ days behind (default: 30)

Outputs: Human-readable report or JSON with per-cluster drift analysis

Examples:

# Generate drift report
python3 scripts/compare-version-drift.py \
  --deployments deployments.json \
  --releases releases.json

# JSON output with custom thresholds
python3 scripts/compare-version-drift.py \
  --deployments deployments.json \
  --releases releases.json \
  --releases-threshold 10 \
  --days-threshold 7 \
  --json

github-release-fetcher.py

Lists ARMOR release tags and classifies correctness/security releases.

Purpose: Produces the release list drift_check.py compares against. CI writes one annotated v<version> tag per published release, so the tags are the release record.

Inputs: --source auto (default) reads the local checkout's v* tags when it has any and falls back to the GitHub tags API otherwise; --source git and --source github force one. Run git fetch --tags before relying on the checkout. --use-releases reads the GitHub Releases API instead of tags.

Outputs: JSON array with release metadata (tag, published_at, is_correctness, url), newest first

Examples:

python3 scripts/github-release-fetcher.py                  # local tags, else GitHub
python3 scripts/github-release-fetcher.py --source github  # always the GitHub API
python3 scripts/github-release-fetcher.py --limit 10

Correctness keywords: correctness, fix, critical, security, bug, patch, hotfix, urgent, vulnerability, cve, issue, regression

armor-version-drift-check.cron

Cron job definition for scheduled drift monitoring.

Purpose: Runs daily version drift checks at 9:17 AM (avoids :00/:00 load spikes).

Setup: Use setup-version-drift-schedule.sh to install

setup-version-drift-schedule.sh

Installs the cron job for automated drift monitoring.

Purpose: Sets up scheduled daily checks via cron.

Inputs: None

Examples:

bash scripts/setup-version-drift-schedule.sh

Logs: Written to logs/version-drift-check.log

Starvation Watching

Watches the bead workspace's ready frontier for starvation — open beads exist while a ready query returns zero candidates.

Full documentation: docs/notes/starvation-watch.md

starvation-watch.py

Purpose: Consumes .beads/diagnostics/pluck-diagnostics.json (rewritten by bead-rs on every ready query) and files one plain task bead per starvation episode, only when total_open_beads > 0 AND final_candidate_count == 0 reproduces across two consecutive snapshots. Healthy frontiers and drained workspaces write nothing. The alert bead embeds both snapshots verbatim plus a per-bead exclusion classification, and is deduplicated per episode via --unique-ref — one episode can never file two beads, even if the watcher's state file is lost.

Inputs: .beads/diagnostics/pluck-diagnostics.json (read-only)

Options:

  • --workspace DIR - Bead workspace to watch (default: this repo's root)
  • --dry-run - Print the would-be alert bead instead of filing
  • --self-test - Run the 21 built-in scenario checks and exit
  • --max-gap-seconds N - Max age of the previous snapshot for the pair to count as consecutive (default: 3600)
  • --diagnostics PATH / --state-file PATH / --bead-bin BIN - Overrides for testing

Scheduling: armor-starvation-watch.{service,timer} — a systemd --user oneshot every 15 minutes at :07/:22/:37/:52 (worker-machine-side; deliberately not a k8s Job and not a NEEDLE worker-loop change — NEEDLE's PluckNoCandidate telemetry already covers the counter side).

Examples:

python3 scripts/starvation-watch.py --self-test        # verify the state machine
python3 scripts/starvation-watch.py --dry-run          # one live cycle, file nothing
bash scripts/setup-starvation-watch-schedule.sh        # install the timer
journalctl --user -u armor-starvation-watch.service -n 20

Testing & Validation

verify-agentation-mount.sh

Browser smoke test for the Agentation toolbar on ARMOR's three web UI entry points (proxy dashboard, demo dashboard, and fleet console). The workspace rule is verify by mounting, never by grepping the tag — a page with the module tag but no import map renders perfectly while the toolbar never mounts — so this script drives a real headless Chromium against every page served from its real route tables and requires #agentation-root in the rendered DOM with the toolbar inside. Exits non-zero if a mount check fails OR if every check skipped (no browser, esm.sh unreachable): a skip means nothing was verified. The structural wiring pins (import map before the module tag before the mount check, module endpoint serving the vendored module) run always, in go test and the definition of done; the browser leg needs this script, which the full definition of done (no --fast) runs as a gate leg — --fast omits it because it needs a Chromium-family browser and esm.sh reachability, so it is neither deterministic nor quick. The inventory and per-page test names are pinned by internal/agentation/enumeration_test.go, so adding a UI entry point without adding its mount check fails the Go suite.

./scripts/verify-agentation-mount.sh
AGENTATION_BROWSER=/path/to/chromium ./scripts/verify-agentation-mount.sh

alerting-rule-unittest.sh

Offline semantics test for the shipped alert set (ADR-002 / ADR-004 §6): extracts the contract-pinned rule group from internal/metrics/testdata/restore-verifier-monitoring.yaml and runs it through promtool test rules (in the pinned prom/prometheus image, via docker) against scripts/alerting-rules-unittest.yaml. Proves in seconds that freshness failures fire past the 12h window, multipart-health transitions trip at 10m and resolve on recovery, and the failure-rate/ratio/divergence alerts match their contracted rendering — without waiting on the live stack or breaking a real canary.

./scripts/alerting-rule-unittest.sh          # SUCCESS or a per-scenario diff
PROMTOOL_IMAGE=prom/prometheus:v3.6.0 ./scripts/alerting-rule-unittest.sh

alerting-smoke-test.sh

Live end-to-end verification of the activated alerting stack (the default profile is iad-ci): collection (VictoriaMetrics scraping the armor server :9001 admin mux and the restore-verifier :9002 listener, numeric armor_* series fresh — including the unix-seconds *_last_check_timestamp gauges the former string-valued diagnostics were converted into when the deployed image exports them), evaluation (vmalert carrying the five shipped rules, expressions matching the contract, no eval errors), consistency (staleness and multipart alerts active exactly when their expression says so — both directions), and delivery (Alertmanager ready, ntfy receiver present, and any pending/firing vmalert alert present in Alertmanager; the rendered config and alert payloads are never printed). A quiet stack is valid by default; ALERTING_REQUIRE_ACTIVE=1 makes a controlled drill require an already-active alert without synthesizing one. Routes through the stack's tailnet-only hostnames, so it must run from inside the tailnet.

./scripts/alerting-smoke-test.sh
VM_BASE=... VMALERT_BASE=... AM_BASE=... ./scripts/alerting-smoke-test.sh
# Require the optional diagnostic timestamp after its image rollout:
ARMOR_EXPECT_CANARY_TIMESTAMP=1 ./scripts/alerting-smoke-test.sh
# Require correlation of a deliberately staged pending/firing alert:
ALERTING_REQUIRE_ACTIVE=1 ./scripts/alerting-smoke-test.sh

The shape knobs (ARMOR_JOB_REGEX, ARMOR_EXPECT_SERVER_TARGETS, ARMOR_EXPECT_VERIFIER, ARMOR_EXPECT_CANARY) also support a future per-cluster rollout to kube-prometheus-stack clusters and ARMOR servers without a verifier; the activation recipe is in §6 of the alerting runbook.

test-armor-endpoints.sh

Tests all ARMOR endpoints for expected responses and status codes.

Purpose: Verifies health, readiness, and S3 endpoints return correct status codes.

Inputs: Environment variables for endpoint configuration

  • NAMESPACE - Kubernetes namespace (default: armor)
  • SERVICE - Kubernetes service name (default: armor)
  • S3_PORT - S3 API port (default: 9000)
  • ADMIN_PORT - Admin API port (default: 9001)
  • TIMEOUT - Request timeout in seconds (default: 5)

Examples:

# Test default configuration
./scripts/test-armor-endpoints.sh

# Test specific namespace
NAMESPACE=armor-production ./scripts/test-armor-endpoints.sh

# Test with custom timeout
TIMEOUT=10 ./scripts/test-armor-endpoints.sh

test_filesystem_backend.sh

Verifies filesystem backend works with aws-cli.

Purpose: Full integration test of filesystem backend including PUT/GET/range/multipart/DELETE cycle.

Inputs: Environment variables

  • ARMOR_BINARY - Path to armor binary (default: ./armor)
  • Auto-generated test data and directories

Examples:

# Test with local armor binary
ARMOR_BINARY=./build/armor ./scripts/test_filesystem_backend.sh

# Test with default binary
./scripts/test_filesystem_backend.sh

Acceptance: ARMOR_BACKEND=filesystem ARMOR_FS_PATH=/tmp/x ARMOR_MEK=... ARMOR_AUTH_ACCESS_KEY=... ARMOR_AUTH_SECRET_KEY=... armor serve starts and passes full S3 cycle

test_invalid_credential_rejection.py

Verifies ARMOR properly rejects invalid credentials.

Purpose: Tests that invalid AWS credentials return 403 Forbidden with meaningful error messages, and rejection happens quickly (no long timeouts).

Inputs: Environment variables

Examples:

# Test default endpoint
python3 scripts/test_invalid_credential_rejection.py

# Test specific endpoint
ARMOR_ENDPOINT=http://localhost:9000 python3 scripts/test_invalid_credential_rejection.py

Test cases: Invalid credentials, malformed signatures, missing auth headers

test_s3_auth_acceptance.py

S3 authentication acceptance test suite.

Purpose: Verifies valid AWS Signature V4 authentication is accepted (ARMOR implements V4 only, not deprecated V2).

Inputs: Environment variables

  • ARMOR_ENDPOINT - ARMOR server URL (default: http://localhost:9000)
  • ARMOR_ACCESS_KEY - Valid access key
  • ARMOR_SECRET_KEY - Valid secret key
  • ARMOR_BUCKET - Test bucket name
  • ARMOR_REGION - AWS region (default: us-east-1)

Examples:

# Test with credentials
ARMOR_ACCESS_KEY=testkey ARMOR_SECRET_KEY=testsecret \
python3 scripts/test_s3_auth_acceptance.py

# Test with full configuration
ARMOR_ENDPOINT=http://localhost:9000 \
ARMOR_ACCESS_KEY=testkey \
ARMOR_SECRET_KEY=testsecret \
ARMOR_BUCKET=test-bucket \
ARMOR_REGION=us-east-1 \
python3 scripts/test_s3_auth_acceptance.py

Note: ARMOR correctly implements AWS Signature V4 only (not V2) for security. AWS deprecated V2 in 2019.

validate-deployment.sh

Performs 50MB multipart round-trip validation against deployed ARMOR instances.

Purpose: Full deployment validation test including multipart upload, download, SHA-256 verification, and metadata checks.

Inputs: Positional arguments

  1. Cluster name
  2. ARMOR endpoint URL
  3. S3 bucket name
  4. Scratch prefix (optional, default: armor-validation-scratch)

Environment variables:

  • ARMOR_AUTH_ACCESS_KEY - Required authentication key
  • ARMOR_AUTH_SECRET_KEY - Required secret key

Examples:

# Validate rs-manager deployment
./scripts/validate-deployment.sh \
  rs-manager \
  http://localhost:9000 \
  rs-manager

# Validate with custom scratch prefix
ARMOR_AUTH_ACCESS_KEY=key ARMOR_AUTH_SECRET_KEY=secret \
./scripts/validate-deployment.sh \
  iad-ci \
  http://armor.iad-ci:9000 \
  armor-bucket \
  test-prefix

Tests: Sequential multipart upload (required by ARMOR), download and SHA-256 verification, object metadata check, cleanup

Deployment & Operations

verify-cloudflare-setup.sh

Verifies Cloudflare B2 proxy setup for ARMOR.

Purpose: Checks DNS resolution, Cloudflare IPs, SSL configuration, and B2 backend connectivity.

Inputs: Positional arguments

  1. Cloudflare domain
  2. B2 bucket name

Examples:

# Verify Cloudflare setup
./scripts/verify-cloudflare-setup.sh armor-b2.example.com my-armor-bucket

# Verify production setup
./scripts/verify-cloudflare-setup.sh armor.ardenone.com armor-prod

Checks: DNS resolution, Cloudflare IP ownership, SSL certificate, B2 bucket accessibility

cleanup-restore-env.sh

Clean restore environment while preserving logs.

Purpose: Archives current logs and cleans working directories for Litestream restore environment.

Inputs: Positional argument

  1. Restore environment path (optional, default: /home/coding/ARMOR/scratch/litestream-restore)

Examples:

# Clean default location
./scripts/cleanup-restore-env.sh

# Clean specific location
./scripts/cleanup-restore-env.sh /path/to/restore-env

# Show help
./scripts/cleanup-restore-env.sh --help

Operations: Archives log files, cleans databases and temp directories, logs cleanup action

reset-restore-env.sh

Complete restore environment reset.

Purpose: Removes and recreates Litestream restore environment directory structure.

Inputs: Positional argument

  1. Restore environment path (optional, default: /home/coding/ARMOR/scratch/litestream-restore)

Examples:

# Reset default location
./scripts/reset-restore-env.sh

# Reset specific location
./scripts/reset-restore-env.sh /path/to/restore-env

# Show help
./scripts/reset-restore-env.sh --help

Operations: Backs up existing logs, recreates directory structure (databases, logs, restored, temp), sets permissions

Development & Debugging

probe-armor-multipart-alignment.py

Validates ARMOR's part-size alignment requirements for multipart uploads.

Purpose: Reproducible test of ADR-015 exemptions for lone parts and final parts, including exact GET/HEAD readback without zero-padding.

Full documentation: probe-armor-multipart-alignment.README.md

Inputs: Environment variables

  • ARMOR_BUCKET - Required: Target bucket
  • ARMOR_ENDPOINT_URL - Optional: ARMOR endpoint (defaults to AWS S3)
  • AWS_ACCESS_KEY_ID - Optional: AWS access key
  • AWS_SECRET_ACCESS_KEY - Optional: AWS secret key

Examples:

# Test against local ARMOR
export ARMOR_BUCKET=devimprint
export ARMOR_ENDPOINT_URL=http://127.0.0.1:9000
python3 scripts/probe-armor-multipart-alignment.py

# Test against AWS S3
export ARMOR_BUCKET=my-test-bucket
python3 scripts/probe-armor-multipart-alignment.py

# Test with explicit credentials
export ARMOR_BUCKET=devimprint
export ARMOR_ENDPOINT_URL=http://127.0.0.1:9000
export AWS_ACCESS_KEY_ID=your-key
export AWS_SECRET_ACCESS_KEY=your-secret
python3 scripts/probe-armor-multipart-alignment.py

Test cases: Single misaligned part, single aligned part, aligned first + short final, misaligned first + short final, exactly-one-block part, zero-byte final part, PutObject misaligned length

Prerequisites: Python 3 with boto3, AWS credentials configured

verify-multipart-integrity.py

Verification script for multipart-era corruption audit.

Purpose: Tests restore/decrypt operations on candidate objects to detect corruption from the multipart alignment bug era.

Inputs: Positional arguments

  1. candidates.json - Output from cross-reference-affected-objects.py
  2. output.json - Optional output file for verification results

Environment variables:

  • ARMOR_DECRYPT_PATH - Path to the armor binary (default: armor from PATH); the script runs armor decrypt

Examples:

# Verify candidates
python3 scripts/verify-multipart-integrity.py candidates.json

# Verify with output file
python3 scripts/verify-multipart-integrity.py candidates.json results.json

# Use a specific armor binary
ARMOR_DECRYPT_PATH=/path/to/armor \
python3 scripts/verify-multipart-integrity.py candidates.json

Outputs: JSON with verification status (VERIFIED, CORRUPTED, FAILED) per object

Note: cross-reference-affected-objects.py has been archived to scripts/archive/

Archived Scripts

One-off investigation scripts and superseded tools are archived in scripts/archive/:

Enumeration scripts:

  • enumerate-all-unaudited-buckets.py
  • enumerate-armor-apexalgo-bucket.py
  • enumerate-large-objects-http.py
  • enumerate-large-objects.py
  • enumerate-rs-manager-bucket.py
  • enumerate-unaudited-buckets.py

Filtering and analysis:

  • filter-affected-objects.py
  • filter-candidate-objects.py
  • cross-reference-affected-objects.py
  • generate-filtered-output.py
  • parse-deployment-windows.py
  • corruption-audit-framework.py

Verification:

  • verify-affected-objects.py
  • verify_candidate_objects.sh

Authentication testing:

  • test_auth.py
  • test_auth_v4.py
  • test_auth.sh

Superseded tools:

  • discover_armor_deployments.py (duplicate of find-armor-deployments.py)
  • check-armor-version-drift.py (duplicate of version-drift-check.py)
  • armor-connectivity-check.sh (replaced by armor check)

These scripts are preserved for reference but are not part of the operational toolkit.