Payments 8/8: Reconciliation, observability, and a demonstrable end-to-end testnet payment flow #145

Problem / Context

There is currently no tooling that cross-checks on-chain reality against expected payment outcomes: no event indexing, no reconciliation job comparing tier-vault Balance/Shares totals against StrategyVault allocations and Harvester bounty payouts, and no operator-facing visibility into stuck or anomalous transactions. Nothing is deployed to testnet or mainnet yet — deployments/testnet.json and deployments/mainnet.json are both placeholder files ("CC_PENDING_TESTNET" / empty strings, "deployedAt": "pending" / null). This issue is the capstone of the payment track: once #138–#144 land, the protocol should have a real, provable, end-to-end payment path — and a way to verify it keeps working over time.

Objective

Deploy the full payment stack to testnet, prove a complete deposit -> status-tracked confirmation -> yield accrual -> withdraw/early-exit loop works via the real SDK and app, and stand up reconciliation + observability so drift is caught automatically rather than discovered by users.

Technical approach

• Deploy VaultRouter, all four tier vaults, StrategyVault, Harvester, Governance, and GuardianMultisig to Soroban testnet; populate deployments/testnet.json with real contract IDs, replacing the CC_PENDING_TESTNET placeholders. Add a repeatable deploy script if one doesn't already exist.
• Build a reconciliation job (script or small scheduled service) that: sums each tier vault's per-user Balance/Shares against its TotalBalance/TotalShares invariants; verifies Harvester's bounty-plus-remainder split against the harvested amount (the invariant is already unit-tested in isolation — contracts/harvester/src/lib.rs lines 160-179 — but never checked against live chain state); and flags any mismatch between a user's SDK-reported position (Payments 4/8: Real transaction status tracking and confirmation lifecycle #141) and the same position read directly from contract storage.
• Add structured event emission for deposit, withdraw, early_exit, and harvest sufficient to drive an indexer/observability pipeline — extending the existing env.events().publish pattern, which today only covers pause/unpause in VaultRouter.
• Wire a minimal operational report (reusing app/src/app/analytics or a new script) surfacing: last harvest ledger vs. expected cooldown-elapsed ledger, current TVL vs. cap per tier, and any reconciliation mismatches from above.
• Write and run an end-to-end demo using the Payments 2/8: Implement real Soroban transaction pipeline in the TypeScript SDK #139 SDK against the deployed testnet contracts: connect a real Freighter testnet wallet (Payments 6/8: Wallet onboarding and non-custodial payment UX for non-crypto-native users #143), deposit into at least two tiers, observe real status transitions (Payments 4/8: Real transaction status tracking and confirmation lifecycle #141) through to confirmation, trigger a harvest, verify yield accrual, then withdraw/early-exit and confirm final balances reconcile with zero discrepancy.

Detailed scope

deployments/testnet.json (and a deploy script, added if none exists), a new reconciliation script/service, event-emission additions across vault_flex/vault_l3/vault_l6/vault_l12/harvester, a demo script under a new scripts/ or e2e/ directory, and documentation of the demo as a reproducible walkthrough.

Important edge cases and failure scenarios

• Reconciliation must handle the emergency-unlock and paused states without false-flagging expected deviations (e.g., a paused protocol legitimately has zero new deposits during that window).
• Stellar testnet is periodically reset — deployment and reconciliation tooling must be easily re-runnable, not a one-shot manual process that breaks on the next reset.
• Harvester bounty-split reconciliation must correctly treat the documented zero-yield path (harvested == 0, contracts/harvester/src/lib.rs lines 80-87) as expected, not as a discrepancy.
• Multi-asset positions (post-Payments 1/8: Fix VaultRouter <-> tier-vault ABI mismatch and unify asset-scoped accounting #138) must reconcile per-asset, not only in aggregate — a shortfall in one asset could otherwise be masked by a surplus in another.

Dependencies

Depends on #138, #139, #140, #141, #142, #143, #144 — this issue is the integration point that proves the whole payment track composes correctly, and cannot meaningfully start until the contract interfaces, SDK pipeline, idempotency, status tracking, failure handling, wallet UX, and authorization hardening from all prior issues exist.

Acceptance criteria

• deployments/testnet.json contains real, live contract addresses.
• A scripted, documented end-to-end run demonstrates: connect wallet -> deposit (2+ tiers) -> confirmed status observed via polling -> harvest triggers yield accrual -> withdraw/early-exit -> final on-chain balances match both SDK-reported and reconciliation-reported values with zero discrepancy.
• The reconciliation job runs (manually or on a schedule) and correctly reports zero mismatches on a clean run, and correctly flags an intentionally-injected mismatch in a test.
• Basic observability (events plus a report/dashboard) lets an operator answer "is the payment system healthy right now?" without reading contract storage by hand.

Definition of done

PR(s) merged; testnet deployment live and documented in the README/deployments/; the end-to-end demo is reproducible by a third party following committed instructions; reconciliation tooling is checked in, runnable, and has its own test coverage for the mismatch-detection logic.