feat: deterministic sorting, failure fixtures, safe ownership transfer, and rollback verification

README.md

@@ -410,6 +410,7 @@ Every markdown document in this repository, grouped by topic:
 - [docs/kubernetes-health-probes.md](docs/kubernetes-health-probes.md) - Liveness, readiness, and startup probe configuration for Kubernetes
 - [docs/operational-rollout-guardrails.md](docs/operational-rollout-guardrails.md) - Operational rollout guardrails, bounded caching, concurrency controls, and privacy-safe data export
 - [docs/OPERATIONAL_RESILIENCE_AND_REMEDIATION.md](docs/OPERATIONAL_RESILIENCE_AND_REMEDIATION.md) - Operational resilience, permission diffs, error remediation hints, fuzz guarantees, and synthetic journey monitoring
+- [docs/MIGRATION_ROLLBACK_AND_DATA_RELIABILITY.md](docs/MIGRATION_ROLLBACK_AND_DATA_RELIABILITY.md) - Post-rollback verification, deterministic mixed-status sorting, failure fixtures, and safe ownership transfer
 - [docs/BOUNDED_QUEUE_BACKPRESSURE.md](docs/BOUNDED_QUEUE_BACKPRESSURE.md) - Bounded queue with 429 backpressure, per-actor fairness and retries for expensive background work
 - [docs/DETERMINISTIC_REPLAY.md](docs/DETERMINISTIC_REPLAY.md) - Deterministic schedule replay and exhaustive interleaving tests for concurrent writes and retries
 - [docs/EVIDENCE_VERIFICATION.md](docs/EVIDENCE_VERIFICATION.md) - Ed25519 verification, key trust and replay protection for imported external evidence

docs/MIGRATION_ROLLBACK_AND_DATA_RELIABILITY.md

@@ -0,0 +1,96 @@
+# Migration Rollback, Deterministic Ordering & Data Reliability Architecture
+
+This document specifies the operational guardrails, deterministic event ordering, rare failure fixture generators, safe ownership transfers, and post-rollback verification protocols implemented in Trellis-API.
+
+---
+
+## 1. Deterministic Sorting for Mixed-Status Records and Events (Issue #286)
+
+### Purpose
+To eliminate pagination drift, duplicate records across page boundaries, and non-deterministic results when sorting mixed-status events or batch-arriving records.
+
+### Multi-Tiered Comparator Hierarchy
+`DeterministicSorter` applies a 4-tier sorting evaluation:
+1. **Status Priority Hierarchy**: Active/urgent statuses (`CRITICAL` -> `RETRYING` -> `PROCESSING` -> `PENDING`) are prioritized before terminal statuses (`COMPLETED` -> `FAILED` -> `CANCELLED` -> `EXPIRED`). Custom maps are supported.
+2. **Timestamp / Sequence**: Normalizes ISO-8601 strings, millisecond Epoch values, or monotonic sequences.
+3. **Deterministic Tie-Breaker**: Lexicographical comparison of unique identifiers (`id` / UUID).
+4. **Opaque Cursors**: Encodes compound sort states into URL-safe base64 cursors (`encodeCursor` / `decodeCursor`) for drift-free cursor pagination.
+
+### Usage Example
+```typescript
+import { DeterministicSorter } from "src/common/sorting";
+
+const paginated = DeterministicSorter.paginate(events, {
+  limit: 20,
+  cursor: req.query.cursor,
+  options: {
+    sortFields: [
+      { field: "status", isStatus: true, direction: "ASC" },
+      { field: "createdAt", direction: "DESC" },
+    ],
+    tieBreakerField: "id",
+  },
+});
+```
+
+---
+
+## 2. Fixture Generators for Rare Failure & Recovery Paths (Issue #287)
+
+### Purpose
+Provides reproducible test fixtures and simulation seeds for rare edge cases that are difficult to trigger in staging environments:
+
+- **Ledger Reorganization (`generateLedgerReorg`)**: Simulates deep chain re-orgs with orphaned transactions and diverging block hashes.
+- **Nonce / Sequence Collision (`generateNonceCollision`)**: Simulates concurrent worker race conditions on account sequence numbers resulting in `txBAD_SEQ`.
+- **Malformed XDR Envelopes (`generateMalformedXdr`)**: Generates truncated payloads, invalid base64, mismatched network passphrases, and corrupted signatures.
+- **Partial DB Commits & Compensations (`generatePartialCommit`)**: Simulates deadlocks in multi-table operations with rollback compensation instructions.
+- **Dead Letter Queue Spills (`generateDeadLetterSpill`)**: Generates payload records after maximum retry exhaustion with DLQ routing keys.
+- **Tampered Audit Trails (`generateTamperedAuditChain`)**: Simulates cryptographic hash chain corruption for tamper-evident ledger testing.
+
+---
+
+## 3. Safe Ownership Transfer with Pending Acceptance & Expiry (Issue #288)
+
+### Problem & Design
+Instantaneous ownership reassignment poses a catastrophic risk of permanent asset loss due to typos in destination account identifiers. Trellis-API implements an atomic two-step transfer pattern:
+
+```
+[Current Owner]  --( initiateTransfer )-->  PENDING (with 72h expiry)
+                                                |
+            +-----------------------------------+---------------------------------+
+            |                                   |                                 |
+ [Proposed Owner Accepts]            [Proposed Owner Rejects]          [Current Owner Cancels]
+            v                                   v                                 v
+        ACCEPTED                            REJECTED                          CANCELLED
+ (Ownership transferred)              (Remains with current)           (Remains with current)
+```
+
+- **Expiration**: Requests past `expiresAt` automatically transition to `EXPIRED` and cannot be accepted.
+- **Self-Transfer Prevention**: Throws `BadRequestException` if current owner matches proposed owner.
+- **Duplicate Prevention**: Rejects opening multiple pending transfers for the same resource simultaneously.
+
+### API Endpoints
+- `POST /ownership/transfers`: Initiates transfer with resource type, ID, proposed owner, and validity window.
+- `POST /ownership/transfers/:id/accept`: Proposed owner accepts transfer.
+- `POST /ownership/transfers/:id/reject`: Proposed owner declines transfer.
+- `DELETE /ownership/transfers/:id`: Current owner revokes pending transfer.
+- `GET /ownership/transfers`: Lists pending incoming/outgoing transfers.
+
+---
+
+## 4. Rollback Verification After Failed Migrations or Deployments (Issue #289)
+
+### Purpose
+Executing a database rollback without verification risks silent data corruption, orphaned foreign keys, hanging migration locks, or broken application entities. `RollbackVerificationService` runs a comprehensive post-rollback health and invariant check.
+
+### Automated Verification Pipeline
+1. **Migration Lock Check**: Ensures `migrations_lock` table is released; detects hanging lock processes.
+2. **Schema Table Invariants**: Confirms all 9 core tables (`users`, `portfolios`, `api_keys`, `wallets`, `reconciliation_invoices`, etc.) are present and match baseline schemas.
+3. **Referential Integrity**: Checks for orphaned records violating foreign key constraints.
+4. **Data Continuity**: Validates row counts and sequence ranges against pre-migration snapshots.
+5. **Database Smoke Queries**: Executes active read/write smoke queries to ensure connection pool and entity mappings are functional.
+
+### Rollback Verdicts & Runbook
+- `VERIFIED`: All checks passed. Safe to route live production traffic (`canResumeTraffic: true`).
+- `WARNING`: Minor non-fatal discrepancies (e.g. minor orphaned child records). Traffic allowed with alerts.
+- `FAILED`: Fatal schema missing or lock acquired. Traffic is blocked (`canResumeTraffic: false`), and exact remediation steps are generated.

src/app.module.ts

@@ -26,6 +26,8 @@ import { AuthModule } from "./core/auth/auth.module";
 import { UserModule } from "./core/user/user.module";
 import { ProfileModule } from "./core/profile/profile.module";
 import { InvitationModule } from "./core/invitation/invitation.module";
+import { OwnershipTransferModule } from "./core/ownership/ownership-transfer.module";
+import { OwnershipTransfer } from "./core/ownership/entities/ownership-transfer.entity";
 
 // Modules – infrastructure
 import { AuditModule } from "./infrastructure/audit/audit.module";
@@ -327,6 +329,7 @@ import { ReleaseModule } from "./common/release/release.module";
             InvariantReportEntity,
             SearchRecord,
             Invitation,
+            OwnershipTransfer,
             RetryOperation,
             PauseScope,
             PauseAuditLog,
@@ -357,6 +360,7 @@ import { ReleaseModule } from "./common/release/release.module";
     UserModule,
     ProfileModule,
     InvitationModule,
+    OwnershipTransferModule,
     AuditModule,
     ApprovalIntentModule,
     OracleModule,

src/common/fixtures/failure-fixture-generator.spec.ts

@@ -0,0 +1,77 @@
+import { FailureFixtureGenerator } from "./failure-fixture-generator";
+
+describe("FailureFixtureGenerator", () => {
+  describe("Stellar Blockchain Fixtures", () => {
+    it("should generate a ledger reorganization fixture with orphaned transactions", () => {
+      const fixture = FailureFixtureGenerator.generateLedgerReorg(50000000, 4);
+
+      expect(fixture.originalLedgerSequence).toBe(50000000);
+      expect(fixture.reorgDepth).toBe(4);
+      expect(fixture.forkedLedgerSequence).toBe(49999997);
+      expect(fixture.orphanedTransactions).toHaveLength(4);
+      expect(fixture.originalBlockHash).toMatch(/^[a-f0-9]{64}$/);
+      expect(fixture.newBlockHash).toMatch(/^[a-f0-9]{64}$/);
+      expect(fixture.originalBlockHash).not.toBe(fixture.newBlockHash);
+    });
+
+    it("should generate a sequence/nonce collision fixture", () => {
+      const fixture = FailureFixtureGenerator.generateNonceCollision(
+        "GACC12345",
+        "99887766",
+      );
+
+      expect(fixture.accountId).toBe("GACC12345");
+      expect(fixture.conflictingSequence).toBe("99887766");
+      expect(fixture.transactions).toHaveLength(2);
+      expect(fixture.transactions[0].status).toBe("FIRST_ACCEPTED");
+      expect(fixture.transactions[1].status).toBe("COLLIDED_BAD_SEQ");
+    });
+
+    it("should generate diverse malformed XDR scenarios", () => {
+      const truncated = FailureFixtureGenerator.generateMalformedXdr("TRUNCATED_XDR");
+      expect(truncated.scenario).toBe("TRUNCATED_XDR");
+      expect(truncated.expectedErrorCode).toBe("INVALID_TRANSACTION_ENVELOPE");
+
+      const invalidBase64 = FailureFixtureGenerator.generateMalformedXdr("INVALID_BASE64");
+      expect(invalidBase64.remediationHint).toContain("base64");
+
+      const passphrase = FailureFixtureGenerator.generateMalformedXdr("MISMATCHED_NETWORK_PASSPHRASE");
+      expect(passphrase.expectedErrorCode).toBe("STELLAR_NETWORK_ERROR");
+
+      const sig = FailureFixtureGenerator.generateMalformedXdr("CORRUPTED_SIGNATURE");
+      expect(sig.expectedErrorCode).toBe("ORACLE_VERIFICATION_FAILED");
+    });
+  });
+
+  describe("Persistence & Distributed Failure Fixtures", () => {
+    it("should generate partial commit fixture with compensation actions", () => {
+      const fixture = FailureFixtureGenerator.generatePartialCommit("tx-deadlock-99");
+
+      expect(fixture.transactionId).toBe("tx-deadlock-99");
+      expect(fixture.committedEntities).toHaveLength(1);
+      expect(fixture.failedEntities).toHaveLength(1);
+      expect(fixture.requiresCompensation).toBe(true);
+      expect(fixture.compensationInstructions.length).toBeGreaterThan(0);
+    });
+
+    it("should generate dead letter queue spill fixture", () => {
+      const fixture = FailureFixtureGenerator.generateDeadLetterSpill("webhook.payment", 5);
+
+      expect(fixture.topic).toBe("webhook.payment");
+      expect(fixture.attemptsMade).toBe(5);
+      expect(fixture.dlqRoutingKey).toBe("dlq.webhook.payment");
+      expect(fixture.lastFailureReason).toContain("503");
+    });
+
+    it("should generate tampered audit trail fixture detecting broken hash chain", () => {
+      const fixture = FailureFixtureGenerator.generateTamperedAuditChain("chain-audit-2026");
+
+      expect(fixture.chainId).toBe("chain-audit-2026");
+      expect(fixture.tamperedEntryIndex).toBe(17);
+      expect(fixture.originalHash).not.toBe(fixture.computedHash);
+      expect(fixture.tamperedPayload.tamperedAmount).toBeGreaterThan(
+        Number(fixture.tamperedPayload.originalAmount),
+      );
+    });
+  });
+});