Skip to content

feat: enhance spending policies and archive audit logs - #456

Open
sherifatolanike wants to merge 1 commit into
ASTROIDX556:mainfrom
sherifatolanike:feat/444-445-spending-policy-audit-archival
Open

sherifatolanike wants to merge 1 commit into
ASTROIDX556:mainfrom
sherifatolanike:feat/444-445-spending-policy-audit-archival

Conversation

@sherifatolanike

Copy link
Copy Markdown

Closes #444
Closes #445

Summary

#444 — Multi-asset spending policy validation

The spending-policy layer now evaluates a transaction intent as one or more asset legs. A single-asset intent is the degenerate case with exactly one leg, so the behaviour of existing policies is unchanged; every leg of a multi-asset transfer is evaluated against the policy limits independently.

  • Per-asset limits. A policy may declare assetLimits (keyed by the upper-case asset code) with its own per-transaction and daily limits. An asset without an entry keeps the policy-wide limits. The periodic limits are compared against the spend accumulated for the asset of the leg, not for the primary asset.
  • Asset identity. A new asset registry resolves an asset to its canonical Stellar identifier: the native asset (XLM), a configured Stellar Anchor ticker (STELLAR_ANCHOR_ASSETS, e.g. USDC=GA5Z…) or an explicit CODE:ISSUER. Issued and anchor assets are stored, aggregated and submitted under one stable identity, so a bare ticker and its credit asset share one budget. Only the asset code is matched against policy configuration, so USDC:GA5Z… matches the USDC entry of an allow list or a per-asset limit.
  • Base-currency conversion. A policy may denominate its limits in a base currency (baseCurrency, e.g. USD). The transfer amount — and the spend already accumulated for the asset — are then converted into that currency before the limits are applied. The conversion uses a new price oracle with two sources: an operator-managed static table (PRICE_ORACLE_STATIC_PRICES) and a provider-neutral HTTP endpoint (PRICE_ORACLE_URL). A missing, malformed, unreachable or stale quote aborts the evaluation with PRICE_UNAVAILABLE (HTTP 503) — an unknown price is never treated as zero, so a limit can never be bypassed.
  • Detailed violations. Every violation now carries the asset it applies to, and the response envelope lists the policy, the code, a human-readable message and the limit that was breached. The violation is written to the audit trail as SPENDING_LIMIT_EXCEEDED before the request is rejected.
  • Asset-specific budgets. A budget is denominated in a single currency, so it is now only used for a transfer of that asset, and a disabled budget is never used as headroom. Comparing an XLM amount against a USDC budget used to silently mix units.
  • Rejections never reach the execution queue. A policy or spending-limit violation is raised before the transaction is persisted and before stellar.submitPayment is called (covered by tests).

#445 — Audit log archival background job

A scheduled job moves audit records older than the retention period out of the actively queried audit_logs table into a compressed archive, so the hot table — and every query against it — stays small while the audit history is preserved.

  • Storage strategy: compressed partitions. The repository has no object-storage client, so the archive is one audit_log_archives row per batch holding the gzip-compressed JSON of the archived rows, a SHA-256 checksum, the record count and the hash-chain anchor of the batch. The issue explicitly allows "cold storage or compressed partitions"; this keeps the archive queryable and needs no new infrastructure.
  • Nothing is deleted before it is archived. The order is write → verify → delete: the compressed batch is stored, read back and checked (record count and checksum) before the source rows are removed. A failure in between leaves the records in place.
  • Idempotent. The batch id is the SHA-256 of the archived row ids, so the same set of records always maps to the same archive. A retried run reuses the existing archive (skipDuplicates) instead of writing a second copy, and deleting already-removed rows is a no-op.
  • Bounded work. At most AUDIT_ARCHIVE_BATCH_SIZE records per organization and batch, at most AUDIT_ARCHIVE_MAX_BATCHES batches per run.
  • Chain preserving. The oldest records are archived first, so the archived rows of an organization are always a prefix of its hash chain. The hash of the last archived entry is stored with the batch and anchors the live rows, which keeps GET /audit/integrity/verify meaningful across the archive boundary.
  • Scheduling and retries. A @Cron(CronExpression.EVERY_HOUR) scheduler enqueues the work on the new audit-archive BullMQ queue; the worker uses the shared job options (3 attempts, exponential backoff) and the shared runWorkerJob error handling, so a terminal failure is routed to the dead-letter queue. The job id is derived from the hour, so a duplicated tick never queues the same archival twice.
  • Observability. Structured logs per run and per batch, plus audit_archived_records_total{status="archived"|"deleted"} and the standard worker_jobs_total / worker_job_duration_seconds series.
  • The archive stays readable. GET /audit/archives lists the batches of the organization and GET /audit/archives/:batchId returns the records of one batch after decompressing and verifying its checksum.

Design decisions

  • Multi-asset payloads are validated end-to-end by the policy layer and accepted by the simulation endpoints (POST /transactions/simulate, POST /policies/simulate). POST /transactions still moves one asset per transaction — the Transaction record, the risk assessment, the budget and the proposal model all describe a single asset — so a transfers array there is rejected with an explanatory error instead of being silently ignored. Executing a multi-asset transfer on-chain would require one Transaction row (and one proposal, one risk assessment and one budget settlement) per leg; that is a separate change with cross-repo contract implications and is left out of this PR.
  • The price oracle is new. The repository had no price-conversion utility or oracle integration to reuse, so a small, injectable service was added instead of duplicating one. It is opt-in: without a configured source, conversions fail closed.
  • The archive lives in PostgreSQL. STORAGE_* is configured in the repository but no S3 client is implemented, and adding an object-storage SDK was not possible offline. The compressed-partition approach satisfies the issue and keeps the archival dependency-free.
  • The archival schedule is fixed to hourly. @nestjs/schedule's SchedulerRegistry needs the cron package to register a dynamic expression, and no new dependency could be installed offline. The retention — the value the issue asks to be configurable — is fully configurable.
  • The archive cutoff must stay below the delete retention. AUDIT_ARCHIVE_RETENTION_DAYS (30, default) is validated against AUDIT_RETENTION_DAYS (90, default) at startup, so records are always archived before the cleanup job may delete them.
  • Single-asset behaviour is preserved. The policy-wide limits, the flat spend fields and the engine's evaluation order are unchanged for single-asset intents; the new transfers/spentByAsset fields are additive.

Configuration

New environment variables (validated at startup, documented in docs/configuration.md and .env.example):

Variable Default Purpose
STELLAR_ANCHOR_ASSETS (empty) CODE=ISSUER pairs of the accepted Stellar Anchor tickers
PRICE_ORACLE_ENABLED false Enables the HTTP price source
PRICE_ORACLE_URL (empty) Provider-neutral JSON price endpoint
PRICE_ORACLE_API_KEY (empty) Optional bearer token for the price endpoint
PRICE_ORACLE_TIMEOUT_MS 3000 HTTP timeout of a quote request
PRICE_ORACLE_MAX_AGE_MS 300000 Maximum age of a quote before it is stale
PRICE_ORACLE_STATIC_PRICES (empty) Operator-managed ASSET=PRICE quote table
AUDIT_ARCHIVE_ENABLED true Enables the scheduled archival job
AUDIT_ARCHIVE_RETENTION_DAYS 30 Age at which a record becomes eligible for archival
AUDIT_ARCHIVE_BATCH_SIZE 500 Records per organization and batch
AUDIT_ARCHIVE_MAX_BATCHES 20 Batches per run
AUDIT_RETENTION_DAYS 90 Hard-delete retention of the cleanup job

Tests actually executed

Command Result
npm test ✅ 178 test files, 1988 tests passed
npm run lint ✅ no problems
npm run typecheck ✅ no errors
npm run db:verify ✅ passed (0 warnings) — static structure, SQL validity and schema validation

New/updated suites: src/utils/asset.util.spec.ts, src/config/asset-and-price-config.spec.ts, src/common/filters/problem-details.integration.spec.ts, src/modules/policies/{asset-registry,price-oracle,transfer-payload,policy.service.evaluation}.spec.ts, src/modules/policies/policy.engine.spec.ts, src/modules/transactions/{transaction.dto,tests/spending-limit.service,tests/spending-limit.guard,tests/transaction.service}.spec.ts, src/modules/budgets/budget.service-lock.spec.ts, src/modules/audit/{audit-hash.service,tests/audit-archive.service,tests/audit-archive.scheduler,tests/audit-archive.worker}.spec.ts.

They cover: valid single-asset and multi-asset transfers; native XLM and issued/anchor assets; amounts at, below and above the per-transaction limit; daily limits with accumulated spend; several assets with different limits; missing or inactive budgets; invalid amounts and malformed payloads; price-conversion failures (missing, stale, invalid, unreachable quotes); detailed violation payloads; that a rejected transaction is never persisted and never submitted to the Stellar client; and — for the archival — retention cutoffs, eligible versus recent records, empty batches, batch-size and batch-count limits, successful archival, failures without premature deletion, retries, duplicate execution and idempotency, partial failures with recovery, scheduling with mock timers, and the logging and metrics of a successful and a failed run.

Limitations

  • Multi-asset transfers are validated and simulated, but not executed on-chain (see the design decision above).
  • The database replay and drift check of npm run db:verify was only run in its static mode: no PostgreSQL instance was reachable in the development environment, so SHADOW_DATABASE_URL was not set. The migration DDL was compared against the DDL Prisma generates from the schema (prisma migrate diff) and matches exactly.
  • The existing audit cleanup job still hard-deletes records that were never archived (for example when the archival is disabled); that behaviour is unchanged.

Closes #444
Closes #445

Implements the multi-asset spending policy validation engine (issue ASTROIDX556#444)
and the scheduled audit log archival job (issue ASTROIDX556#445).

Spending policies (ASTROIDX556#444)
- The policy engine now evaluates a transaction intent as one or more asset
  legs. A single-asset intent is the degenerate case, so existing policies
  keep their behaviour; every leg of a multi-asset transfer is evaluated
  against the policy limits independently.
- New per-asset limit overrides (`assetLimits`, keyed by the upper-case asset
  code) let a policy declare different per-transaction and daily limits per
  asset, and the periodic limits are compared against the spend accumulated
  for the asset of the leg.
- New asset registry resolves an asset to its canonical Stellar identifier:
  the native asset, a configured Stellar Anchor ticker (`STELLAR_ANCHOR_ASSETS`)
  or an explicit `CODE:ISSUER`. Issued and anchor assets are stored,
  aggregated and submitted under one stable identity, so a bare ticker and
  its credit asset share one budget.
- New price oracle converts amounts into a base currency when a policy
  denominates its limits in one (`baseCurrency`). A missing, malformed or
  stale quote aborts the evaluation with `PRICE_UNAVAILABLE` instead of
  treating an unknown price as zero, so a limit can never be bypassed.
- Policy violations now carry the asset they apply to and are reported in the
  established problem-details envelope. Budgets are checked against the
  asset of the transfer and an inactive budget is never used as headroom.
- Multi-asset payloads are parsed by the spending policy layer and accepted
  by the simulation endpoints; a transaction still moves a single asset and
  rejects a `transfers` array with an explanatory error.

Audit archival (ASTROIDX556#445)
- New scheduled job moves audit records older than the retention period out
  of the actively queried `audit_logs` table into a compressed archive. The
  archive keeps the full row content, a checksum and the hash-chain anchor
  of the batch, so the history is preserved while the hot table stays small.
- Records are selected in bounded batches, archived per organization and
  only removed from the source table after the compressed batch was written
  and read back. The batch id is derived from the archived row ids, so a
  retried run reuses the existing archive instead of writing a duplicate.
- Because the oldest records are archived first, the archived rows are a
  prefix of the organization hash chain: the stored anchor keeps
  `GET /audit/integrity/verify` meaningful across the archive boundary.
- The archival runs in a BullMQ worker with the shared retry and backoff
  policy, emits structured logs and `audit_archived_records_total`, and the
  archived records stay readable through `GET /audit/archives`.
@drips-wave

drips-wave Bot commented Oct 20, 2026

Copy link
Copy Markdown

@sherifatolanike Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement Audit Log Streaming and Archival Background Job Enhance Spending Policy Validation Engine for Multi-Asset Stellar Transfers

1 participant