Repository navigation
feat: enhance spending policies and archive audit logs - #456
Open
sherifatolanike wants to merge 1 commit into
Open
sherifatolanike wants to merge 1 commit into
sherifatolanike wants to merge 1 commit into
Conversation
Implements the multi-asset spending policy validation engine (issue ASTROIDX556#444) and the scheduled audit log archival job (issue ASTROIDX556#445). Spending policies (ASTROIDX556#444) - The policy engine now evaluates a transaction intent as one or more asset legs. A single-asset intent is the degenerate case, so existing policies keep their behaviour; every leg of a multi-asset transfer is evaluated against the policy limits independently. - New per-asset limit overrides (`assetLimits`, keyed by the upper-case asset code) let a policy declare different per-transaction and daily limits per asset, and the periodic limits are compared against the spend accumulated for the asset of the leg. - New asset registry resolves an asset to its canonical Stellar identifier: the native asset, a configured Stellar Anchor ticker (`STELLAR_ANCHOR_ASSETS`) or an explicit `CODE:ISSUER`. Issued and anchor assets are stored, aggregated and submitted under one stable identity, so a bare ticker and its credit asset share one budget. - New price oracle converts amounts into a base currency when a policy denominates its limits in one (`baseCurrency`). A missing, malformed or stale quote aborts the evaluation with `PRICE_UNAVAILABLE` instead of treating an unknown price as zero, so a limit can never be bypassed. - Policy violations now carry the asset they apply to and are reported in the established problem-details envelope. Budgets are checked against the asset of the transfer and an inactive budget is never used as headroom. - Multi-asset payloads are parsed by the spending policy layer and accepted by the simulation endpoints; a transaction still moves a single asset and rejects a `transfers` array with an explanatory error. Audit archival (ASTROIDX556#445) - New scheduled job moves audit records older than the retention period out of the actively queried `audit_logs` table into a compressed archive. The archive keeps the full row content, a checksum and the hash-chain anchor of the batch, so the history is preserved while the hot table stays small. - Records are selected in bounded batches, archived per organization and only removed from the source table after the compressed batch was written and read back. The batch id is derived from the archived row ids, so a retried run reuses the existing archive instead of writing a duplicate. - Because the oldest records are archived first, the archived rows are a prefix of the organization hash chain: the stored anchor keeps `GET /audit/integrity/verify` meaningful across the archive boundary. - The archival runs in a BullMQ worker with the shared retry and backoff policy, emits structured logs and `audit_archived_records_total`, and the archived records stay readable through `GET /audit/archives`.
|
@sherifatolanike Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits. You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #444
Closes #445
Summary
#444 — Multi-asset spending policy validation
The spending-policy layer now evaluates a transaction intent as one or more asset legs. A single-asset intent is the degenerate case with exactly one leg, so the behaviour of existing policies is unchanged; every leg of a multi-asset transfer is evaluated against the policy limits independently.
assetLimits(keyed by the upper-case asset code) with its own per-transaction and daily limits. An asset without an entry keeps the policy-wide limits. The periodic limits are compared against the spend accumulated for the asset of the leg, not for the primary asset.XLM), a configured Stellar Anchor ticker (STELLAR_ANCHOR_ASSETS, e.g.USDC=GA5Z…) or an explicitCODE:ISSUER. Issued and anchor assets are stored, aggregated and submitted under one stable identity, so a bare ticker and its credit asset share one budget. Only the asset code is matched against policy configuration, soUSDC:GA5Z…matches theUSDCentry of an allow list or a per-asset limit.baseCurrency, e.g.USD). The transfer amount — and the spend already accumulated for the asset — are then converted into that currency before the limits are applied. The conversion uses a new price oracle with two sources: an operator-managed static table (PRICE_ORACLE_STATIC_PRICES) and a provider-neutral HTTP endpoint (PRICE_ORACLE_URL). A missing, malformed, unreachable or stale quote aborts the evaluation withPRICE_UNAVAILABLE(HTTP 503) — an unknown price is never treated as zero, so a limit can never be bypassed.SPENDING_LIMIT_EXCEEDEDbefore the request is rejected.stellar.submitPaymentis called (covered by tests).#445 — Audit log archival background job
A scheduled job moves audit records older than the retention period out of the actively queried
audit_logstable into a compressed archive, so the hot table — and every query against it — stays small while the audit history is preserved.audit_log_archivesrow per batch holding the gzip-compressed JSON of the archived rows, a SHA-256 checksum, the record count and the hash-chain anchor of the batch. The issue explicitly allows "cold storage or compressed partitions"; this keeps the archive queryable and needs no new infrastructure.skipDuplicates) instead of writing a second copy, and deleting already-removed rows is a no-op.AUDIT_ARCHIVE_BATCH_SIZErecords per organization and batch, at mostAUDIT_ARCHIVE_MAX_BATCHESbatches per run.GET /audit/integrity/verifymeaningful across the archive boundary.@Cron(CronExpression.EVERY_HOUR)scheduler enqueues the work on the newaudit-archiveBullMQ queue; the worker uses the shared job options (3 attempts, exponential backoff) and the sharedrunWorkerJoberror handling, so a terminal failure is routed to the dead-letter queue. The job id is derived from the hour, so a duplicated tick never queues the same archival twice.audit_archived_records_total{status="archived"|"deleted"}and the standardworker_jobs_total/worker_job_duration_secondsseries.GET /audit/archiveslists the batches of the organization andGET /audit/archives/:batchIdreturns the records of one batch after decompressing and verifying its checksum.Design decisions
POST /transactions/simulate,POST /policies/simulate).POST /transactionsstill moves one asset per transaction — theTransactionrecord, the risk assessment, the budget and the proposal model all describe a single asset — so atransfersarray there is rejected with an explanatory error instead of being silently ignored. Executing a multi-asset transfer on-chain would require oneTransactionrow (and one proposal, one risk assessment and one budget settlement) per leg; that is a separate change with cross-repo contract implications and is left out of this PR.STORAGE_*is configured in the repository but no S3 client is implemented, and adding an object-storage SDK was not possible offline. The compressed-partition approach satisfies the issue and keeps the archival dependency-free.@nestjs/schedule'sSchedulerRegistryneeds thecronpackage to register a dynamic expression, and no new dependency could be installed offline. The retention — the value the issue asks to be configurable — is fully configurable.AUDIT_ARCHIVE_RETENTION_DAYS(30, default) is validated againstAUDIT_RETENTION_DAYS(90, default) at startup, so records are always archived before the cleanup job may delete them.transfers/spentByAssetfields are additive.Configuration
New environment variables (validated at startup, documented in
docs/configuration.mdand.env.example):STELLAR_ANCHOR_ASSETSCODE=ISSUERpairs of the accepted Stellar Anchor tickersPRICE_ORACLE_ENABLEDfalsePRICE_ORACLE_URLPRICE_ORACLE_API_KEYPRICE_ORACLE_TIMEOUT_MS3000PRICE_ORACLE_MAX_AGE_MS300000PRICE_ORACLE_STATIC_PRICESASSET=PRICEquote tableAUDIT_ARCHIVE_ENABLEDtrueAUDIT_ARCHIVE_RETENTION_DAYS30AUDIT_ARCHIVE_BATCH_SIZE500AUDIT_ARCHIVE_MAX_BATCHES20AUDIT_RETENTION_DAYS90Tests actually executed
npm testnpm run lintnpm run typechecknpm run db:verifyNew/updated suites:
src/utils/asset.util.spec.ts,src/config/asset-and-price-config.spec.ts,src/common/filters/problem-details.integration.spec.ts,src/modules/policies/{asset-registry,price-oracle,transfer-payload,policy.service.evaluation}.spec.ts,src/modules/policies/policy.engine.spec.ts,src/modules/transactions/{transaction.dto,tests/spending-limit.service,tests/spending-limit.guard,tests/transaction.service}.spec.ts,src/modules/budgets/budget.service-lock.spec.ts,src/modules/audit/{audit-hash.service,tests/audit-archive.service,tests/audit-archive.scheduler,tests/audit-archive.worker}.spec.ts.They cover: valid single-asset and multi-asset transfers; native XLM and issued/anchor assets; amounts at, below and above the per-transaction limit; daily limits with accumulated spend; several assets with different limits; missing or inactive budgets; invalid amounts and malformed payloads; price-conversion failures (missing, stale, invalid, unreachable quotes); detailed violation payloads; that a rejected transaction is never persisted and never submitted to the Stellar client; and — for the archival — retention cutoffs, eligible versus recent records, empty batches, batch-size and batch-count limits, successful archival, failures without premature deletion, retries, duplicate execution and idempotency, partial failures with recovery, scheduling with mock timers, and the logging and metrics of a successful and a failed run.
Limitations
npm run db:verifywas only run in its static mode: no PostgreSQL instance was reachable in the development environment, soSHADOW_DATABASE_URLwas not set. The migration DDL was compared against the DDL Prisma generates from the schema (prisma migrate diff) and matches exactly.Closes #444
Closes #445