Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,31 @@ STELLAR_NETWORK=testnet
STELLAR_HORIZON_URL=https://horizon-testnet.stellar.org
STELLAR_SOROBAN_RPC_URL=https://soroban-testnet.stellar.org
STELLAR_USE_MOCK=true
# Stellar Anchor assets accepted as a bare ticker, as CODE=ISSUER pairs.
# A ticker that is not listed can still be requested explicitly as CODE:ISSUER.
STELLAR_ANCHOR_ASSETS=

# Price oracle — used when a spending policy denominates its limits in a base
# currency (e.g. USD). Without a source, conversions fail closed: the
# transaction is blocked instead of the limit being skipped.
PRICE_ORACLE_ENABLED=false
PRICE_ORACLE_URL=
PRICE_ORACLE_API_KEY=
PRICE_ORACLE_TIMEOUT_MS=3000
PRICE_ORACLE_MAX_AGE_MS=300000
# Static quote table consulted before the HTTP source (ASSET=PRICE,...).
PRICE_ORACLE_STATIC_PRICES=

# Audit retention and archival
# Records older than AUDIT_ARCHIVE_RETENTION_DAYS are moved out of the
# actively queried audit_logs table into the compressed archive. The cleanup
# job hard-deletes records older than AUDIT_RETENTION_DAYS, so the archive
# cutoff must stay below it.
AUDIT_ARCHIVE_ENABLED=true
AUDIT_ARCHIVE_RETENTION_DAYS=30
AUDIT_ARCHIVE_BATCH_SIZE=500
AUDIT_ARCHIVE_MAX_BATCHES=20
AUDIT_RETENTION_DAYS=90

# Storage (S3-compatible)
STORAGE_ENDPOINT=http://localhost:9000
Expand Down
128 changes: 123 additions & 5 deletions API_DOCUMENTATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -188,7 +188,7 @@ Create a transaction (runs the full governance pipeline).
| walletId | string | Yes | Valid UUID | Sender wallet ID |
| agentId | string | No | Valid UUID | Initiating agent ID |
| budgetId | string | No | Valid UUID | Budget to charge against |
| asset | string | No | 1-24 chars | Asset code (default: XLM) |
| asset | string | No | `CODE` or `CODE:ISSUER`, max 80 chars | Asset code (default: XLM). A Stellar Anchor ticker (e.g. `USDC`) is resolved to its issuing account when it is configured in `STELLAR_ANCHOR_ASSETS`; an issued asset can always be requested explicitly as `CODE:ISSUER` |
| amount | string | Yes | Positive decimal, max 7 decimal places | Transaction amount |
| recipientAddress | string | Yes | Non-empty | Stellar destination address |
| memo | string | No | Max 28 chars | Transaction memo |
Expand All @@ -197,14 +197,40 @@ Create a transaction (runs the full governance pipeline).

**Response:** Transaction object with governance evaluation result

A transaction moves a single asset. A payload that carries a `transfers` array
is rejected with HTTP `422` and a message pointing at
`POST /transactions/simulate`, which evaluates a multi-asset transfer against
the spending policies without moving funds.

**Policy violations:** a transaction blocked by a policy is rejected with HTTP
`422` and a `POLICY_VIOLATION` problem-details body that lists every violated
rule, the asset it applies to and the limit that was breached. A blocked
transaction is never persisted and never reaches the blockchain execution
queue.

### POST `/transactions/simulate`
Dry-run the governance pipeline without moving funds.

**Authentication:** Bearer token required

**Request Body:** Same as POST `/transactions`
**Request Body:** Either the single-asset body of `POST /transactions` (without
`metadata` being required) or a multi-asset payload:

**Response:** Simulation result with policy evaluation
| Field | Type | Required | Constraints | Description |
|-------|------|----------|-------------|-------------|
| walletId | string | Yes | Valid UUID | Sender wallet ID |
| agentId | string | No | Valid UUID | Initiating agent ID |
| budgetId | string | No | Valid UUID | Budget to charge against |
| asset | string | No | `CODE` or `CODE:ISSUER` | Asset of a single-asset payload |
| amount | string | No | Positive decimal, max 7 decimal places | Amount of a single-asset payload |
| transfers | array | No | 1-10 legs | Multi-asset payload: one `{ asset, amount }` leg per asset. Mutually exclusive with `asset`/`amount` |
| recipientAddress | string | Yes | Non-empty | Stellar destination address |
| memo | string | No | Max 28 chars | Transaction memo |
| purpose | string | No | Max 280 chars | Transaction purpose |
| metadata | object | No | Arbitrary JSON | Additional metadata |

**Response:** Simulation result with policy evaluation, the evaluated
`transfers` and the risk assessment of the riskiest leg

### GET `/transactions/:id`
Get a specific transaction.
Expand Down Expand Up @@ -270,6 +296,8 @@ Create a policy.
| requiresApproval | boolean | Whether approval is required |
| approvalThreshold | number | Amount threshold for approval |
| emergencyLock | boolean | Emergency lock flag (blocks all spending) |
| assetLimits | object | Per-asset limit overrides keyed by the upper-case asset code (e.g. `{ "USDC": { "maxAmount": 1000, "dailyLimit": 5000 } }`). An issued asset (`USDC:GA5Z…`) matches the `USDC` entry |
| baseCurrency | string | Currency the amount limits are denominated in (e.g. `USD`). Transfer amounts — and the spend already accumulated for the asset — are converted into that currency before the limits are applied. A missing or stale price rejects the transaction with `PRICE_UNAVAILABLE` (HTTP 503) instead of skipping the limit |

**Response:** Created policy object

Expand All @@ -283,8 +311,9 @@ Simulate a transaction intent against active policies.
|-------|------|----------|-------------|-------------|
| agentId | string | No | Valid UUID | Agent ID |
| walletId | string | No | Valid UUID | Wallet ID |
| asset | string | Yes | Non-empty | Asset code |
| amount | number | Yes | Positive | Transaction amount |
| asset | string | No | `CODE` or `CODE:ISSUER` | Asset of a single-asset payload |
| amount | number | No | Positive | Amount of a single-asset payload |
| transfers | array | No | 1-10 legs | Multi-asset payload: one `{ asset, amount }` leg per asset. Mutually exclusive with `asset`/`amount` |
| recipientAddress | string | Yes | Non-empty | Destination address |
| spentToday | number | No | Non-negative | Amount spent today |
| spentThisWeek | number | No | Non-negative | Amount spent this week |
Expand Down Expand Up @@ -391,6 +420,64 @@ The response `meta` includes `limit`, `hasNext`, and `nextCursor` (null on the f

**Example:** `GET /audit?limit=20&actorId=user-123&action=wallet.created`

### Audit archival

Audit records older than `AUDIT_ARCHIVE_RETENTION_DAYS` (default 30) are moved
out of the actively queried `audit_logs` table into a compressed archive
(`audit_log_archives`) by a scheduled background job. The archive keeps the
full row content, a SHA-256 checksum and the hash-chain anchor of the batch,
so the audit history is preserved while the hot table — and every query
against it — stays small.

- The job runs hourly, processes at most `AUDIT_ARCHIVE_BATCH_SIZE` records
(default 500) per organization and batch and at most
`AUDIT_ARCHIVE_MAX_BATCHES` (default 20) batches per run.
- A batch is written and read back (checksum and record count verified)
*before* the source rows are removed. A failure in between leaves the
records in place and the job is retried with the shared queue backoff.
- The batch id is derived from the archived row ids, so a retried run reuses
the existing archive instead of writing a duplicate.
- Because the oldest rows are archived first, the archived rows of an
organization are always a prefix of its hash chain. `GET
/audit/integrity/verify` anchors the live rows on the archived prefix and
keeps reporting a valid chain across the archive boundary.
- The hard-delete retention of the cleanup job (`AUDIT_RETENTION_DAYS`,
default 90) stays above the archive cutoff, so records are always archived
before they are deleted. The configuration is validated at startup.

### GET `/audit/archives`
Lists the archive batches of the organization, newest first. Only the batch
metadata is returned — the compressed payload is fetched one batch at a time.

**Query Parameters:** `limit` defaults to 50 and is bounded to 1-100.

**Example:** `GET /audit/archives?limit=10`

### GET `/audit/archives/{batchId}`
Returns the audit records of one archived batch. The compressed payload is
decompressed and its checksum verified before it is returned, so a corrupted
archive answers with an error instead of silently mangled records. A batch of
another organization is reported as `404`.

**Example:** `GET /audit/archives/8f1c3d…`

### GET `/audit/archives`
List the archive batches of the organization, newest first. Only the batch
metadata is returned (batch id, cutoff, record count, checksum, chain anchor);
the compressed payloads are not loaded.

**Authentication:** Requires roles: OWNER, ADMIN, AUDITOR

**Query Parameters:** `limit` defaults to 50 and is bounded to 1-100.

### GET `/audit/archives/:batchId`
Read the audit records of one archived batch. The payload is decompressed and
its checksum verified before it is returned, so a corrupted archive surfaces as
an error instead of as silently mangled audit records. A batch of another
organization is reported as `404`.

**Authentication:** Requires roles: OWNER, ADMIN, AUDITOR

## Outbound Webhook Signatures

Webhook creation and secret rotation responses disclose the signing secret once. Later list, get, update, delivery, and audit responses never include it. Store the secret securely and rotate it when compromised.
Expand Down Expand Up @@ -523,6 +610,37 @@ All endpoints return errors as [RFC 9457](https://www.rfc-editor.org/rfc/rfc9457

Unhandled server errors always return `500` with `code: "INTERNAL_ERROR"` and a generic `detail`; internal information is only written to the server logs under the `requestId`.

A transaction that is blocked by a spending policy returns `422` with
`code: "POLICY_VIOLATION"`. The `details.violations` array lists every
violated rule with its `policyId`, `policyName`, `code`, `message` and the
`asset` the rule applies to:

```json
{
"type": "urn:astroid:problem:policy-violation",
"title": "Policy Violation",
"status": 422,
"code": "POLICY_VIOLATION",
"detail": "Transaction blocked by spending limit policy: USDC cap: Projected daily spend 600 USDC exceeds limit 500 USDC",
"details": {
"violations": [
{
"policyId": "018f...",
"policyName": "USDC cap",
"code": "DAILY_LIMIT_EXCEEDED",
"message": "Projected daily spend 600 USDC exceeds limit 500 USDC",
"asset": "USDC"
}
]
}
}
```

When a policy denominates its limits in a base currency and no usable price is
available, the transaction is rejected with `503` and
`code: "PRICE_UNAVAILABLE"` instead of being let through with an unknown
price.

### Authentication
Most endpoints require Bearer token authentication in the format:
```
Expand Down
9 changes: 8 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,14 @@ Full OpenAPI spec available at `/docs` when the server is running.

## Data Model

17 Prisma models: `Organization`, `User`, `Agent`, `Wallet`, `Policy`, `Budget`, `Transaction`, `Proposal`, `Approval`, `AuditLog`, `Notification`, `ApiKey`, `Webhook`, `Session`, `MemoryRecord`, `PasskeyCredential`, `DomainEvent`. Schema in [`prisma/`](prisma).
18 Prisma models: `Organization`, `User`, `Agent`, `Wallet`, `Policy`, `Budget`, `Transaction`, `Proposal`, `Approval`, `AuditLog`, `AuditLogArchive`, `Notification`, `ApiKey`, `Webhook`, `Session`, `MemoryRecord`, `PasskeyCredential`, `DomainEvent`. Schema in [`prisma/`](prisma).

## Spending Policies and Audit Archival

- [`docs/spending-policies.md`](docs/spending-policies.md) — multi-asset policy
validation, per-asset limits and base-currency conversion.
- [`docs/audit-archival.md`](docs/audit-archival.md) — the scheduled archival of
expired audit records into the compressed archive.

## Environment Variables

Expand Down
91 changes: 91 additions & 0 deletions docs/audit-archival.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
# Audit Log Archival

Autonomous agents generate audit records at a high rate. The archival job
moves the records that are older than the configured retention period out of
the actively queried `audit_logs` table into a compressed archive, so the hot
table — and every query against it — stays small while the audit history is
preserved.

## Architecture

| Piece | Location | Responsibility |
|-------|----------|----------------|
| Scheduler | `src/modules/audit/audit-archive.scheduler.ts` | Enqueues the archival job once per hour. |
| Worker | `src/modules/audit/queues/audit-archive.worker.ts` | Consumes the job with the shared retry/backoff policy. |
| Service | `src/modules/audit/audit-archive.service.ts` | Selects, archives and removes the expired records. |
| Repository | `src/modules/audit/audit-archive.repository.ts` | Reads the expired records and writes the archive. |

The scheduler only enqueues work; the archival itself runs in the BullMQ
worker, where it inherits the platform-wide retry policy (three attempts with
exponential backoff) and the structured worker logging.

## Storage strategy: compressed partitions

The repository has no object-storage client, so the archive is a **compressed
partition inside PostgreSQL**: one `audit_log_archives` row per batch holding

- the archived rows as a single gzip-compressed JSON document,
- a SHA-256 checksum of the uncompressed payload,
- the number of archived rows,
- the hash-chain anchor of the batch.

This keeps the archive queryable for compliance reviews, needs no new
infrastructure and satisfies the requirement of the issue ("cold storage or
compressed partitions"). The table is only ever appended to, never queried by
the hot path.

## Configuration

| Variable | Default | Meaning |
|----------|---------|---------|
| `AUDIT_ARCHIVE_ENABLED` | `true` | Enables the scheduled job. |
| `AUDIT_ARCHIVE_RETENTION_DAYS` | `30` | Age at which a record becomes eligible for archival. |
| `AUDIT_ARCHIVE_BATCH_SIZE` | `500` | Maximum number of records per organization and batch. |
| `AUDIT_ARCHIVE_MAX_BATCHES` | `20` | Maximum number of batches per run. |
| `AUDIT_RETENTION_DAYS` | `90` | Hard-delete retention of the cleanup job. |

The startup validation rejects an archive cutoff that is not below the
delete retention: records must be archived before the cleanup job may delete
them.

## Safety properties

- **Bounded work.** A run selects at most `AUDIT_ARCHIVE_BATCH_SIZE` records
per batch and processes at most `AUDIT_ARCHIVE_MAX_BATCHES` batches, so the
table is never locked for long and memory usage stays flat.
- **Nothing is deleted before it is archived.** The order is *write → verify →
delete*: the compressed batch is stored, read back and checked (record count
and checksum) before the source rows are removed. A failure in between
leaves the records in place.
- **Idempotent.** The batch id is the SHA-256 of the archived row ids, so the
same set of records always maps to the same archive. A retried run reuses
the existing archive (`skipDuplicates`) instead of writing a second copy,
and deleting already-removed rows is a no-op.
- **Partial failures are safe.** Batches are archived per organization. When
one organization fails, the organizations that were already archived stay
archived and the run is retried; the failed organization keeps its records.
- **Chain preserving.** The oldest records are archived first, so the archived
rows of an organization are always a *prefix* of its hash chain. The hash of
the last archived entry is stored with the batch and anchors the live rows,
which keeps `GET /audit/integrity/verify` meaningful across the archive
boundary.

## Reading the archive

Archived records stay accessible:

- `GET /audit/archives` lists the batches of the organization (metadata only).
- `GET /audit/archives/:batchId` returns the records of one batch. The
compressed payload is decompressed and its checksum verified before it is
returned, so a corrupted archive surfaces as an error instead of as silently
mangled audit records. A batch of another organization is reported as `404`.

## Observability

- Structured logs: one line per run (cutoff, batch size, archived records,
duration) and one line per archived batch.
- Metrics: `audit_archived_records_total{status="archived"|"deleted"}` counts
the processed records, and the worker instrumentation adds
`worker_jobs_total` and `worker_job_duration_seconds` for the queue.
- A failed run is rethrown so BullMQ retries it; a terminal failure is routed
to the dead-letter queue by the shared queue failure listener.
Loading
Loading