List endpoints return unbounded collections from in-memory stores with no pagination #229

Priority: Medium  ·  Area: API design / resource limits  ·  Est. effort: 7–10 h

📌 Problem

The service exposes collection endpoints across src/routes/anchors.ts, liquidity.ts and settlements.ts, backed by repositories that hold every record in memory (src/repositories/inMemoryRepository.ts).

Establish and report the current behaviour first: for each list endpoint, does it accept any limit or cursor, or does it serialise the entire collection on every request?

If collections are returned whole, three problems follow:

1. Response size grows without bound. Settlements are append-only; the list only ever gets longer, so response time and memory degrade continuously with no ceiling.
2. Adding pagination later is breaking. Clients written against an unpaginated endpoint assume the response is complete. Introducing pagination after adoption silently truncates for anyone who did not update — the worst failure mode, because it looks like data loss rather than an error.
3. Amplification. GET is unauthenticated and unrate-limited (both tracked separately), so an unbounded list endpoint is a cheap way to make the server do a lot of work.

Doing this now, while the API is young, is far cheaper than after clients depend on the current shape.

🎯 Design decision required

State and defend:

1. Offset or cursor pagination? Offset is simpler; cursor is stable under concurrent inserts — and settlements are append-only, which is exactly where offset pagination skips or repeats rows. Argue one against this data.
2. Default and maximum page size. Justify both numbers. A default that returns everything is not a default.
3. Response envelope. Changing the shape from a bare array to { items, next } is breaking. Decide whether to break now, version, or add pagination while keeping the current shape for unparameterised requests, and argue it.
4. Ordering. Pagination requires a stable sort. State the ordering per collection and why it is deterministic.

🧩 Requirements and context

• The maximum page size must be enforced server-side — a client asking for more must be clamped, not obeyed. Test it.
• Ordering must be deterministic, including for records that would otherwise tie. Test that too.
• src/openapi.ts must document the parameters and the response shape.
• Coordinate with the persistence issue: pagination implemented against in-memory arrays should map cleanly onto database queries later. Say how your design does.
• All 42 test files must pass.

🛠️ Suggested execution

1. Report each list endpoint's current behaviour.
2. Post your pagination model and envelope decision on this issue; wait for agreement if it is breaking.
3. Implement, with server-side clamping.
4. Add tests: clamping, stable ordering, traversal to the end without skips or duplicates.
5. Update the OpenAPI spec.

✅ Acceptance criteria

• The PR reports current behaviour for every list endpoint.
• Pagination is implemented with documented default and maximum page sizes.
• Requests exceeding the maximum are clamped server-side, with a test.
• Ordering is deterministic, with a test covering ties.
• A test traverses a full collection and asserts no skipped or duplicated records.
• src/openapi.ts documents parameters and response shape.
• Any breaking change is stated with a migration note.
• npm run lint, npm run build and npm test pass.

🚫 Out of scope

• The persistence layer — separate issue.
• Authentication and rate limiting on reads — separate issues.
• Adding filtering or search.

🧪 Verification

npm ci
npm test src/routes src/repositories
npm run lint && npm run build && npm test

📤 What your PR must include

1. Current behaviour per list endpoint.
2. Your pagination model, page sizes and envelope decision with reasoning.
3. The traversal test proving no skips or duplicates.
4. How the design maps onto a future database query.
5. Closes #<n>.

🔒 Security notes

An unbounded list endpoint that requires no authentication and is not rate limited is a resource-exhaustion vector: each request forces the server to serialise the entire collection, and the cost to the attacker is one HTTP request. Because settlements are append-only, the asymmetry worsens over time — the same request gets more expensive for the server and no more expensive for the caller.

📋 Guidelines

• Minimum 95% test coverage on changed lines
• Clear documentation
• Timeframe: 96 hours from assignment
• One logical change per commit; no merge commits

💬 Join our community

Working on this, or want to sanity-check your approach before you start? Come and ask — the maintainers are there and happy to help.

Telegram: https://t.me/Grainlify