Skip to content

Asynchronous / scalable EvolutionEngine #53

Description

@Teingi

Description

EN. The EvolutionEngine runs incrementally after writes or on compact(),
which is fine for small stores but stalls at scale. Re-architect it to run
asynchronously: a durable work queue of evolution tasks (merge, resolve, stage
advance, distill), idempotent workers, backpressure, and progress/metrics via
the observability layer (src/contextseek/observability/). Writes should enqueue
work and return fast; evolution happens in the background.

Tasks: design the task model + queue (pluggable: in-process, then a real
broker); make each evolution step idempotent & retry-safe; expose metrics and a
CLI/HTTP status view; ensure compact() can still run synchronously for tests;
load-test and document.

Acceptance: writes are not blocked by evolution; background processing is
observable, idempotent, and recoverable after a crash.

ZH. EvolutionEngine 现在在写后或 compact() 时增量运行,小库没问题,规模上来就
会卡。请将其重构为异步运行:一个持久化的进化任务队列(合并、消解、阶段晋升、蒸馏)、
幂等 worker、背压、以及经可观测层(src/contextseek/observability/)暴露进度/指标。
写入只入队并快速返回,进化在后台进行。

任务: 设计任务模型 + 队列(可插拔:先进程内,再接真正的 broker);让每个进化步骤
幂等、可重试;暴露指标和 CLI/HTTP 状态视图;保证 compact() 仍能同步运行以便测试;
做压测并文档化。

验收: 写入不被进化阻塞;后台处理可观测、幂等、崩溃后可恢复。

Metadata

Metadata

Assignees

No one assigned

    Labels

    comp:evolutionContextSeek component: evolutionhelp wantedExtra attention is neededpriority:lowNice to havestatus:needs-designDesign or decision required before implementation

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions