From 61199414580f4c78d6bc8fb79aecbbd367cd85d3 Mon Sep 17 00:00:00 2001 From: Chojan Shang Date: Thu, 27 Aug 2026 16:35:14 +0800 Subject: [PATCH 1/9] docs: define source observation contract --- ...source_definition_and_observation_model.md | 471 ++++++++++++++++++ ...source_definition_and_observation_model.md | 444 +++++++++++++++++ zensical.toml | 2 + 3 files changed, 917 insertions(+) create mode 100644 docs/en/rfcs/0000_source_definition_and_observation_model.md create mode 100644 docs/zh/rfcs/0000_source_definition_and_observation_model.md diff --git a/docs/en/rfcs/0000_source_definition_and_observation_model.md b/docs/en/rfcs/0000_source_definition_and_observation_model.md new file mode 100644 index 000000000..54f7ed7d2 --- /dev/null +++ b/docs/en/rfcs/0000_source_definition_and_observation_model.md @@ -0,0 +1,471 @@ +- Proposal Name: `source_definition_and_observation_model` +- Start Date: 2026-08-27 +- Related Discussion: [oceanbase/powercontext#1240](https://github.com/oceanbase/powercontext/issues/1240), + [oceanbase/powercontext#1363](https://github.com/oceanbase/powercontext/issues/1363) +- Related Design: [oceanbase/powercontext#1345](https://github.com/oceanbase/powercontext/pull/1345) +- Related RFCs: [RFC 0002](0002_core_sdk_product_model.md), [RFC 0014](0014_memory_layer_design.md), + [RFC 0019](0019_local_source_memory_runtime.md), [RFC 0048](0048_handoff_artifact.md) + +# Summary + +This RFC defines the standard Source model and the contract for defining additional Source types. + +A Source belongs to exactly one Scope. Within that Scope, a `SourceKey` identifies one logical source and a +`SourceRef` identifies one immutable observation of that source. Advancing the current observation, observing a +deletion, changing an external locator, or disconnecting a Connector does not alter an earlier observation or move +it to another Scope. + +A Source Definition gives one stable Source type its value schema, provenance schema, identity rules, observation +rules, materialization contract, canonicalization, and compatibility policy. Definitions are registered explicitly +and remain fixed for the lifetime of a composed Runtime. Persistence, transport, and Artifact consumers route by the +stable definition name and version rather than by a concrete Python class. + +Materialization identifies the authority used to resolve an exact observation. A captured observation is resolved +from the canonical value retained by PowerContext. A referenced observation is resolved from an immutable external +revision. An external locator, modification time, ETag, or current-provider read does not by itself satisfy the +referenced contract. + +`ContentSource` remains a simple captured-text Source. Its caller-stable identity and immutable-payload conflict rule +make it useful for one-shot content capture, but it is not the general external integration model. A document Source +serves as the first conformance validation of the standard definition contract, not the definition of that contract. + +This RFC defines semantics and conformance. It does not define a Connector runtime, plugin discovery mechanism, +storage schema, public transport operation, synchronization algorithm, scheduler, or document implementation. + +# Motivation + +`ContentSource` and `POST /v1/sources/content` provide captured-text ingestion. The caller +chooses one `source_id`; replaying an identical payload is idempotent, while reusing that identity with a different +payload is a conflict. This gives exact evidence only when the caller treats the identity as immutable. + +External systems usually expose a different lifecycle. A wiki page, issue, object, message, or file has one logical +identity but may produce several values over time. The external object can be renamed, revised, deleted, restored, +or become temporarily unreadable. Artifacts that used an earlier value must continue to cite that exact evidence. +The two-part `(source_type, source_id)` Source reference cannot express both the stable logical object and its immutable observation +without making every integration invent a composite `source_id`. + +The extension boundary is also incomplete. A Source adapter binds a native input class to a concrete +Source class and a read result, while the built-in Runtime and relational persistence assemble a fixed adapter set. +This does not state the durable rules an independently defined Source type must follow across identity, persistence, +transport, and Artifact evidence. + +The standard model must answer five questions without assigning them to one identifier: + +1. Which Scope owns this evidence? +2. Which logical external or internal source does it describe? +3. Which exact observed value did an Artifact use? +4. Where does PowerContext read that exact value from? +5. Which definition gives the value and provenance their meaning? + +Connector concerns are adjacent but distinct. Discovery, credentials, filtering, checkpoints, retries, provider +change handling, and deletion detection decide which observations are submitted. They do not define Source identity, +weaken exact evidence, or change Scope ownership. + +# Guide-level explanation + +## Domain model + +Read the model by establishing ownership first, then logical identity, exact observation, materialization authority, +and type semantics: + +| Concept | Representation | Question answered | +| --- | --- | --- | +| Ownership | Scope | Where does the Source belong? | +| Logical identity | `SourceKey` | Which continuing source is this? | +| Exact evidence | `SourceRef` | Which immutable observation is cited? | +| Read authority | materialization | Where is that exact value resolved? | +| Type semantics | Source Definition | How are value, provenance, and identity interpreted? | +| Acquisition | Connector or direct caller | How are new observations found and submitted? | + +These responsibilities form one direction of dependency: + +```text +Connector or direct caller + | + v +Source Definition + | + v +Scope-owned Source history + | + +---- mutable head selection + | + `---- exact SourceRef ----> Artifact evidence +``` + +A Connector can use one Source Definition, several Connectors can use the same Definition, and a direct caller can +submit a Source without a Connector. Connector identity therefore does not become Source type identity. + +## Scope ownership + +Every SourceKey and observation belongs to exactly one Scope. Scope ownership is not inferred from an external +workspace, path, repository, provider account, Connector instance, or Source locator. Those values may contribute to +binding or provenance, but they do not allocate or replace `scope_id`. + +The fully qualified logical identity is: + +```text +SourceKey = (scope_id, source_type, source_id) +``` + +The fully qualified exact identity is: + +```text +SourceRef = (scope_id, source_type, source_id, observation_id) +``` + +A scope-bound operation may obtain `scope_id` from its fixed request binding instead of accepting it as an arbitrary +argument. The durable resolved reference still retains the owner Scope so that evidence remains unambiguous after +publication, reporting, or export. + +Changing a Scope Parent, Context References, an Agent binding, or an observation selection changes no SourceKey or +SourceRef. Publishing an Artifact across Scopes preserves the original Scope and exact SourceRef in provenance. It +does not move or implicitly copy the Source history. + +## Logical Source and immutable observation + +`source_id` names a logical source within one `(scope_id, source_type)` namespace. Its meaning is defined by the +Source Definition. It may correspond to a provider object ID, a stable import identity, or another normalized key. +It must not silently change when a new value is observed. + +`observation_id` names one immutable observation under a SourceKey. It is opaque to generic PowerContext components. +It may be derived from a provider revision, a canonical value digest, or a definition-specific combination. It does +not imply an integer sequence, timestamp order, or ancestry. + +The following invariants apply: + +- one `(SourceKey, observation_id)` identifies one canonical observation forever; +- re-observing the same canonical observation is idempotent; +- a different canonical observation cannot reuse an observation ID; +- one SourceKey may have several observations with the same value digest when their identity-bearing provenance is + different; +- observations with the same value digest are not automatically the same logical Source; and +- an Artifact cites an exact SourceRef, never a moving SourceKey or `latest` observation. + +For example, a document update retains one SourceKey and produces another SourceRef: + +```text +SourceKey(scope-a, document, provider-page-42) +|-- SourceRef(..., observation-1) "Initial decision" +`-- SourceRef(..., observation-2) "Revised decision" +``` + +An Artifact derived from `observation-1` continues to cite it after `observation-2` becomes current. + +## Source Definition + +A Source Definition is the durable semantic contract for one `source_type`. It declares: + +- a stable definition name and version; +- the Source value and typed provenance shapes; +- Source ID normalization and equality; +- observation ID normalization and equality; +- identity-bearing fields and non-identifying annotations; +- canonical bytes and the value digest algorithm; +- supported materialization modes and exact-read requirements; +- limits and validation failures; and +- compatibility rules for older definition versions. + +A Definition resolves definition-native input into a canonical observation and reads the definition-owned value from +an exact persisted observation. Resolution does not select a Scope, mutate a catalog, advance a head, or discover +external objects. Reading does not resolve `latest` or substitute another observation. + +Definitions are explicit and typed. A new integration must not simulate a new Source type by placing an undocumented +schema inside `ContentSource.metadata`. Provider-specific provenance may extend a Definition's declared schema, but +fields that affect identity, exactness, or compatibility must be named by the Definition. + +## Materialization authority + +Materialization answers where the value returned for an exact SourceRef comes from: + +| Materialization | Authority | Required guarantee | +| --- | --- | --- | +| `captured` | Canonical value retained by PowerContext | The retained value matches the observation digest | +| `referenced` | Immutable external revision | Re-reading the reference returns the same canonical value and digest | + +A captured Source may retain an external locator, provider revision, and digest as provenance. It remains captured +because the retained value is the read authority. This covers the useful part of a hybrid design without creating a +third mode with ambiguous fallback semantics. + +A Definition can use referenced materialization only when the external system and its reader can address immutable +historical values. Reading the current value at a path, page ID, issue ID, or URL is not sufficient. Modification +times and ETags may contribute to provenance or conflict detection, but a Definition must state whether the provider +guarantees that they address an immutable value. + +When the referenced value is unavailable or its digest differs, exact resolution fails. PowerContext does not return +the current provider value, a stale cache entry, or another observation. A provider that cannot satisfy this rule +must use captured materialization or reject the observation. + +## Current head and deletion + +A Source history is immutable; its current head is a mutable catalog selection. The head can select one exact +SourceRef or record that the logical Source was positively observed as deleted. The head is useful for current-state +queries and later acquisition, but it is not evidence and cannot appear in an Artifact citation. + +Advancing or deleting a head changes no observation. A timeout, permission failure, incomplete listing, unavailable +Connector, or disconnect is not positive deletion evidence and does not change the head. A Source Definition may +define a tombstone value only when deletion itself is meaningful Source evidence; a generic head deletion does not +fabricate one. + +## ContentSource + +`ContentSource` remains the neutral captured-text path defined by RFC 0019. Its caller chooses an identity that can be +committed once with one canonical payload. The persistence conflict rule makes an accepted ContentSource exact, but +it does not provide a separate logical Source lifecycle. + +The standard model treats this as a valid single-observation Source implementation: + +- the existing identity remains immutable; +- an identical replay remains idempotent; +- different content under the same identity remains a conflict; +- references that resolve ContentSource remain exact and unchanged; and +- no mutable head or multi-observation behavior is inferred from metadata. + +ContentSource is suitable for prompts, explicit text capture, import records, and other cases where the caller +already owns an immutable identity. Integrations that observe one logical object over time should define or reuse a +multi-observation Source type instead. + +## Document Source as the first validation + +The first validation Source represents a logical document with immutable observations. It is deliberately separate +from ContentSource. Its conformance scenarios require: + +- one logical document retaining its SourceKey across updates; +- each changed canonical document observation receiving an exact SourceRef; +- an unchanged observation replaying idempotently; +- an earlier observation remaining readable after update or deletion; +- provider locator changes not rewriting an accepted observation; +- incomplete discovery or permission loss not becoming deletion; and +- captured materialization when the provider cannot resolve immutable historical revisions. + +The validation does not make documents the universal Source value. Issues, messages, traces, code states, reviews, +and other Source types can define different values and provenance while following the same identity and observation +contract. + +# Reference-level explanation + +## Source identity contract + +`scope_id` is the ownership boundary defined by the Scope organization design. `source_type` is the stable Source +Definition name. `source_id` is a non-empty, normalized identifier whose equality and bounds are declared by that +Definition. + +Source identity is Scope-local. Two Scopes may contain equivalent external material without sharing ownership or +identity. A Definition may include a stable external instance or connection discriminator in its `source_id` rules +when required to prevent collisions, but the discriminator does not replace `scope_id`. + +Renames are definition-specific. A provider object ID may preserve SourceKey across locator changes. A path-derived +identity normally treats a rename as one logical deletion and one creation. A Definition must not claim rename-stable +identity when its provider and acquisition path cannot prove it. + +## Observation contract + +An observation contains these standard fields: + +```text +SourceObservation +|-- source_key +|-- observation_id +|-- definition_version +|-- materialization +|-- value_digest +|-- provenance +`-- definition-owned value or exact external reference +``` + +`value_digest` uses SHA-256 over the canonical bytes declared by the Definition and is encoded as +`sha256:`. For structured values, the Definition specifies a deterministic canonicalization. The +digest verifies value equality; it does not replace SourceKey or observation identity. + +The canonical observation contains every field that the Definition says affects identity or exact meaning. +Operational facts such as a retry count, last scan time, or processing status are not Source value and do not change +observation identity. If a timestamp or provider attribute affects provenance meaning, the Definition must classify +and canonicalize it explicitly. + +## Source reference contract + +A SourceRef identifies an exact observation and includes its owner Scope. It never accepts an absent observation ID, +`latest`, a head version, or a current provider locator. + +Within a scope-bound operation, a compact local representation may omit a repeated `scope_id` only while the current +Scope is fixed and the resolved durable value restores it. Any reference that crosses a Scope boundary, leaves the +Runtime, or enters durable cross-Scope provenance carries the owner Scope explicitly. + +Reference resolution verifies all four identity components and the stored observation's definition version and +digest. Failure to resolve the exact observation is distinct from the logical Source being deleted, the head having +advanced, or the Connector being unavailable. + +## Definition registration contract + +A composed Runtime has one explicit Definition registry. Registration validates stable Definition name and version, +declared value and provenance schemas, identity rules, materialization support, and read behavior. Two incompatible +Definitions cannot claim the same `(source_type, definition_version)`. + +Registration is fixed for the Runtime lifetime. Catalog decoding, Source reads, and Artifact validation use the same +registry view. A persisted observation whose Definition is unavailable remains stored but cannot be interpreted or +advertised as readable. It is not decoded into a base Source with discarded fields. + +Definition discovery and registration are separate. A package entry point or another discovery mechanism may report +available Definitions, but installation does not imply activation. This RFC does not select entry +points, a central settings format, pluggy, or a Connector marketplace. + +## Definition compatibility contract + +The Definition name remains stable across compatible schema evolution. Each persisted observation records the +Definition version used to validate and canonicalize it. A newer Definition version must either declare how it reads +an older observation without changing its canonical meaning or coexist with a reader for the older version. + +A Definition change is incompatible when it changes SourceKey equality, observation equality, canonical value bytes, +provenance meaning, or materialization guarantees for an accepted observation. Such a change requires a new +Definition version and cannot rewrite existing SourceRefs. + +Renaming a Definition creates a new `source_type`. Reclassifying an existing observation under another Definition is +an explicit derivation with provenance, not an in-place migration of identity. + +## Connector boundary + +A Connector owns provider interaction: discovery, credentials, filtering, checkpoints, retries, rate limits, +provider change handling, and positive deletion detection. It submits definition-native inputs against a Scope +binding and receives exact accepted SourceRefs. + +A Source Definition owns semantic normalization: logical identity, observation identity, canonical value, +provenance, materialization validity, and exact read. A Connector cannot override those rules. If the intersection of +provider capabilities, Connector behavior, and Definition requirements cannot satisfy a selected materialization, +the observation is rejected or captured under a valid mode. + +```text +provider capabilities + intersect Connector behavior + intersect Source Definition requirements + = valid Source observation +``` + +This RFC does not define Connector lifecycle interfaces or require that a Connector run inside the PowerContext +Server. Direct imports, local tools, hosted Connectors, and external synchronization services can submit the same +definition-native observations. + +## Artifact evidence and cross-Scope delivery + +An Artifact revision records exact SourceRefs used directly by its computation. Advancing a Source head does not +change existing Artifact lineage. Recalculation against a newer observation produces a new Artifact revision rather +than rewriting prior evidence. + +Sources remain in their producing Scope. A Context Reference may expand a read selection according to the Scope +organization contract, but it does not change Source ownership. Exact Artifact publication across Scopes retains the +origin Scope and exact SourceRef in lineage. Publishing an Artifact does not publish every Source in its origin Scope. + +If an application deliberately captures the same external value into another Scope, the target receives a new +Scope-owned Source observation. Its provenance may cite the origin Scoped SourceRef, but the original Source is not +moved and the two SourceKeys are not made identical. + +## Conformance + +A Source Definition can be supported only after its mandatory contract passes conformance scenarios for: + +- identity normalization and collision rejection; +- identical observation replay; +- conflicting payload rejection for one observation ID; +- several immutable observations under one SourceKey; +- exact old-observation reads after head advancement and deletion; +- digest verification for captured and referenced values; +- referenced-value unavailability and mutation; +- Scope isolation and explicit owner preservation; +- Definition version compatibility and unavailable-definition behavior; and +- explicit registration conflict handling. + +The first document validation additionally covers provider update, locator change, positive deletion, incomplete +discovery, permission loss, and providers without immutable revision reads. Passing document validation proves the +standard contract can support one document Source; it does not add document fields to the standard Source model. + +# Drawbacks + +- Separating SourceKey, SourceRef, Source head, and Definition version introduces more concepts than one immutable + `(source_type, source_id)` pair. +- Exact SourceRefs retain owner Scope and observation identity, increasing lineage payload size. +- Definition authors must specify canonicalization, provenance, and compatibility instead of relying on arbitrary + metadata. +- Referenced Sources are unavailable for providers that expose only current values, so some integrations must retain + captured data. +- Explicit registration requires deployment coordination before a persisted custom Source can be read. + +# Rationale and alternatives + +## Extend ContentSource into the general integration model + +Adding provider fields to ContentSource would preserve the `POST /v1/sources/content` capture API, but it would keep logical identity, +observation identity, and provenance inside caller conventions. Different integrations would encode incompatible +schemas in metadata, and non-text Source values would still need another model. ContentSource remains a useful +single-observation implementation instead. + +## Use one opaque Source envelope + +A universal JSON payload would make persistence and transport uniform, but would move schema validation and +compatibility into runtime conventions. Definition-owned typed values and provenance make the extension boundary +reviewable and allow consumers to reject unsupported Source types before interpretation. + +## Put an observation digest inside source_id + +An integration can preserve the two-part SourceRef shape by composing logical identity and digest into `source_id`. This +makes immutable capture possible but hides the continuing logical Source from the catalog. Updates, current-head +selection, deletion, and provider identity then become integration-private conventions. The standard model represents +both identities directly. + +## Make SourceRef logical and add a separate ObservationRef + +Two public reference types would make SourceRef logical, but Artifact evidence would need +to reject SourceRef and accept only ObservationRef. Defining SourceRef itself as exact follows the existing ArtifactRef +principle that durable lineage references immutable state. + +## Add hybrid materialization + +A third mode that sometimes reads externally and sometimes falls back to captured data obscures which value is +authoritative and which failures are visible. A captured observation can retain a complete external reference as +provenance. A referenced observation either resolves exactly or fails. + +## Let Parent or Connector identity own Sources + +Scope Parent is organization, and Connector identity is acquisition provenance. Neither is a durable ownership +boundary. Using either would conflict with the Scope organization contract and would make reorganization or +Connector replacement change Source identity. + +# Prior art + +- The [Scope organization and Agent integration design](https://github.com/oceanbase/powercontext/pull/1345) separates + Scope ownership, read sharing, organization, delivery, and observation. This RFC applies the same separation to + Source ownership, identity, exact evidence, and acquisition. +- [Apache OpenDAL OFS RFC-0016](https://github.com/apache/opendal-ofs/blob/main/rfcs/0016_filesystem_architecture.md) + separates namespace authority from access frontends and forbids a frontend from advertising guarantees that the + underlying layers cannot enforce. Source materialization follows the same authority rule. +- [opendalfs](https://github.com/fsspec/opendalfs) exposes OpenDAL services through the fsspec interface and is a + candidate acquisition layer for the first filesystem-backed document Connector. Its paths and file metadata do + not define Source identity or immutable revision semantics. A backend read can satisfy referenced materialization + only when the complete stack addresses and verifies an immutable revision; otherwise the document is captured. +- DataHub stateful ingestion separates connector checkpoints and stale-entity detection from emitted metadata + identity. Airbyte treats connector state as an opaque recovery boundary rather than record identity. +- OpenMetadata separates the Source that emits records from connection checks, workflow status, and the sink. +- Nowledge Mem's TiddlyWiki importer uses stable logical IDs, canonical payload digests, source revalidation, and + per-item outcomes. Those behaviors motivate the document validation without defining the standard Source value. + +# Unresolved questions + +- Must every durable SourceRef carry `scope_id` directly, or may a canonical scoped envelope contain a local exact + SourceRef while preserving the same fully qualified identity? +- Which Source Definition versions must a Runtime retain simultaneously before a Definition can be considered + supported? +- Should Source head deletion be one common catalog state, or should the first standard contract expose only an + active exact head and leave deletion entirely to Connector state? +- Which normalized value categories, if any, should Artifact families share without requiring them to understand a + complete definition-owned value schema? + +# Future possibilities + +Connector lifecycle, checkpoints, run status, and explicit plugin discovery require a separate contract. Document +ingestion supplies a conformance case for that contract without changing Source identity or materialization semantics. + +Definitions may advertise optional projections, such as text, structured records, or binary attachments, for +Artifact families that cannot consume the complete native value. Projection identity and digest rules require their +own contract and do not weaken the original Source observation. + +Retention policies may reclaim captured values only after defining how exact Artifact evidence reports unavailable +content and how legal or user-requested deletion interacts with immutable lineage. A Source head deletion alone does +not authorize evidence removal. diff --git a/docs/zh/rfcs/0000_source_definition_and_observation_model.md b/docs/zh/rfcs/0000_source_definition_and_observation_model.md new file mode 100644 index 000000000..3f855b827 --- /dev/null +++ b/docs/zh/rfcs/0000_source_definition_and_observation_model.md @@ -0,0 +1,444 @@ +- Proposal Name: `source_definition_and_observation_model` +- Start Date: 2026-08-27 +- Related Discussion: [oceanbase/powercontext#1240](https://github.com/oceanbase/powercontext/issues/1240), + [oceanbase/powercontext#1363](https://github.com/oceanbase/powercontext/issues/1363) +- Related Design: [oceanbase/powercontext#1345](https://github.com/oceanbase/powercontext/pull/1345) +- Related RFCs: [RFC 0002](0002_core_sdk_product_model.md)、[RFC 0014](0014_memory_layer_design.md)、 + [RFC 0019](0019_local_source_memory_runtime.md)、[RFC 0048](0048_handoff_artifact.md) + +# Summary + +本 RFC 定义标准 Source 模型，以及新增 Source 类型时必须遵守的契约。 + +每个 Source 只属于一个 Scope。在该 Scope 内，`SourceKey` 标识一个逻辑 Source，`SourceRef` 标识这个 +Source 的一次不可变观察。推进当前观察、观察到删除、修改外部 locator 或断开 Connector，都不会改变 +已经接受的观察，也不会将其移动到另一个 Scope。 + +Source Definition 为一个稳定的 Source 类型定义 value schema、provenance schema、身份规则、观察规则、 +materialization 契约、canonicalization 与兼容策略。Definition 显式注册，并在组合完成的 Runtime 生命周期内 +保持不变。持久化、传输与 Artifact consumer 按稳定的 Definition 名称和版本路由，而不是按具体 Python 类路由。 + +Materialization 表达解析某个精确观察时所依赖的权威来源。Captured observation 从 PowerContext 保留的 +canonical value 解析；referenced observation 从外部不可变 revision 解析。仅有外部 locator、修改时间、 +ETag 或 provider 当前值读取，并不能满足 referenced 契约。 + +`ContentSource` 继续作为简单的 captured-text Source。调用方提供稳定身份，加上 immutable-payload 冲突规则， +适合一次性内容捕获，但它不是通用的外部集成模型。Document Source 是标准 Definition 契约的首个 conformance 验证对象， +而不是标准契约本身。 + +本 RFC 只定义语义与 conformance，不定义 Connector runtime、插件发现机制、存储 schema、公开 transport +operation、同步算法、scheduler 或文档实现。 + +# Motivation + +`ContentSource` 与 `POST /v1/sources/content` 提供 captured-text ingestion。调用方选择一个 +`source_id`；使用完全相同的 payload 重放具有幂等性，而用不同 payload 复用该身份会产生冲突。只有调用方把 +这个身份当作不可变身份时，它才能表达精确证据。 + +外部系统通常具有不同的生命周期。Wiki 页面、issue、object、message 或 file 拥有一个逻辑身份，但会随时间 +产生多个值。外部对象可能被重命名、修订、删除、恢复或暂时无法读取。使用过旧值的 Artifact 必须继续引用当时的 +精确证据。二元 `(source_type, source_id)` Source reference 无法同时表达稳定的逻辑对象和不可变观察，只能迫使每个集成自行发明复合 +`source_id`。 + +扩展边界也不完整。Source adapter 将 native input class 绑定到具体 Source class 和读取结果，而内置 +Runtime 与关系型持久化会组装固定 adapter 集合。它没有说明独立定义的 Source 类型在身份、持久化、传输与 +Artifact evidence 上必须长期满足哪些规则。 + +标准模型必须回答五个问题，且不能把它们压进同一个 identifier： + +1. 哪个 Scope 拥有这份证据？ +2. 它描述哪个逻辑上的外部或内部 Source？ +3. Artifact 使用的是哪个精确观察值？ +4. PowerContext 从哪里读取该精确值？ +5. 哪个 Definition 赋予 value 与 provenance 语义？ + +Connector concerns 与此相邻但不同。Discovery、credentials、filtering、checkpoints、retries、provider +change handling 与 deletion detection 决定提交哪些观察；它们不定义 Source identity，不能削弱精确证据， +也不能改变 Scope ownership。 + +# Guide-level explanation + +## Domain model + +理解该模型时，依次确定 ownership、logical identity、exact observation、materialization authority 与 type +semantics： + +| Concept | Representation | Question answered | +| --- | --- | --- | +| Ownership | Scope | Source 属于哪里？ | +| Logical identity | `SourceKey` | 这是哪个持续存在的 Source？ | +| Exact evidence | `SourceRef` | 引用的是哪个不可变观察？ | +| Read authority | materialization | 从哪里解析该精确值？ | +| Type semantics | Source Definition | 如何解释 value、provenance 与 identity？ | +| Acquisition | Connector or direct caller | 如何发现并提交新观察？ | + +这些职责形成单向依赖： + +```text +Connector or direct caller + | + v +Source Definition + | + v +Scope-owned Source history + | + +---- mutable head selection + | + `---- exact SourceRef ----> Artifact evidence +``` + +一个 Connector 可以使用一个 Source Definition，多个 Connector 可以共用同一个 Definition，直接调用方也可以 +在没有 Connector 的情况下提交 Source。因此 Connector identity 不会成为 Source type identity。 + +## Scope ownership + +每个 SourceKey 与 observation 都只属于一个 Scope。Scope ownership 不从外部 workspace、path、repository、 +provider account、Connector instance 或 Source locator 推导。这些值可以参与 binding 或 provenance，但不能 +分配或替代 `scope_id`。 + +完整限定的逻辑身份为： + +```text +SourceKey = (scope_id, source_type, source_id) +``` + +完整限定的精确身份为： + +```text +SourceRef = (scope_id, source_type, source_id, observation_id) +``` + +Scope-bound operation 可以从固定 request binding 获得 `scope_id`，而不把它作为任意参数接收。持久化的解析结果 +仍然保留 owner Scope，使证据在 publication、reporting 或 export 后仍无歧义。 + +修改 Scope Parent、Context References、Agent binding 或 observation selection，都不会改变 SourceKey 或 +SourceRef。跨 Scope 发布 Artifact 时，provenance 保留原始 Scope 与精确 SourceRef；不会移动或隐式复制 +Source history。 + +## Logical Source and immutable observation + +`source_id` 在一个 `(scope_id, source_type)` namespace 内命名逻辑 Source，其含义由 Source Definition +规定。它可以对应 provider object ID、稳定 import identity 或其他 normalized key。观察到新值时，它不能静默改变。 + +`observation_id` 在一个 SourceKey 下命名一次不可变观察。对通用 PowerContext component 而言它是不透明的； +可以派生自 provider revision、canonical value digest 或 Definition 特有组合。它不隐含整数序列、时间顺序或祖先关系。 + +适用以下不变量： + +- 一个 `(SourceKey, observation_id)` 永远标识同一个 canonical observation； +- 再次观察相同 canonical observation 具有幂等性； +- 不同 canonical observation 不能复用 observation ID； +- 如果 identity-bearing provenance 不同，同一个 SourceKey 下可以有 value digest 相同的多个 observation； +- value digest 相同的 observation 不会自动成为同一个逻辑 Source； +- Artifact 只引用精确 SourceRef，绝不引用移动的 SourceKey 或 `latest` observation。 + +例如，一次文档更新保留同一 SourceKey 并产生新的 SourceRef： + +```text +SourceKey(scope-a, document, provider-page-42) +|-- SourceRef(..., observation-1) "Initial decision" +`-- SourceRef(..., observation-2) "Revised decision" +``` + +即使 `observation-2` 已成为 current，派生自 `observation-1` 的 Artifact 仍然引用后者。 + +## Source Definition + +Source Definition 是一个 `source_type` 的持久语义契约。它声明： + +- 稳定的 Definition name 与 version； +- Source value 与 typed provenance 的结构； +- Source ID normalization 与 equality； +- observation ID normalization 与 equality； +- identity-bearing fields 与 non-identifying annotations； +- canonical bytes 与 value digest algorithm； +- 支持的 materialization modes 与 exact-read requirements； +- limits 与 validation failures； +- older Definition versions 的 compatibility rules。 + +Definition 将 definition-native input 解析为 canonical observation，并从精确的 persisted observation 读取 +Definition 拥有的 value。解析不会选择 Scope、修改 catalog、推进 head 或发现外部 object；读取不会解析 +`latest`，也不会替换为另一个 observation。 + +Definition 必须显式且类型化。新的集成不能通过在 `ContentSource.metadata` 中放置未声明 schema 来模拟新 +Source 类型。Provider-specific provenance 可以扩展 Definition 声明的 schema，但影响 identity、exactness +或 compatibility 的字段必须由 Definition 命名。 + +## Materialization authority + +Materialization 回答精确 SourceRef 的返回值来自哪里： + +| Materialization | Authority | Required guarantee | +| --- | --- | --- | +| `captured` | PowerContext 保留的 canonical value | 保留值与 observation digest 一致 | +| `referenced` | Immutable external revision | 重读 reference 得到相同 canonical value 与 digest | + +Captured Source 可以把 external locator、provider revision 与 digest 保留为 provenance。因为读取权威仍是 +保留值，所以它依然是 captured。这覆盖了 hybrid design 中有价值的部分，而不引入 fallback 语义含糊的第三种模式。 + +只有当外部系统及其 reader 能够寻址不可变历史值时，Definition 才能使用 referenced materialization。读取 +path、page ID、issue ID 或 URL 的当前值并不足够。Modification time 与 ETag 可以参与 provenance 或 conflict +detection，但 Definition 必须说明 provider 是否保证它们指向不可变值。 + +Referenced value 不可用或 digest 不同时，精确解析失败。PowerContext 不返回 provider 当前值、stale cache +entry 或其他 observation。不能满足该规则的 provider 必须使用 captured materialization，或者拒绝该 observation。 + +## Current head and deletion + +Source history 不可变；current head 是可变的 catalog selection。Head 可以选择一个精确 SourceRef，或记录已 +明确观察到逻辑 Source 被删除。Head 可用于 current-state query 与后续 acquisition，但它不是 evidence，不能 +出现在 Artifact citation 中。 + +推进或删除 head 不改变任何 observation。Timeout、permission failure、incomplete listing、Connector +unavailable 或 disconnect 都不是明确的 deletion evidence，不能改变 head。只有当 deletion 本身是有意义的 +Source evidence 时，Source Definition 才可以定义 tombstone value；通用 head deletion 不会伪造这种值。 + +## ContentSource + +`ContentSource` 继续作为 RFC 0019 定义的 neutral captured-text path。调用方选择一个只能与一个 canonical +payload 一起提交的身份。Persistence conflict rule 使接受后的 ContentSource 可作为精确证据，但它不提供独立的 +logical Source lifecycle。 + +标准模型把它视为有效的 single-observation Source： + +- 现有 identity 保持不可变； +- 相同内容重放继续保持幂等； +- 同一 identity 下的不同内容继续发生冲突； +- 解析 ContentSource 的 reference 保持精确且不变； +- 不从 metadata 推导 mutable head 或 multi-observation behavior。 + +ContentSource 适合 prompt、显式文本捕获、import record，以及调用方已经拥有不可变身份的其他场景。持续观察同一 +逻辑对象的集成应定义或复用 multi-observation Source type。 + +## Document Source as the first validation + +首个验证 Source 表示具有不可变 observation 的逻辑 document，并刻意与 ContentSource 分离。它的 conformance +scenario 要求： + +- 一个逻辑 document 在更新过程中保持 SourceKey； +- canonical document observation 每次变化都得到精确 SourceRef； +- 未变化的 observation 可幂等重放； +- 更新或删除之后，旧 observation 仍可读取； +- provider locator 变化不重写已接受的 observation； +- incomplete discovery 或 permission loss 不会变成 deletion； +- provider 不能读取不可变历史 revision 时使用 captured materialization。 + +该验证不会让 document 成为通用 Source value。Issue、message、trace、code state、review 以及其他 Source 类型 +可以定义不同 value 与 provenance，同时遵循相同 identity 与 observation 契约。 + +# Reference-level explanation + +## Source identity contract + +`scope_id` 是 Scope organization design 定义的 ownership boundary。`source_type` 是稳定的 Source Definition +name。`source_id` 是非空的 normalized identifier，其 equality 与 bounds 由 Definition 声明。 + +Source identity 以 Scope 为本地边界。两个 Scope 可以包含等价的外部材料，但不共享 ownership 或 identity。 +需要避免碰撞时，Definition 可以在 `source_id` 规则中包含稳定的 external instance 或 connection discriminator， +但 discriminator 不替代 `scope_id`。 + +Rename 行为由 Definition 决定。Provider object ID 可以在 locator 变化时保留 SourceKey；path-derived identity +通常把 rename 视为一次逻辑 deletion 与一次 creation。当 provider 与 acquisition path 无法证明 rename-stable +identity 时，Definition 不能宣称支持它。 + +## Observation contract + +Observation 包含以下标准字段： + +```text +SourceObservation +|-- source_key +|-- observation_id +|-- definition_version +|-- materialization +|-- value_digest +|-- provenance +`-- definition-owned value or exact external reference +``` + +`value_digest` 对 Definition 声明的 canonical bytes 使用 SHA-256，并编码为 `sha256:`。对结构化 +value，Definition 指定 deterministic canonicalization。Digest 用于验证 value equality，不替代 SourceKey +或 observation identity。 + +Canonical observation 包含所有被 Definition 认定会影响 identity 或 exact meaning 的字段。Retry count、 +last scan time 或 processing status 等 operational facts 不是 Source value，不改变 observation identity。 +如果 timestamp 或 provider attribute 会影响 provenance meaning，Definition 必须显式分类并 canonicalize。 + +## Source reference contract + +SourceRef 标识精确 observation，并包含 owner Scope。它不接受缺失 observation ID、`latest`、head version 或 +current provider locator。 + +在 scope-bound operation 内，只有当 current Scope 固定且解析出的 durable value 会恢复 `scope_id` 时，紧凑的 +local representation 才可以省略重复的 `scope_id`。跨越 Scope boundary、离开 Runtime 或进入 durable +cross-Scope provenance 的 reference 必须显式携带 owner Scope。 + +Reference resolution 会验证全部四个 identity components，以及 stored observation 的 Definition version 与 +digest。无法解析精确 observation，不等同于 logical Source 已删除、head 已推进或 Connector 不可用。 + +## Definition registration contract + +组合后的 Runtime 拥有一个显式 Definition registry。注册时验证稳定的 Definition name 与 version、声明的 value +与 provenance schemas、identity rules、materialization support 和 read behavior。两个不兼容 Definition 不能 +声明同一个 `(source_type, definition_version)`。 + +Registry 在 Runtime 生命周期内固定。Catalog decoding、Source reads 与 Artifact validation 使用同一个 registry +view。Definition 不可用时，已经持久化的 observation 仍保留，但不能被解释或宣称为 readable；不能把它解码成 +丢失字段的 base Source。 + +Definition discovery 与 registration 相互独立。Package entry point 或其他 discovery mechanism 可以报告 +可用 Definition，但安装不意味着激活。本 RFC 不选择 entry points、central settings format、pluggy 或 +Connector marketplace。 + +## Definition compatibility contract + +Definition name 在兼容 schema 演进中保持稳定。每个 persisted observation 记录验证和 canonicalize 它时使用的 +Definition version。新的 Definition version 必须声明如何在不改变 canonical meaning 的前提下读取旧 observation， +或与旧版本 reader 共存。 + +如果 Definition change 会改变已接受 observation 的 SourceKey equality、observation equality、canonical value +bytes、provenance meaning 或 materialization guarantee，它就是不兼容变更。此类变更需要新的 Definition version， +且不能重写已有 SourceRef。 + +重命名 Definition 会产生新的 `source_type`。把已有 observation 重新分类到另一个 Definition 是带 provenance +的显式 derivation，不是 identity 的原地 migration。 + +## Connector boundary + +Connector 负责 provider interaction：discovery、credentials、filtering、checkpoints、retries、rate limits、 +provider change handling 与 positive deletion detection。它依据 Scope binding 提交 definition-native input， +并接收接受后的精确 SourceRef。 + +Source Definition 负责 semantic normalization：logical identity、observation identity、canonical value、 +provenance、materialization validity 与 exact read。Connector 不能覆盖这些规则。如果 provider capabilities、 +Connector behavior 与 Definition requirements 的交集无法满足选定 materialization，则拒绝 observation，或在 +合法模式下 captured。 + +```text +provider capabilities + intersect Connector behavior + intersect Source Definition requirements + = valid Source observation +``` + +本 RFC 不定义 Connector lifecycle interface，也不要求 Connector 运行在 PowerContext Server 内。Direct +import、local tool、hosted Connector 与 external synchronization service 都可以提交相同的 definition-native +observation。 + +## Artifact evidence and cross-Scope delivery + +Artifact revision 记录其计算直接使用的精确 SourceRef。推进 Source head 不改变现有 Artifact lineage。针对较新 +observation 的重新计算会产生新的 Artifact revision，而不是重写旧 evidence。 + +Source 保留在 producing Scope。Context Reference 可以按照 Scope organization contract 扩展 read selection， +但不会改变 Source ownership。跨 Scope 的精确 Artifact publication 在 lineage 中保留 origin Scope 与精确 +SourceRef。发布 Artifact 不会发布其 origin Scope 中的所有 Source。 + +如果 application 刻意把同一个外部值 captured 到另一个 Scope，target 会得到由该 Scope 拥有的新 Source +observation。其 provenance 可以引用 origin scoped SourceRef，但原始 Source 不会移动，两个 SourceKey 也不会 +因此变成同一 identity。 + +## Conformance + +Source Definition 只有在以下 mandatory contract 的 conformance scenario 通过后才能被支持： + +- identity normalization 与 collision rejection； +- identical observation replay； +- 同一 observation ID 的 conflicting payload rejection； +- 一个 SourceKey 下的多个 immutable observation； +- head advancement 与 deletion 后仍能精确读取旧 observation； +- captured 与 referenced value 的 digest verification； +- referenced-value unavailability 与 mutation； +- Scope isolation 与显式 owner preservation； +- Definition version compatibility 与 unavailable-definition behavior； +- explicit registration conflict handling。 + +首个 document validation 还覆盖 provider update、locator change、positive deletion、incomplete discovery、 +permission loss，以及没有 immutable revision read 的 provider。通过文档验证只证明标准契约可以支持一个 +Document Source，不会把文档字段加入标准 Source 模型。 + +# Drawbacks + +- 分离 SourceKey、SourceRef、Source head 与 Definition version，比一个不可变的 `(source_type, source_id)` pair + 引入更多概念。 +- 精确 SourceRef 保留 owner Scope 与 observation identity，会增加 lineage payload 大小。 +- Definition author 必须声明 canonicalization、provenance 与 compatibility，而不能依赖任意 metadata。 +- 只暴露当前值的 provider 无法使用 Referenced Source，因此部分集成必须保留 captured data。 +- Persisted custom Source 可读之前，显式 registration 需要部署协调。 + +# Rationale and alternatives + +## Extend ContentSource into the general integration model + +向 ContentSource 添加 provider fields 可以保留 `POST /v1/sources/content` capture API，但仍会把 logical identity、observation identity +与 provenance 留在调用方约定中。不同集成会在 metadata 中编码不兼容 schema，non-text Source value 仍需要另一 +套模型。因此 ContentSource 继续作为有用的 single-observation implementation。 + +## Use one opaque Source envelope + +通用 JSON payload 可以统一 persistence 与 transport，但会把 schema validation 和 compatibility 推给 runtime +convention。Definition-owned typed value 与 provenance 让扩展边界可审查，并允许 consumer 在解释前拒绝不支持的 +Source type。 + +## Put an observation digest inside source_id + +集成可以把 logical identity 与 digest 组合进 `source_id`，从而维持二元 SourceRef 形态。这可以表达 immutable +capture，却会在 catalog 中隐藏持续存在的 logical Source。Update、current-head selection、deletion 与 provider +identity 都会变成 integration-private convention。标准模型直接表达两类 identity。 + +## Make SourceRef logical and add a separate ObservationRef + +两个 public reference type 可以让 SourceRef 表示逻辑身份，但 Artifact evidence 必须拒绝 SourceRef，只接受 +ObservationRef。让 SourceRef 本身保持精确，符合现有 ArtifactRef 原则：durable lineage 引用 immutable state。 + +## Add hybrid materialization + +增加一种有时从外部读取、有时回退到 captured data 的第三种模式，会掩盖哪个 value 才是 authoritative，以及哪些 +failure 应对外可见。Captured observation 可以把完整 external reference 保留为 provenance；referenced +observation 要么被精确解析，要么失败。 + +## Let Parent or Connector identity own Sources + +Scope Parent 用于 organization，Connector identity 是 acquisition provenance，二者都不是持久 ownership +boundary。使用其中任意一个都会与 Scope organization contract 冲突，并让 reorganization 或 Connector +replacement 改变 Source identity。 + +# Prior art + +- [Scope organization and Agent integration design](https://github.com/oceanbase/powercontext/pull/1345) 分离 + Scope ownership、read sharing、organization、delivery 与 observation。本 RFC 对 Source ownership、identity、 + exact evidence 与 acquisition 应用同样的分离原则。 +- [Apache OpenDAL OFS RFC-0016](https://github.com/apache/opendal-ofs/blob/main/rfcs/0016_filesystem_architecture.md) + 分离 namespace authority 与 access frontend，并禁止 frontend 宣称底层无法兑现的保证。Source materialization + 遵循同样的 authority rule。 +- [opendalfs](https://github.com/fsspec/opendalfs) 通过 fsspec interface 暴露 OpenDAL services，可以作为首个 + filesystem-backed document Connector 的候选 acquisition layer。它的 path 与 file metadata 不定义 Source + identity 或 immutable revision semantics。只有完整调用链能够寻址并验证不可变 revision 时，backend read 才能 + 满足 referenced materialization；否则 document 必须被 captured。 +- DataHub stateful ingestion 把 connector checkpoint 与 stale-entity detection 同 emitted metadata identity + 分离。Airbyte 把 connector state 当作 opaque recovery boundary，而不是 record identity。 +- OpenMetadata 把负责生成 record 的 Source 与 connection check、workflow status、sink 分离。 +- Nowledge Mem 的 TiddlyWiki importer 使用 stable logical ID、canonical payload digest、source revalidation 与 + per-item outcome。这些行为为 document validation 提供依据，但不定义标准 Source value。 + +# Unresolved questions + +- 每个 durable SourceRef 是否必须直接携带 `scope_id`，还是可以由 canonical scoped envelope 包含 local exact + SourceRef，同时保留相同的 fully qualified identity？ +- Runtime 必须同时保留哪些 Source Definition version，才能宣称某个 Definition 受支持？ +- Source head deletion 应是通用 catalog state，还是首个标准契约只暴露 active exact head，并把 deletion 完全留给 + Connector state？ +- Artifact family 可以共享哪些 normalized value category，而不要求理解完整的 definition-owned value schema？ + +# Future possibilities + +Connector lifecycle、checkpoint、run status 与显式 plugin discovery 需要独立契约。Document ingestion 为该 +契约提供 conformance case，但不能改变 Source identity 或 materialization semantics。 + +Definition 可以为无法消费完整 native value 的 Artifact family 声明 text、structured record 或 binary +attachment 等 optional projection。Projection identity 与 digest rules 需要自己的契约，且不能削弱原始 Source +observation。 + +Retention policy 只有在定义精确 Artifact evidence 如何报告 unavailable content，以及 legal/user-requested +deletion 如何与 immutable lineage 交互之后，才能回收 captured value。Source head deletion 本身不授权删除证据。 diff --git a/zensical.toml b/zensical.toml index ae3526af4..d535fabd8 100644 --- a/zensical.toml +++ b/zensical.toml @@ -57,6 +57,7 @@ nav = [ { "RFCs & Meetings" = [ { "RFCs" = [ { "Overview" = "en/rfcs/README.md" }, + { "0000 Source Definition and Observation Model" = "en/rfcs/0000_source_definition_and_observation_model.md" }, { "1229 Unified Workloads and Long-Horizon Memory Evaluation" = "en/rfcs/1229_unified_workloads_and_long_horizon_memory_evaluation.md" }, { "1223 Human-Agent Work Continuity" = "en/rfcs/1223_human_agent_work_continuity.md" }, { "0082 Handoff Report" = "en/rfcs/0082_handoff_report.md" }, @@ -132,6 +133,7 @@ nav = [ { "RFC 与会议纪要" = [ { "RFC" = [ { "概览" = "zh/rfcs/README.md" }, + { "0000 Source 定义与观察模型" = "zh/rfcs/0000_source_definition_and_observation_model.md" }, { "1229 统一工作负载与长程 Memory 评估" = "zh/rfcs/1229_unified_workloads_and_long_horizon_memory_evaluation.md" }, { "1223 人与 Agent 工作连续性" = "zh/rfcs/1223_human_agent_work_continuity.md" }, { "0082 Handoff 报告" = "zh/rfcs/0082_handoff_report.md" }, From 07ff8ac2d0ed64fddad29d2e09cc8cb447407224 Mon Sep 17 00:00:00 2001 From: Chojan Shang Date: Thu, 27 Aug 2026 16:45:32 +0800 Subject: [PATCH 2/9] docs: refine source extension contracts --- ...source_definition_and_observation_model.md | 139 ++++++++++++------ ...source_definition_and_observation_model.md | 130 ++++++++++------ 2 files changed, 178 insertions(+), 91 deletions(-) diff --git a/docs/en/rfcs/0000_source_definition_and_observation_model.md b/docs/en/rfcs/0000_source_definition_and_observation_model.md index 54f7ed7d2..c0a4ce338 100644 --- a/docs/en/rfcs/0000_source_definition_and_observation_model.md +++ b/docs/en/rfcs/0000_source_definition_and_observation_model.md @@ -20,17 +20,25 @@ rules, materialization contract, canonicalization, and compatibility policy. Def and remain fixed for the lifetime of a composed Runtime. Persistence, transport, and Artifact consumers route by the stable definition name and version rather than by a concrete Python class. +A Definition may advertise named projection capabilities for consumers that do not understand its native value. +Each projection has an independently versioned schema and deterministic meaning over one exact observation. A +consumer selects a projection by capability name and version, never by inspecting a concrete Source class. + +A Connector lifecycle binds provider acquisition to a Scope, submits definition-native observations, records +per-item outcomes, and advances an opaque checkpoint only after accepted observations are durable. Connector runs +distinguish complete discovery from incomplete discovery so that absence is not silently converted into deletion. + Materialization identifies the authority used to resolve an exact observation. A captured observation is resolved from the canonical value retained by PowerContext. A referenced observation is resolved from an immutable external revision. An external locator, modification time, ETag, or current-provider read does not by itself satisfy the referenced contract. `ContentSource` remains a simple captured-text Source. Its caller-stable identity and immutable-payload conflict rule -make it useful for one-shot content capture, but it is not the general external integration model. A document Source -serves as the first conformance validation of the standard definition contract, not the definition of that contract. +make it useful for one-shot content capture, but it is not the general external integration model. -This RFC defines semantics and conformance. It does not define a Connector runtime, plugin discovery mechanism, -storage schema, public transport operation, synchronization algorithm, scheduler, or document implementation. +This RFC defines Source, projection, and Connector lifecycle semantics and conformance. It does not define a hosting +runtime, plugin discovery mechanism, storage schema, public transport operation, synchronization algorithm, +scheduler, credential transport, concrete Source family, or Connector implementation. # Motivation @@ -49,13 +57,14 @@ Source class and a read result, while the built-in Runtime and relational persis This does not state the durable rules an independently defined Source type must follow across identity, persistence, transport, and Artifact evidence. -The standard model must answer five questions without assigning them to one identifier: +The standard model must answer six questions without assigning them to one identifier: 1. Which Scope owns this evidence? 2. Which logical external or internal source does it describe? 3. Which exact observed value did an Artifact use? 4. Where does PowerContext read that exact value from? 5. Which definition gives the value and provenance their meaning? +6. Which declared view may a consumer use without understanding the native value? Connector concerns are adjacent but distinct. Discovery, credentials, filtering, checkpoints, retries, provider change handling, and deletion detection decide which observations are submitted. They do not define Source identity, @@ -75,6 +84,7 @@ and type semantics: | Exact evidence | `SourceRef` | Which immutable observation is cited? | | Read authority | materialization | Where is that exact value resolved? | | Type semantics | Source Definition | How are value, provenance, and identity interpreted? | +| Consumer view | named projection | Which declared representation may a consumer use? | | Acquisition | Connector or direct caller | How are new observations found and submitted? | These responsibilities form one direction of dependency: @@ -142,12 +152,12 @@ The following invariants apply: - observations with the same value digest are not automatically the same logical Source; and - an Artifact cites an exact SourceRef, never a moving SourceKey or `latest` observation. -For example, a document update retains one SourceKey and produces another SourceRef: +For example, updating one logical Source retains its SourceKey and produces another SourceRef: ```text -SourceKey(scope-a, document, provider-page-42) -|-- SourceRef(..., observation-1) "Initial decision" -`-- SourceRef(..., observation-2) "Revised decision" +SourceKey(scope-a, record, provider-object-42) +|-- SourceRef(..., observation-1) "Initial value" +`-- SourceRef(..., observation-2) "Revised value" ``` An Artifact derived from `observation-1` continues to cite it after `observation-2` becomes current. @@ -174,6 +184,25 @@ Definitions are explicit and typed. A new integration must not simulate a new So schema inside `ContentSource.metadata`. Provider-specific provenance may extend a Definition's declared schema, but fields that affect identity, exactness, or compatibility must be named by the Definition. +## Named projection capabilities + +A named projection is an optional, Definition-owned view of one exact observation. It allows an Artifact family or +another consumer to use a declared representation without knowing the native Source value or concrete Python class. + +A projection is selected by a stable name and version. Its Definition declares the output schema, canonicalization, +digest rules, and failures. The projection is evaluated against an exact SourceRef and cannot resolve a head, +`latest`, or a current provider value. For the same Definition version, projection version, and exact observation, it +returns the same canonical result. + +Projection capability is explicit. A consumer that requires a projection rejects a Source that does not advertise a +compatible capability; it does not infer content from metadata or fall back to a similarly shaped Source class. A +projection can be cached or persisted as a derivative, but its authority remains the exact Source observation and +its lineage retains that SourceRef. + +This contract does not prescribe a catalog of standard projection names or payload schemas. A projection becomes a +shared standard only after interoperating definitions and consumers demonstrate that its semantics are stable. Until +then, a Definition may expose namespaced projections without making them mandatory for other Source types. + ## Materialization authority Materialization answers where the value returned for an exact SourceRef comes from: @@ -225,23 +254,6 @@ ContentSource is suitable for prompts, explicit text capture, import records, an already owns an immutable identity. Integrations that observe one logical object over time should define or reuse a multi-observation Source type instead. -## Document Source as the first validation - -The first validation Source represents a logical document with immutable observations. It is deliberately separate -from ContentSource. Its conformance scenarios require: - -- one logical document retaining its SourceKey across updates; -- each changed canonical document observation receiving an exact SourceRef; -- an unchanged observation replaying idempotently; -- an earlier observation remaining readable after update or deletion; -- provider locator changes not rewriting an accepted observation; -- incomplete discovery or permission loss not becoming deletion; and -- captured materialization when the provider cannot resolve immutable historical revisions. - -The validation does not make documents the universal Source value. Issues, messages, traces, code states, reviews, -and other Source types can define different values and provenance while following the same identity and observation -contract. - # Reference-level explanation ## Source identity contract @@ -301,6 +313,10 @@ A composed Runtime has one explicit Definition registry. Registration validates declared value and provenance schemas, identity rules, materialization support, and read behavior. Two incompatible Definitions cannot claim the same `(source_type, definition_version)`. +Registration also validates each advertised projection name and version, its declared output schema, and its +canonicalization contract. Two incompatible projections cannot claim the same capability key within one Definition +version. + Registration is fixed for the Runtime lifetime. Catalog decoding, Source reads, and Artifact validation use the same registry view. A persisted observation whose Definition is unavailable remains stored but cannot be interpreted or advertised as readable. It is not decoded into a base Source with discarded fields. @@ -319,10 +335,14 @@ A Definition change is incompatible when it changes SourceKey equality, observat provenance meaning, or materialization guarantees for an accepted observation. Such a change requires a new Definition version and cannot rewrite existing SourceRefs. +A projection change is incompatible when it changes the output schema, canonical bytes, or meaning for an accepted +observation. Such a change requires a new projection version. It does not require a new Source Definition version +when the Source value and observation semantics remain unchanged. + Renaming a Definition creates a new `source_type`. Reclassifying an existing observation under another Definition is an explicit derivation with provenance, not an in-place migration of identity. -## Connector boundary +## Connector lifecycle contract A Connector owns provider interaction: discovery, credentials, filtering, checkpoints, retries, rate limits, provider change handling, and positive deletion detection. It submits definition-native inputs against a Scope @@ -340,8 +360,32 @@ provider capabilities = valid Source observation ``` -This RFC does not define Connector lifecycle interfaces or require that a Connector run inside the PowerContext -Server. Direct imports, local tools, hosted Connectors, and external synchronization services can submit the same +A Connector type declares a stable name and version, its configuration schema, the Source Definitions it can submit, +and the acquisition capabilities it provides. Capabilities are optional and explicit. Typical capabilities include a +complete snapshot, a change feed, checkpoint resume, and authoritative deletion events. A Connector cannot advertise +a capability that its provider and acquisition path cannot enforce. + +A Connector binding activates one Connector configuration for exactly one Scope. The binding has a stable identity +for checkpoint and provider-namespace continuity, but it does not own Sources and does not replace `scope_id` or +`source_type`. Credentials are resolved by the hosting environment and do not become Source value or provenance. + +A Connector run begins from an opaque binding checkpoint, submits zero or more definition-native observations, and +records an outcome for every submitted item. An accepted or idempotently replayed observation returns its exact +SourceRef. A rejected or failed item remains visible in the run outcome and cannot be hidden by advancing the +checkpoint past work that is not safely replayable. + +A run finishes as complete or incomplete. A complete snapshot may produce positive deletion evidence for previously +known provider objects that are absent. An incomplete listing, timeout, permission failure, cancellation, or lost +connection produces no absence-based deletion evidence. An authoritative provider deletion event may produce +positive deletion evidence independently of snapshot completeness when its binding and object identity are verified. + +The completed checkpoint advances only after its accepted observations and deletion evidence are durable. Retrying +from an earlier checkpoint is valid because Source observation submission is idempotent. Connector checkpoint, +health, retry, and run-status records are operational state rather than Source observations or Artifact evidence. + +Installation, discovery, activation, and execution are separate concerns. Installing a Connector package does not +activate a binding. This contract does not require that a Connector run inside the PowerContext Server; direct tools, +hosted workers, and external synchronization services can follow the same lifecycle and submit the same definition-native observations. ## Artifact evidence and cross-Scope delivery @@ -373,9 +417,13 @@ A Source Definition can be supported only after its mandatory contract passes co - Definition version compatibility and unavailable-definition behavior; and - explicit registration conflict handling. -The first document validation additionally covers provider update, locator change, positive deletion, incomplete -discovery, permission loss, and providers without immutable revision reads. Passing document validation proves the -standard contract can support one document Source; it does not add document fields to the standard Source model. +A named projection can be advertised only after conformance verifies deterministic output for exact observations, +schema and version conflict handling, exact SourceRef lineage, and explicit failure when the capability is absent. + +A Connector capability can be advertised only after conformance verifies checkpoint replay, per-item outcome +visibility, durable checkpoint ordering, complete-versus-incomplete run behavior, and the claimed deletion evidence. +Provider-specific behavior is established by its implementation evidence rather than generalized into the standard +contract. # Drawbacks @@ -384,6 +432,7 @@ standard contract can support one document Source; it does not add document fiel - Exact SourceRefs retain owner Scope and observation identity, increasing lineage payload size. - Definition authors must specify canonicalization, provenance, and compatibility instead of relying on arbitrary metadata. +- Named projections and Connector lifecycle state add contracts that must evolve independently from Source values. - Referenced Sources are unavailable for providers that expose only current values, so some integrations must retain captured data. - Explicit registration requires deployment coordination before a persisted custom Source can be read. @@ -436,15 +485,15 @@ Connector replacement change Source identity. - [Apache OpenDAL OFS RFC-0016](https://github.com/apache/opendal-ofs/blob/main/rfcs/0016_filesystem_architecture.md) separates namespace authority from access frontends and forbids a frontend from advertising guarantees that the underlying layers cannot enforce. Source materialization follows the same authority rule. -- [opendalfs](https://github.com/fsspec/opendalfs) exposes OpenDAL services through the fsspec interface and is a - candidate acquisition layer for the first filesystem-backed document Connector. Its paths and file metadata do - not define Source identity or immutable revision semantics. A backend read can satisfy referenced materialization - only when the complete stack addresses and verifies an immutable revision; otherwise the document is captured. +- [opendalfs](https://github.com/fsspec/opendalfs) exposes OpenDAL services through the fsspec interface and + demonstrates backend-neutral filesystem acquisition. Its paths and file metadata do not define Source identity or + immutable revision semantics. A backend read can satisfy referenced materialization only when the complete stack + addresses and verifies an immutable revision. - DataHub stateful ingestion separates connector checkpoints and stale-entity detection from emitted metadata identity. Airbyte treats connector state as an opaque recovery boundary rather than record identity. - OpenMetadata separates the Source that emits records from connection checks, workflow status, and the sink. - Nowledge Mem's TiddlyWiki importer uses stable logical IDs, canonical payload digests, source revalidation, and - per-item outcomes. Those behaviors motivate the document validation without defining the standard Source value. + per-item outcomes. Those behaviors inform the separation between Source observations and Connector run state. # Unresolved questions @@ -452,19 +501,17 @@ Connector replacement change Source identity. SourceRef while preserving the same fully qualified identity? - Which Source Definition versions must a Runtime retain simultaneously before a Definition can be considered supported? -- Should Source head deletion be one common catalog state, or should the first standard contract expose only an +- Should Source head deletion be one common catalog state, or should the standard contract expose only an active exact head and leave deletion entirely to Connector state? -- Which normalized value categories, if any, should Artifact families share without requiring them to understand a - complete definition-owned value schema? +- Which projection names and schemas have enough implementation evidence to become shared standards rather than + namespaced capabilities? +- Which Connector hosting and scheduling contracts, if any, must be standardized beyond the lifecycle semantics in + this RFC? # Future possibilities -Connector lifecycle, checkpoints, run status, and explicit plugin discovery require a separate contract. Document -ingestion supplies a conformance case for that contract without changing Source identity or materialization semantics. - -Definitions may advertise optional projections, such as text, structured records, or binary attachments, for -Artifact families that cannot consume the complete native value. Projection identity and digest rules require their -own contract and do not weaken the original Source observation. +Explicit plugin discovery and deployment policy may build on Definition and Connector registration without making +package installation equivalent to activation. Retention policies may reclaim captured values only after defining how exact Artifact evidence reports unavailable content and how legal or user-requested deletion interacts with immutable lineage. A Source head deletion alone does diff --git a/docs/zh/rfcs/0000_source_definition_and_observation_model.md b/docs/zh/rfcs/0000_source_definition_and_observation_model.md index 3f855b827..0d248883c 100644 --- a/docs/zh/rfcs/0000_source_definition_and_observation_model.md +++ b/docs/zh/rfcs/0000_source_definition_and_observation_model.md @@ -18,16 +18,24 @@ Source Definition 为一个稳定的 Source 类型定义 value schema、provenan materialization 契约、canonicalization 与兼容策略。Definition 显式注册，并在组合完成的 Runtime 生命周期内 保持不变。持久化、传输与 Artifact consumer 按稳定的 Definition 名称和版本路由，而不是按具体 Python 类路由。 +Definition 可以为无法理解 native value 的 consumer 声明 named projection capability。每个 projection 拥有独立 +版本的 schema，并对一个精确 observation 具有确定语义。Consumer 按 capability name 与 version 选择 projection， +而不是检查具体 Source class。 + +Connector lifecycle 将 provider acquisition 绑定到 Scope，提交 definition-native observation，记录 per-item +outcome，并且只在接受的 observation 已持久化后推进 opaque checkpoint。Connector run 区分 complete discovery +与 incomplete discovery，避免把缺失对象静默转换为删除。 + Materialization 表达解析某个精确观察时所依赖的权威来源。Captured observation 从 PowerContext 保留的 canonical value 解析；referenced observation 从外部不可变 revision 解析。仅有外部 locator、修改时间、 ETag 或 provider 当前值读取，并不能满足 referenced 契约。 `ContentSource` 继续作为简单的 captured-text Source。调用方提供稳定身份，加上 immutable-payload 冲突规则， -适合一次性内容捕获，但它不是通用的外部集成模型。Document Source 是标准 Definition 契约的首个 conformance 验证对象， -而不是标准契约本身。 +适合一次性内容捕获，但它不是通用的外部集成模型。 -本 RFC 只定义语义与 conformance，不定义 Connector runtime、插件发现机制、存储 schema、公开 transport -operation、同步算法、scheduler 或文档实现。 +本 RFC 定义 Source、projection 与 Connector lifecycle 的语义和 conformance，不定义 hosting runtime、插件发现 +机制、存储 schema、公开 transport operation、同步算法、scheduler、credential transport、具体 Source family +或 Connector 实现。 # Motivation @@ -44,13 +52,14 @@ operation、同步算法、scheduler 或文档实现。 Runtime 与关系型持久化会组装固定 adapter 集合。它没有说明独立定义的 Source 类型在身份、持久化、传输与 Artifact evidence 上必须长期满足哪些规则。 -标准模型必须回答五个问题，且不能把它们压进同一个 identifier： +标准模型必须回答六个问题，且不能把它们压进同一个 identifier： 1. 哪个 Scope 拥有这份证据？ 2. 它描述哪个逻辑上的外部或内部 Source？ 3. Artifact 使用的是哪个精确观察值？ 4. PowerContext 从哪里读取该精确值？ 5. 哪个 Definition 赋予 value 与 provenance 语义？ +6. Consumer 可以使用哪个 declared view，而不必理解 native value？ Connector concerns 与此相邻但不同。Discovery、credentials、filtering、checkpoints、retries、provider change handling 与 deletion detection 决定提交哪些观察；它们不定义 Source identity，不能削弱精确证据， @@ -70,6 +79,7 @@ semantics： | Exact evidence | `SourceRef` | 引用的是哪个不可变观察？ | | Read authority | materialization | 从哪里解析该精确值？ | | Type semantics | Source Definition | 如何解释 value、provenance 与 identity？ | +| Consumer view | named projection | Consumer 可以使用哪个 declared representation？ | | Acquisition | Connector or direct caller | 如何发现并提交新观察？ | 这些职责形成单向依赖： @@ -133,12 +143,12 @@ Source history。 - value digest 相同的 observation 不会自动成为同一个逻辑 Source； - Artifact 只引用精确 SourceRef，绝不引用移动的 SourceKey 或 `latest` observation。 -例如，一次文档更新保留同一 SourceKey 并产生新的 SourceRef： +例如，更新一个逻辑 Source 会保留其 SourceKey，并产生新的 SourceRef： ```text -SourceKey(scope-a, document, provider-page-42) -|-- SourceRef(..., observation-1) "Initial decision" -`-- SourceRef(..., observation-2) "Revised decision" +SourceKey(scope-a, record, provider-object-42) +|-- SourceRef(..., observation-1) "Initial value" +`-- SourceRef(..., observation-2) "Revised value" ``` 即使 `observation-2` 已成为 current，派生自 `observation-1` 的 Artifact 仍然引用后者。 @@ -165,6 +175,23 @@ Definition 必须显式且类型化。新的集成不能通过在 `ContentSource Source 类型。Provider-specific provenance 可以扩展 Definition 声明的 schema，但影响 identity、exactness 或 compatibility 的字段必须由 Definition 命名。 +## Named projection capabilities + +Named projection 是一个 exact observation 的可选 Definition-owned view。它让 Artifact family 或其他 consumer +无需理解 native Source value 或具体 Python class，就能使用声明过的 representation。 + +Projection 通过稳定的 name 与 version 选择。其 Definition 声明 output schema、canonicalization、digest rules +与 failures。Projection 针对精确 SourceRef 求值，不能解析 head、`latest` 或 provider current value。对于相同的 +Definition version、projection version 与 exact observation，它必须返回相同的 canonical result。 + +Projection capability 必须显式声明。需要某个 projection 的 consumer 会拒绝未声明兼容 capability 的 Source， +而不会从 metadata 推断 content，也不会回退到形态相似的 Source class。Projection 可以作为 derivative 被缓存或 +持久化，但其 authority 仍是 exact Source observation，lineage 保留对应 SourceRef。 + +本契约不规定标准 projection name 或 payload schema 的目录。只有当多个 Definition 与 consumer 的互操作证明其 +语义稳定后，projection 才成为 shared standard。在此之前，Definition 可以暴露 namespaced projection，但不会让 +它成为其他 Source type 的 mandatory capability。 + ## Materialization authority Materialization 回答精确 SourceRef 的返回值来自哪里： @@ -211,22 +238,6 @@ logical Source lifecycle。 ContentSource 适合 prompt、显式文本捕获、import record，以及调用方已经拥有不可变身份的其他场景。持续观察同一 逻辑对象的集成应定义或复用 multi-observation Source type。 -## Document Source as the first validation - -首个验证 Source 表示具有不可变 observation 的逻辑 document，并刻意与 ContentSource 分离。它的 conformance -scenario 要求： - -- 一个逻辑 document 在更新过程中保持 SourceKey； -- canonical document observation 每次变化都得到精确 SourceRef； -- 未变化的 observation 可幂等重放； -- 更新或删除之后，旧 observation 仍可读取； -- provider locator 变化不重写已接受的 observation； -- incomplete discovery 或 permission loss 不会变成 deletion； -- provider 不能读取不可变历史 revision 时使用 captured materialization。 - -该验证不会让 document 成为通用 Source value。Issue、message、trace、code state、review 以及其他 Source 类型 -可以定义不同 value 与 provenance，同时遵循相同 identity 与 observation 契约。 - # Reference-level explanation ## Source identity contract @@ -283,6 +294,9 @@ digest。无法解析精确 observation，不等同于 logical Source 已删除 与 provenance schemas、identity rules、materialization support 和 read behavior。两个不兼容 Definition 不能 声明同一个 `(source_type, definition_version)`。 +注册还会验证每个声明的 projection name 与 version、output schema 和 canonicalization contract。两个不兼容的 +projection 不能在同一个 Definition version 内声明相同 capability key。 + Registry 在 Runtime 生命周期内固定。Catalog decoding、Source reads 与 Artifact validation 使用同一个 registry view。Definition 不可用时，已经持久化的 observation 仍保留，但不能被解释或宣称为 readable；不能把它解码成 丢失字段的 base Source。 @@ -301,10 +315,14 @@ Definition version。新的 Definition version 必须声明如何在不改变 ca bytes、provenance meaning 或 materialization guarantee，它就是不兼容变更。此类变更需要新的 Definition version， 且不能重写已有 SourceRef。 +如果 projection change 会改变已接受 observation 的 output schema、canonical bytes 或 meaning，它就是不兼容 +变更，需要新的 projection version。如果 Source value 与 observation semantics 保持不变，则不要求新的 Source +Definition version。 + 重命名 Definition 会产生新的 `source_type`。把已有 observation 重新分类到另一个 Definition 是带 provenance 的显式 derivation，不是 identity 的原地 migration。 -## Connector boundary +## Connector lifecycle contract Connector 负责 provider interaction：discovery、credentials、filtering、checkpoints、retries、rate limits、 provider change handling 与 positive deletion detection。它依据 Scope binding 提交 definition-native input， @@ -322,9 +340,30 @@ provider capabilities = valid Source observation ``` -本 RFC 不定义 Connector lifecycle interface，也不要求 Connector 运行在 PowerContext Server 内。Direct -import、local tool、hosted Connector 与 external synchronization service 都可以提交相同的 definition-native -observation。 +Connector type 声明稳定的 name 与 version、configuration schema、可提交的 Source Definition，以及它提供的 +acquisition capability。Capability 是可选且显式的，通常包括 complete snapshot、change feed、checkpoint resume +和 authoritative deletion event。Connector 不能声明 provider 与 acquisition path 无法兑现的 capability。 + +Connector binding 为一个 Scope 激活一份 Connector configuration。Binding 拥有用于 checkpoint 与 provider +namespace continuity 的稳定 identity，但不拥有 Source，也不替代 `scope_id` 或 `source_type`。Credential 由 +hosting environment 解析，不会成为 Source value 或 provenance。 + +Connector run 从 opaque binding checkpoint 开始，提交零个或多个 definition-native observation，并记录每个 +submitted item 的 outcome。Accepted 或 idempotently replayed observation 返回精确 SourceRef。Rejected 或 failed +item 会保留在 run outcome 中；如果尚不能安全重放，checkpoint 不能越过这些工作。 + +Run 以 complete 或 incomplete 结束。Complete snapshot 可以为之前已知但本次缺失的 provider object 产生 positive +deletion evidence。Incomplete listing、timeout、permission failure、cancellation 或 lost connection 不会产生 +absence-based deletion evidence。当 binding 与 object identity 均已验证时，authoritative provider deletion event +可以独立于 snapshot completeness 产生 positive deletion evidence。 + +Completed checkpoint 只有在 accepted observation 与 deletion evidence 均已持久化后才能推进。由于 Source +observation submission 具有幂等性，从更早 checkpoint 重试是合法行为。Connector checkpoint、health、retry 与 +run-status record 是 operational state，而不是 Source observation 或 Artifact evidence。 + +Installation、discovery、activation 与 execution 相互独立。安装 Connector package 不会激活 binding。本契约不 +要求 Connector 运行在 PowerContext Server 内；direct tool、hosted worker 与 external synchronization service +都可以遵循相同 lifecycle，提交相同 definition-native observation。 ## Artifact evidence and cross-Scope delivery @@ -354,9 +393,12 @@ Source Definition 只有在以下 mandatory contract 的 conformance scenario - Definition version compatibility 与 unavailable-definition behavior； - explicit registration conflict handling。 -首个 document validation 还覆盖 provider update、locator change、positive deletion、incomplete discovery、 -permission loss，以及没有 immutable revision read 的 provider。通过文档验证只证明标准契约可以支持一个 -Document Source，不会把文档字段加入标准 Source 模型。 +Named projection 只有在 conformance 验证 exact observation 的 deterministic output、schema 与 version conflict +handling、exact SourceRef lineage，以及 capability 缺失时显式失败之后才能被声明。 + +Connector capability 只有在 conformance 验证 checkpoint replay、per-item outcome visibility、durable checkpoint +ordering、complete-versus-incomplete run behavior，以及其声明的 deletion evidence 后才能被声明。Provider-specific +behavior 由对应实现证据确定，不会被直接推广为标准契约。 # Drawbacks @@ -364,6 +406,7 @@ Document Source，不会把文档字段加入标准 Source 模型。 引入更多概念。 - 精确 SourceRef 保留 owner Scope 与 observation identity，会增加 lineage payload 大小。 - Definition author 必须声明 canonicalization、provenance 与 compatibility，而不能依赖任意 metadata。 +- Named projection 与 Connector lifecycle state 增加了需要独立于 Source value 演进的契约。 - 只暴露当前值的 provider 无法使用 Referenced Source，因此部分集成必须保留 captured data。 - Persisted custom Source 可读之前，显式 registration 需要部署协调。 @@ -412,33 +455,30 @@ replacement 改变 Source identity。 - [Apache OpenDAL OFS RFC-0016](https://github.com/apache/opendal-ofs/blob/main/rfcs/0016_filesystem_architecture.md) 分离 namespace authority 与 access frontend，并禁止 frontend 宣称底层无法兑现的保证。Source materialization 遵循同样的 authority rule。 -- [opendalfs](https://github.com/fsspec/opendalfs) 通过 fsspec interface 暴露 OpenDAL services，可以作为首个 - filesystem-backed document Connector 的候选 acquisition layer。它的 path 与 file metadata 不定义 Source - identity 或 immutable revision semantics。只有完整调用链能够寻址并验证不可变 revision 时，backend read 才能 - 满足 referenced materialization；否则 document 必须被 captured。 +- [opendalfs](https://github.com/fsspec/opendalfs) 通过 fsspec interface 暴露 OpenDAL services，展示了 + backend-neutral filesystem acquisition。它的 path 与 file metadata 不定义 Source identity 或 immutable + revision semantics。只有完整调用链能够寻址并验证不可变 revision 时，backend read 才能满足 referenced + materialization。 - DataHub stateful ingestion 把 connector checkpoint 与 stale-entity detection 同 emitted metadata identity 分离。Airbyte 把 connector state 当作 opaque recovery boundary，而不是 record identity。 - OpenMetadata 把负责生成 record 的 Source 与 connection check、workflow status、sink 分离。 - Nowledge Mem 的 TiddlyWiki importer 使用 stable logical ID、canonical payload digest、source revalidation 与 - per-item outcome。这些行为为 document validation 提供依据，但不定义标准 Source value。 + per-item outcome。这些行为为 Source observation 与 Connector run state 的分离提供依据。 # Unresolved questions - 每个 durable SourceRef 是否必须直接携带 `scope_id`，还是可以由 canonical scoped envelope 包含 local exact SourceRef，同时保留相同的 fully qualified identity？ - Runtime 必须同时保留哪些 Source Definition version，才能宣称某个 Definition 受支持？ -- Source head deletion 应是通用 catalog state，还是首个标准契约只暴露 active exact head，并把 deletion 完全留给 +- Source head deletion 应是通用 catalog state，还是标准契约只暴露 active exact head，并把 deletion 完全留给 Connector state？ -- Artifact family 可以共享哪些 normalized value category，而不要求理解完整的 definition-owned value schema？ +- 哪些 projection name 与 schema 已有足够实现证据，可以成为 shared standard 而不是 namespaced capability？ +- 除本 RFC 的 lifecycle semantics 外，是否还需要标准化 Connector hosting 与 scheduling contract？ # Future possibilities -Connector lifecycle、checkpoint、run status 与显式 plugin discovery 需要独立契约。Document ingestion 为该 -契约提供 conformance case，但不能改变 Source identity 或 materialization semantics。 - -Definition 可以为无法消费完整 native value 的 Artifact family 声明 text、structured record 或 binary -attachment 等 optional projection。Projection identity 与 digest rules 需要自己的契约，且不能削弱原始 Source -observation。 +显式 plugin discovery 与 deployment policy 可以建立在 Definition 和 Connector registration 之上，但不会让 +package installation 等同于 activation。 Retention policy 只有在定义精确 Artifact evidence 如何报告 unavailable content，以及 legal/user-requested deletion 如何与 immutable lineage 交互之后，才能回收 captured value。Source head deletion 本身不授权删除证据。 From 4dc4635c8b85f6a0360b091c8e3f24ceb440740c Mon Sep 17 00:00:00 2001 From: Chojan Shang Date: Thu, 27 Aug 2026 17:10:52 +0800 Subject: [PATCH 3/9] feat(sources): add definition and connector contracts --- src/powercontext/__init__.py | 52 +++ .../builtin/persistence/sources.py | 55 ++- .../builtin/runtime/composition.py | 60 ++-- .../builtin/runtime/relational.py | 27 +- src/powercontext/builtin/sources/__init__.py | 17 + src/powercontext/builtin/sources/content.py | 32 ++ .../builtin/sources/external_skill.py | 4 +- src/powercontext/errors.py | 65 ++++ src/powercontext/sources/__init__.py | 43 ++- src/powercontext/sources/catalog.py | 90 ++--- src/powercontext/sources/connectors.py | 338 ++++++++++++++++++ src/powercontext/sources/definitions.py | 246 +++++++++++++ src/powercontext/sources/models.py | 24 +- tests/builtin/persistence/test_provider.py | 60 +++- tests/test_connectors.py | 238 ++++++++++++ tests/test_sources.py | 76 ++++ 16 files changed, 1291 insertions(+), 136 deletions(-) create mode 100644 src/powercontext/sources/connectors.py create mode 100644 src/powercontext/sources/definitions.py create mode 100644 tests/test_connectors.py diff --git a/src/powercontext/__init__.py b/src/powercontext/__init__.py index 9ae96033b..2c68383b2 100644 --- a/src/powercontext/__init__.py +++ b/src/powercontext/__init__.py @@ -27,30 +27,57 @@ ArtifactError, ArtifactFamilyMismatchError, ArtifactNotFoundError, + ConnectorError, InvalidArtifactReferenceError, + InvalidConnectorError, + InvalidConnectorRunError, InvalidSourceAdapterError, + InvalidSourceDefinitionError, InvalidSourceEntryError, + InvalidSourceProjectionError, InvalidSourceReferenceError, InvalidSourceResultError, PowerContextError, RevisionConflictError, SourceAdapterNotFoundError, SourceConflictError, + SourceDefinitionNotFoundError, SourceError, SourceNotFoundError, + SourceProjectionNotFoundError, ) from powercontext.sources import ( + AdapterSourceDefinition, + CatalogConnectorSourceSink, + Connector, + ConnectorBinding, + ConnectorCapability, + ConnectorCheckpointStore, + ConnectorItemOutcome, + ConnectorLifecycle, + ConnectorRunCompletion, + ConnectorRunResult, + ConnectorRunSession, + ConnectorRunStatus, + ConnectorSourceSink, + ConnectorSubmissionResult, + ConnectorSubmissionStatus, Source, SourceAdapter, SourceCatalog, SourceCatalogBackend, + SourceDefinition, + SourceDefinitionRegistry, SourceMaterialization, + SourceProjection, + SourceProjectionKey, SourceRef, SourceStore, ) from powercontext.triggers import PolicyTransition, Trigger __all__ = [ + "AdapterSourceDefinition", "Artifact", "ArtifactCatalog", "ArtifactDraft", @@ -61,9 +88,28 @@ "ArtifactRef", "ArtifactStore", "Artifacts", + "CatalogConnectorSourceSink", + "Connector", + "ConnectorBinding", + "ConnectorCapability", + "ConnectorCheckpointStore", + "ConnectorError", + "ConnectorItemOutcome", + "ConnectorLifecycle", + "ConnectorRunCompletion", + "ConnectorRunResult", + "ConnectorRunSession", + "ConnectorRunStatus", + "ConnectorSourceSink", + "ConnectorSubmissionResult", + "ConnectorSubmissionStatus", "InvalidArtifactReferenceError", + "InvalidConnectorError", + "InvalidConnectorRunError", "InvalidSourceAdapterError", + "InvalidSourceDefinitionError", "InvalidSourceEntryError", + "InvalidSourceProjectionError", "InvalidSourceReferenceError", "InvalidSourceResultError", "PolicyTransition", @@ -76,9 +122,15 @@ "SourceCatalog", "SourceCatalogBackend", "SourceConflictError", + "SourceDefinition", + "SourceDefinitionNotFoundError", + "SourceDefinitionRegistry", "SourceError", "SourceMaterialization", "SourceNotFoundError", + "SourceProjection", + "SourceProjectionKey", + "SourceProjectionNotFoundError", "SourceRef", "SourceStore", "Sources", diff --git a/src/powercontext/builtin/persistence/sources.py b/src/powercontext/builtin/persistence/sources.py index 7a9425702..cc1154ee0 100644 --- a/src/powercontext/builtin/persistence/sources.py +++ b/src/powercontext/builtin/persistence/sources.py @@ -34,9 +34,9 @@ StoredPayloadConflictError, ) from powercontext.builtin.persistence.tables import SOURCE_JOURNAL_HEADS_TABLE, SOURCES_TABLE -from powercontext.errors import SourceAdapterNotFoundError, SourceConflictError +from powercontext.errors import SourceDefinitionNotFoundError from powercontext.limits import MAX_SCOPE_ID_LENGTH -from powercontext.sources import Source, SourceAdapter, SourceRef +from powercontext.sources import Source, SourceAdapter, SourceDefinitionRegistry, SourceRef _AnySourceAdapter = SourceAdapter[Any, Any, Any] @@ -50,18 +50,18 @@ class StoredSource(BaseModel): class SourceRepository: - """Persist Sources using their concrete adapter routes.""" - - def __init__(self, adapters: Iterable[_AnySourceAdapter], /) -> None: - self._by_name: dict[str, _AnySourceAdapter] = {} - self._by_source: dict[type[Source], _AnySourceAdapter] = {} - for adapter in adapters: - if adapter.name in self._by_name: - raise SourceConflictError("name", adapter.name) - if adapter.source_class in self._by_source: - raise SourceConflictError("source_class", adapter.source_class) - self._by_name[adapter.name] = adapter - self._by_source[adapter.source_class] = adapter + """Persist Sources using the Runtime's fixed Source Definition registry.""" + + def __init__( + self, + definitions: SourceDefinitionRegistry | Iterable[_AnySourceAdapter], + /, + ) -> None: + self._registry = ( + definitions + if isinstance(definitions, SourceDefinitionRegistry) + else SourceDefinitionRegistry.from_adapters(definitions) + ) async def add( self, @@ -73,9 +73,9 @@ async def add( """Add one stable Source or return an identical existing capture.""" _require_identity("scope_id", scope_id, MAX_SCOPE_ID_LENGTH) - adapter = self._adapter_for_value(source) - ref = SourceRef(source_type=adapter.name, source_id=source.name) - payload = dump_model(source, kind="source", name=adapter.name) + definition = self._registry.definition_for_source(source) + ref = SourceRef(source_type=definition.name, source_id=source.name) + payload = dump_model(source, kind="source", name=definition.name) await _lock_journal_head(connection, scope_id) existing = await self._find_row(connection, scope_id, ref) if existing is not None: @@ -163,17 +163,11 @@ async def journal_position(self, connection: AsyncConnection, scope_id: str, /) raise InvalidStoredColumnError("journal_position", "an integer") return int(value) - def _adapter_for_value(self, source: Source) -> _AnySourceAdapter: - try: - return self._by_source[type(source)] - except KeyError: - raise SourceAdapterNotFoundError("source", type(source)) from None - - def _adapter_by_name(self, name: str) -> _AnySourceAdapter: + def _definition_by_name(self, name: str) -> _AnySourceAdapter: try: - return self._by_name[name] - except KeyError: - raise RepositoryNotFoundError("source-adapter", name) from None + return self._registry.definition_for_name(name) + except SourceDefinitionNotFoundError: + raise RepositoryNotFoundError("source-definition", name) from None async def _find_row( self, @@ -198,15 +192,16 @@ async def _find_row( def _decode_row(self, row: Mapping[Any, Any]) -> StoredSource: source_type = str(row["source_type"]) source_id = str(row["source_id"]) - adapter = self._adapter_by_name(source_type) + definition = self._definition_by_name(source_type) source = load_model( - adapter.source_class, + definition.source_class, stored_bytes(row["payload"], column="payload"), kind="source", name=source_type, ) indexed = SourceRef(source_type=source_type, source_id=source_id) - decoded = SourceRef(source_type=adapter.name, source_id=source.name) + self._registry.definition_for_source(source) + decoded = SourceRef(source_type=definition.name, source_id=source.name) if indexed != decoded: raise IdentityMismatchError("source", indexed, decoded) return StoredSource( diff --git a/src/powercontext/builtin/runtime/composition.py b/src/powercontext/builtin/runtime/composition.py index 7c623b068..c8419a80a 100644 --- a/src/powercontext/builtin/runtime/composition.py +++ b/src/powercontext/builtin/runtime/composition.py @@ -72,8 +72,9 @@ dependency_readiness_probe, ) from powercontext.builtin.runtime.relational import RelationalContexts -from powercontext.builtin.sources import CONTENT_SOURCE_NAME, ContentSource -from powercontext.sources import Source +from powercontext.builtin.sources import BUILTIN_SOURCE_REGISTRY, TEXT_EVIDENCE_PROJECTION_KEY +from powercontext.errors import SourceProjectionNotFoundError +from powercontext.sources import Source, SourceDefinitionRegistry, SourceProjectionKey if TYPE_CHECKING: from pydantic_ai.models.instrumented import InstrumentationSettings @@ -96,30 +97,30 @@ def __init__(self, issue: str) -> None: super().__init__(messages[issue]) -class _ContentEvidenceProjector(DefaultMemoryEvidenceProjector): +class _DefinitionEvidenceProjector(DefaultMemoryEvidenceProjector): + def __init__(self, definitions: SourceDefinitionRegistry, projection: SourceProjectionKey) -> None: + self._definitions = definitions + self._projection = projection + @override def project_source(self, source: Source, /) -> JsonValue: - if isinstance(source, ContentSource): - return { - "source_type": CONTENT_SOURCE_NAME, - "source_id": source.name, - "content": source.content, - "metadata": source.model_dump(mode="json")["metadata"], - } - return super().project_source(source) + try: + return self._definitions.project(source, self._projection) + except SourceProjectionNotFoundError: + return super().project_source(source) + +class _DefinitionHandoffEvidenceProjector(DefaultHandoffEvidenceProjector): + def __init__(self, definitions: SourceDefinitionRegistry, projection: SourceProjectionKey) -> None: + self._definitions = definitions + self._projection = projection -class _ContentHandoffEvidenceProjector(DefaultHandoffEvidenceProjector): @override def project_source(self, source: Source, /) -> JsonValue: - if isinstance(source, ContentSource): - return { - "source_type": CONTENT_SOURCE_NAME, - "source_id": source.name, - "content": source.content, - "metadata": source.model_dump(mode="json")["metadata"], - } - return super().project_source(source) + try: + return self._definitions.project(source, self._projection) + except SourceProjectionNotFoundError: + return super().project_source(source) class _TracingMemoryReranker: @@ -170,10 +171,12 @@ async def open_builtin_runtime( instrumentation: InstrumentationSettings | None = None, scope_cache_observer: ScopeCacheObserver | None = None, tracing: RuntimeTracing | None = None, + source_registry: SourceDefinitionRegistry | None = None, ) -> AsyncIterator[BuiltinRuntime]: """Open the selected database, inference adapters, and built-in runtime.""" async with AsyncExitStack() as resources: + configured_source_registry = source_registry or BUILTIN_SOURCE_REGISTRY ( generated_memory, generated_incubation, @@ -183,7 +186,13 @@ async def open_builtin_runtime( generated_reranker, generation_readiness, ) = ( - await _generation_pipelines(config.inference, config.runtime, resources, instrumentation) + await _generation_pipelines( + config.inference, + config.runtime, + resources, + instrumentation, + configured_source_registry, + ) if ( candidate_pipeline is None or experience_pipeline is None @@ -237,6 +246,7 @@ async def open_builtin_runtime( embedding_model=configured_embedding, token_estimator=token_estimator, memory_reranker=configured_reranker, + source_registry=configured_source_registry, ) ) readiness_probes: dict[str, ReadinessProbeDefinition] = { @@ -318,6 +328,7 @@ async def open_builtin_contexts( embedding_model: EmbeddingModel | None = None, token_estimator: TokenEstimator | None = None, memory_reranker: MemoryReranker | None = None, + source_registry: SourceDefinitionRegistry | None = None, ) -> AsyncIterator[RelationalContexts]: """Open the selected database and expose scope-bound PowerContext providers.""" @@ -352,6 +363,7 @@ async def open_builtin_contexts( token_estimator=configured_token_estimator, memory_reranker=memory_reranker, memory_rerank_candidate_limit=config.runtime.memory_rerank_candidate_limit, + source_registry=source_registry, ) return experience_index = OceanBaseExperienceFTSIndex() @@ -384,6 +396,7 @@ async def open_builtin_contexts( token_estimator=configured_token_estimator, memory_reranker=memory_reranker, memory_rerank_candidate_limit=config.runtime.memory_rerank_candidate_limit, + source_registry=source_registry, ) @@ -392,6 +405,7 @@ async def _generation_pipelines( runtime: RuntimeConfig, resources: AsyncExitStack, instrumentation: InstrumentationSettings | None, + source_registry: SourceDefinitionRegistry, ) -> tuple[ CandidatePipeline | None, ExperienceCandidatePipeline | None, @@ -512,14 +526,14 @@ async def probe_generation() -> None: return ( LLMMemoryCandidatePipeline( UsageReportingStructuredGenerator(memory_generator), - evidence_projector=_ContentEvidenceProjector(), + evidence_projector=_DefinitionEvidenceProjector(source_registry, TEXT_EVIDENCE_PROJECTION_KEY), ), LLMExperienceCandidatePipeline(UsageReportingStructuredGenerator(experience_generator)), LLMExperienceGenerator(UsageReportingStructuredGenerator(explicit_experience_generator)), LLMSkillGenerator(UsageReportingStructuredGenerator(skill_generator)), LLMHandoffGenerationPipeline( UsageReportingStructuredGenerator(handoff_generator), - evidence_projector=_ContentHandoffEvidenceProjector(), + evidence_projector=_DefinitionHandoffEvidenceProjector(source_registry, TEXT_EVIDENCE_PROJECTION_KEY), ), (None if rerank_generator is None else LLMMemoryReranker(UsageReportingStructuredGenerator(rerank_generator))), CachedReadinessProbe(dependency_readiness_probe(probe_generation)), diff --git a/src/powercontext/builtin/runtime/relational.py b/src/powercontext/builtin/runtime/relational.py index 534a828f5..120dda956 100644 --- a/src/powercontext/builtin/runtime/relational.py +++ b/src/powercontext/builtin/runtime/relational.py @@ -19,7 +19,7 @@ import asyncio from collections.abc import Callable from dataclasses import dataclass -from typing import Any, cast +from typing import cast from uuid import uuid4 from sqlalchemy import select @@ -90,7 +90,7 @@ from powercontext.builtin.runtime.recall import RelationalRecallTokenEstimator from powercontext.builtin.runtime.statistics import RelationalScopedStatistics from powercontext.builtin.sources import ( - CONTENT_SOURCE_ADAPTER, + BUILTIN_SOURCE_REGISTRY, EXTERNAL_SKILL_SNAPSHOT_SOURCE_ADAPTER, ExternalSkillImportMode, ExternalSkillSnapshotCapture, @@ -112,16 +112,12 @@ from powercontext.errors import ArtifactNotFoundError, SourceConflictError, SourceNotFoundError from powercontext.sources import ( Source, - SourceAdapter, SourceCatalog, + SourceDefinitionRegistry, SourceRef, ) IdFactory = Callable[[str], str] -_SOURCE_ADAPTERS: tuple[SourceAdapter[Any, Any, Any], ...] = ( - CONTENT_SOURCE_ADAPTER, - EXTERNAL_SKILL_SNAPSHOT_SOURCE_ADAPTER, -) @dataclass(frozen=True, slots=True) @@ -158,6 +154,7 @@ class _ScopedServices: memory_artifact_id: str source_lock: asyncio.Lock token_estimator: TokenEstimator | None + source_registry: SourceDefinitionRegistry def sources( self, @@ -166,12 +163,12 @@ def sources( backend = _RelationalSources( database=self.database, scope_id=self.scope_id, - adapters=_SOURCE_ADAPTERS, + registry=self.source_registry, repository=self.repositories.sources, write_lock=self.source_lock, connection=connection, ) - return backend, SourceCatalog(backend=backend, adapters=_SOURCE_ADAPTERS) + return backend, SourceCatalog(backend=backend, registry=self.source_registry) def memory( self, @@ -303,12 +300,14 @@ def __init__( id_factory: IdFactory | None = None, handoff_artifact_id: str = "handoff", memory_artifact_id: str = "memory", + source_registry: SourceDefinitionRegistry | None = None, ) -> None: self.database = database + self.source_registry = source_registry or BUILTIN_SOURCE_REGISTRY self.index = NoMemoryIndex() if index is None else index self.experience_index = NoExperienceIndex() if experience_index is None else experience_index self.repositories = _Repositories( - sources=SourceRepository(_SOURCE_ADAPTERS), + sources=SourceRepository(self.source_registry), artifacts=ArtifactRepository((Handoff, Memory, Experience, Skill)), candidates=CandidateRepository({ Experience.family: ExperienceContent, @@ -522,6 +521,7 @@ def _services_for(self, scope_id: str) -> _ScopedServices: memory_artifact_id=self._memory_artifact_id, source_lock=self._source_locks.setdefault(scope, asyncio.Lock()), token_estimator=self._token_estimator, + source_registry=self.source_registry, ) @@ -531,14 +531,14 @@ def __init__( *, database: AsyncDatabase, scope_id: str, - adapters: tuple[SourceAdapter[Any, Any, Any], ...], + registry: SourceDefinitionRegistry, repository: SourceRepository, write_lock: asyncio.Lock, connection: AsyncConnection | None = None, ) -> None: self._database = database self._scope_id = scope_id - self._source_names = {adapter.source_class: adapter.name for adapter in adapters} + self._registry = registry self._repository = repository self._write_lock = write_lock self._bound_connection = connection @@ -585,7 +585,8 @@ async def entries(self) -> tuple[SourceJournalEntry, ...]: ) def _as_ref(self, source: Source) -> SourceRef: - return SourceRef(source_type=self._source_names[type(source)], source_id=source.name) + definition = self._registry.definition_for_source(source) + return SourceRef(source_type=definition.name, source_id=source.name) class _RelationalArtifactResolver: diff --git a/src/powercontext/builtin/sources/__init__.py b/src/powercontext/builtin/sources/__init__.py index d596a4588..109a21316 100644 --- a/src/powercontext/builtin/sources/__init__.py +++ b/src/powercontext/builtin/sources/__init__.py @@ -16,13 +16,18 @@ from powercontext.builtin.sources.content import ( CONTENT_SOURCE_ADAPTER, + CONTENT_SOURCE_DEFINITION, CONTENT_SOURCE_NAME, + TEXT_EVIDENCE_PROJECTION_KEY, ContentCapture, ContentSource, ContentSourceAdapter, + ContentTextEvidence, + ContentTextEvidenceProjection, ) from powercontext.builtin.sources.external_skill import ( EXTERNAL_SKILL_SNAPSHOT_SOURCE_ADAPTER, + EXTERNAL_SKILL_SNAPSHOT_SOURCE_DEFINITION, EXTERNAL_SKILL_SNAPSHOT_SOURCE_NAME, ExternalSkillImportMode, ExternalSkillSnapshotCapture, @@ -35,15 +40,27 @@ SourceJournalEntry, validate_scope_id, ) +from powercontext.sources import SourceDefinitionRegistry + +BUILTIN_SOURCE_REGISTRY = SourceDefinitionRegistry(( + CONTENT_SOURCE_DEFINITION, + EXTERNAL_SKILL_SNAPSHOT_SOURCE_DEFINITION, +)) __all__ = [ + "BUILTIN_SOURCE_REGISTRY", "CONTENT_SOURCE_ADAPTER", + "CONTENT_SOURCE_DEFINITION", "CONTENT_SOURCE_NAME", "EXTERNAL_SKILL_SNAPSHOT_SOURCE_ADAPTER", + "EXTERNAL_SKILL_SNAPSHOT_SOURCE_DEFINITION", "EXTERNAL_SKILL_SNAPSHOT_SOURCE_NAME", + "TEXT_EVIDENCE_PROJECTION_KEY", "ContentCapture", "ContentSource", "ContentSourceAdapter", + "ContentTextEvidence", + "ContentTextEvidenceProjection", "ExternalSkillImportMode", "ExternalSkillSnapshotCapture", "ExternalSkillSnapshotSource", diff --git a/src/powercontext/builtin/sources/content.py b/src/powercontext/builtin/sources/content.py index d516f0471..c444b8020 100644 --- a/src/powercontext/builtin/sources/content.py +++ b/src/powercontext/builtin/sources/content.py @@ -20,9 +20,11 @@ from pydantic import BaseModel, Field, JsonValue, field_validator +from powercontext.sources import AdapterSourceDefinition, SourceProjectionKey from powercontext.sources.models import Source, SourceMaterialization CONTENT_SOURCE_NAME = "content" +TEXT_EVIDENCE_PROJECTION_KEY = SourceProjectionKey(name="powercontext.builtin.text-evidence", version="1") NonEmptyText = Annotated[str, Field(min_length=1)] @@ -48,6 +50,15 @@ class ContentSource(Source): metadata: dict[str, JsonValue] = Field(default_factory=dict) +class ContentTextEvidence(BaseModel): + """Schema for the built-in text evidence projection.""" + + source_type: str + source_id: str + content: str + metadata: dict[str, JsonValue] = Field(default_factory=dict) + + class ContentSourceAdapter: """Resolve and read the runtime's built-in captured-text Source.""" @@ -71,4 +82,25 @@ async def read(self, source: ContentSource, /) -> ContentCapture: ) +class ContentTextEvidenceProjection: + """Expose captured text without coupling consumers to ``ContentSource``.""" + + name = TEXT_EVIDENCE_PROJECTION_KEY.name + version = TEXT_EVIDENCE_PROJECTION_KEY.version + source_class = ContentSource + output_class: type[BaseModel] = ContentTextEvidence + + def project(self, source: ContentSource, /) -> ContentTextEvidence: + return ContentTextEvidence( + source_type=CONTENT_SOURCE_NAME, + source_id=source.name, + content=source.content, + metadata=source.metadata, + ) + + CONTENT_SOURCE_ADAPTER = ContentSourceAdapter() +CONTENT_SOURCE_DEFINITION = AdapterSourceDefinition( + CONTENT_SOURCE_ADAPTER, + projections=(ContentTextEvidenceProjection(),), +) diff --git a/src/powercontext/builtin/sources/external_skill.py b/src/powercontext/builtin/sources/external_skill.py index 2aa9d6ad4..f49ccbb84 100644 --- a/src/powercontext/builtin/sources/external_skill.py +++ b/src/powercontext/builtin/sources/external_skill.py @@ -22,7 +22,7 @@ from pydantic import BaseModel from powercontext.builtin.artifacts.skill import ExternalSkillSnapshot -from powercontext.sources import Source, SourceMaterialization +from powercontext.sources import AdapterSourceDefinition, Source, SourceMaterialization EXTERNAL_SKILL_SNAPSHOT_SOURCE_NAME = "external-skill-snapshot" @@ -82,9 +82,11 @@ def _snapshot_id(value: ExternalSkillSnapshotCapture) -> str: EXTERNAL_SKILL_SNAPSHOT_SOURCE_ADAPTER = ExternalSkillSnapshotSourceAdapter() +EXTERNAL_SKILL_SNAPSHOT_SOURCE_DEFINITION = AdapterSourceDefinition(EXTERNAL_SKILL_SNAPSHOT_SOURCE_ADAPTER) __all__ = [ "EXTERNAL_SKILL_SNAPSHOT_SOURCE_ADAPTER", + "EXTERNAL_SKILL_SNAPSHOT_SOURCE_DEFINITION", "EXTERNAL_SKILL_SNAPSHOT_SOURCE_NAME", "ExternalSkillImportMode", "ExternalSkillSnapshotCapture", diff --git a/src/powercontext/errors.py b/src/powercontext/errors.py index 81214d4e9..172032796 100644 --- a/src/powercontext/errors.py +++ b/src/powercontext/errors.py @@ -99,6 +99,71 @@ def __init__( ) +class InvalidSourceDefinitionError(SourceError, TypeError): + """Raised when a Source Definition violates its registration contract.""" + + def __init__(self, definition_type: type[object], field: str, detail: str) -> None: + self.definition_type = definition_type + self.field = field + self.detail = detail + super().__init__(f"invalid Source Definition {_type_name(definition_type)} {field}: {detail}") + + +class SourceDefinitionNotFoundError(SourceError, LookupError): + """Raised when the active registry does not contain a Source Definition.""" + + def __init__(self, name: str, version: str | None = None) -> None: + self.name = name + self.version = version + suffix = "" if version is None else f" version {version!r}" + super().__init__(f"Source Definition {name!r}{suffix} is not registered") + + +class SourceProjectionNotFoundError(SourceError, LookupError): + """Raised when a Source Definition does not provide a requested projection.""" + + def __init__(self, source_type: str, projection_name: str, projection_version: str) -> None: + self.source_type = source_type + self.projection_name = projection_name + self.projection_version = projection_version + super().__init__( + f"Source Definition {source_type!r} does not provide projection " + f"{projection_name!r} version {projection_version!r}" + ) + + +class InvalidSourceProjectionError(SourceError, TypeError): + """Raised when a named Source projection violates its declared contract.""" + + def __init__(self, projection_name: str, field: str, detail: str) -> None: + self.projection_name = projection_name + self.field = field + self.detail = detail + super().__init__(f"invalid Source projection {projection_name!r} {field}: {detail}") + + +class ConnectorError(PowerContextError): + """Base exception for Connector contracts and run lifecycle failures.""" + + +class InvalidConnectorError(ConnectorError, TypeError): + """Raised when a Connector or binding violates its declared contract.""" + + def __init__(self, field: str, detail: str) -> None: + self.field = field + self.detail = detail + super().__init__(f"invalid Connector {field}: {detail}") + + +class InvalidConnectorRunError(ConnectorError, RuntimeError): + """Raised when a Connector run would violate replay or checkpoint safety.""" + + def __init__(self, issue: str, detail: str) -> None: + self.issue = issue + self.detail = detail + super().__init__(f"invalid Connector run {issue}: {detail}") + + class ArtifactError(PowerContextError): """Base exception for Artifact lookup and lifecycle failures.""" diff --git a/src/powercontext/sources/__init__.py b/src/powercontext/sources/__init__.py index ead637bd8..fed1522b2 100644 --- a/src/powercontext/sources/__init__.py +++ b/src/powercontext/sources/__init__.py @@ -14,15 +14,56 @@ from powercontext.sources.adapters import SourceAdapter from powercontext.sources.catalog import SourceCatalog -from powercontext.sources.models import Source, SourceMaterialization, SourceRef +from powercontext.sources.connectors import ( + CatalogConnectorSourceSink, + Connector, + ConnectorBinding, + ConnectorCapability, + ConnectorCheckpointStore, + ConnectorItemOutcome, + ConnectorLifecycle, + ConnectorRunCompletion, + ConnectorRunResult, + ConnectorRunSession, + ConnectorRunStatus, + ConnectorSourceSink, + ConnectorSubmissionResult, + ConnectorSubmissionStatus, +) +from powercontext.sources.definitions import ( + AdapterSourceDefinition, + SourceDefinition, + SourceDefinitionRegistry, + SourceProjection, +) +from powercontext.sources.models import Source, SourceMaterialization, SourceProjectionKey, SourceRef from powercontext.sources.protocols import SourceCatalogBackend, SourceStore __all__ = [ + "AdapterSourceDefinition", + "CatalogConnectorSourceSink", + "Connector", + "ConnectorBinding", + "ConnectorCapability", + "ConnectorCheckpointStore", + "ConnectorItemOutcome", + "ConnectorLifecycle", + "ConnectorRunCompletion", + "ConnectorRunResult", + "ConnectorRunSession", + "ConnectorRunStatus", + "ConnectorSourceSink", + "ConnectorSubmissionResult", + "ConnectorSubmissionStatus", "Source", "SourceAdapter", "SourceCatalog", "SourceCatalogBackend", + "SourceDefinition", + "SourceDefinitionRegistry", "SourceMaterialization", + "SourceProjection", + "SourceProjectionKey", "SourceRef", "SourceStore", ] diff --git a/src/powercontext/sources/catalog.py b/src/powercontext/sources/catalog.py index 37a04238b..182446f18 100644 --- a/src/powercontext/sources/catalog.py +++ b/src/powercontext/sources/catalog.py @@ -14,19 +14,17 @@ from __future__ import annotations -from collections.abc import Iterable, Mapping -from typing import Any, cast +from collections.abc import Iterable +from typing import Any + +from pydantic import JsonValue from powercontext.errors import ( - InvalidSourceAdapterError, - InvalidSourceEntryError, - InvalidSourceResultError, - SourceAdapterNotFoundError, - SourceConflictError, SourceNotFoundError, ) from powercontext.sources.adapters import SourceAdapter -from powercontext.sources.models import Source, SourceRef +from powercontext.sources.definitions import SourceDefinitionRegistry +from powercontext.sources.models import Source, SourceProjectionKey, SourceRef from powercontext.sources.protocols import SourceCatalogBackend _AnySourceAdapter = SourceAdapter[Any, Any, Any] @@ -39,21 +37,14 @@ def __init__( self, *, backend: SourceCatalogBackend, - adapters: Iterable[_AnySourceAdapter], + adapters: Iterable[_AnySourceAdapter] = (), + registry: SourceDefinitionRegistry | None = None, ) -> None: - by_input: dict[type[object], _AnySourceAdapter] = {} - by_source: dict[type[Source], _AnySourceAdapter] = {} - for adapter in adapters: - input_class, source_class = _validate_adapter(adapter) - if input_class in by_input: - raise SourceConflictError("input_class", input_class) - if source_class in by_source: - raise SourceConflictError("source_class", source_class) - by_input[input_class] = adapter - by_source[source_class] = adapter + adapter_values = tuple(adapters) + if registry is not None and adapter_values: + raise TypeError("SourceCatalog accepts either registry or adapters, not both") # noqa: TRY003 self._backend = backend - self._by_input = by_input - self._by_source = by_source + self._registry = registry or SourceDefinitionRegistry.from_adapters(adapter_values) async def list(self) -> tuple[Source, ...]: sources = await self._backend.list() @@ -70,50 +61,21 @@ async def get(self, source: Source, /) -> Source: return stored def as_ref(self, source: Source, /) -> SourceRef: - adapter = _adapter_for_source(source, self._by_source) - return SourceRef(source_type=adapter.name, source_id=source.name) + definition = self._registry.definition_for_source(source) + return SourceRef(source_type=definition.name, source_id=source.name) async def resolve(self, value: object, /) -> Source: - input_class = type(value) - try: - adapter = self._by_input[input_class] - except KeyError: - raise SourceAdapterNotFoundError("input", input_class) from None - source = await adapter.resolve(value) - if type(source) is not adapter.source_class: - raise InvalidSourceResultError(adapter.name, "resolve", adapter.source_class, type(source)) - self.as_ref(source) - return cast(Source, source) + return await self._registry.resolve(value) async def read(self, source: Source, /) -> object: - adapter = _adapter_for_source(source, self._by_source) - return await adapter.read(source) - - -def _validate_adapter(adapter: object) -> tuple[type[object], type[Source]]: - adapter_type = type(adapter) - input_class = getattr(adapter, "input_class", None) - if not isinstance(input_class, type): - raise InvalidSourceAdapterError(adapter_type, "input_class", "must be a type") - name = getattr(adapter, "name", None) - if not isinstance(name, str) or not name.strip(): - raise InvalidSourceAdapterError(adapter_type, "name", "must be a non-empty string") - source_class = getattr(adapter, "source_class", None) - if not isinstance(source_class, type) or not issubclass(source_class, Source): - raise InvalidSourceAdapterError(adapter_type, "source_class", "must be a Source subclass") - for method_name in ("resolve", "read"): - if not callable(getattr(adapter, method_name, None)): - raise InvalidSourceAdapterError(adapter_type, method_name, "must be callable") - return cast(type[object], input_class), cast(type[Source], source_class) - - -def _adapter_for_source( - source: object, - adapters: Mapping[type[Source], _AnySourceAdapter], -) -> _AnySourceAdapter: - if not isinstance(source, Source): - raise InvalidSourceEntryError(type(source)) - try: - return adapters[type(source)] - except KeyError: - raise SourceAdapterNotFoundError("source", type(source)) from None + return await self._registry.read(source) + + def projection_keys(self, source: Source, /) -> tuple[SourceProjectionKey, ...]: + """Return the exact named projection capabilities advertised for ``source``.""" + + return self._registry.projection_keys(source) + + def project(self, source: Source, key: SourceProjectionKey, /) -> JsonValue: + """Evaluate one named projection against an exact Source value.""" + + return self._registry.project(source, key) diff --git a/src/powercontext/sources/connectors.py b/src/powercontext/sources/connectors.py new file mode 100644 index 000000000..774bc85d8 --- /dev/null +++ b/src/powercontext/sources/connectors.py @@ -0,0 +1,338 @@ +# Copyright (c) 2026 OceanBase. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Provider-neutral Connector run and durable checkpoint contracts.""" + +from __future__ import annotations + +from enum import StrEnum +from typing import Protocol + +from pydantic import BaseModel, ConfigDict, Field, JsonValue, field_validator, model_validator + +from powercontext.errors import InvalidConnectorError, InvalidConnectorRunError +from powercontext.limits import MAX_SCOPE_ID_LENGTH, MAX_SOURCE_ID_LENGTH, MAX_SOURCE_TYPE_LENGTH +from powercontext.sources.catalog import SourceCatalog +from powercontext.sources.models import Source, SourceRef +from powercontext.sources.protocols import SourceStore + + +class ConnectorCapability(StrEnum): + """Acquisition guarantees a Connector can actually enforce.""" + + COMPLETE_SNAPSHOT = "complete_snapshot" + CHANGE_FEED = "change_feed" + CHECKPOINT_RESUME = "checkpoint_resume" + AUTHORITATIVE_DELETION = "authoritative_deletion" + + +class ConnectorRunStatus(StrEnum): + """Whether the Connector completed the provider work represented by a run.""" + + COMPLETE = "complete" + INCOMPLETE = "incomplete" + + +class ConnectorSubmissionStatus(StrEnum): + """Durable outcome for one definition-native item submission.""" + + ACCEPTED = "accepted" + REPLAYED = "replayed" + REJECTED = "rejected" + FAILED = "failed" + + +class ConnectorBinding(BaseModel): + """Activate one Connector identity for exactly one Scope.""" + + model_config = ConfigDict(frozen=True) + + scope_id: str = Field(max_length=MAX_SCOPE_ID_LENGTH) + binding_id: str = Field(max_length=MAX_SOURCE_ID_LENGTH) + connector_name: str = Field(max_length=MAX_SOURCE_TYPE_LENGTH) + connector_version: str = Field(max_length=MAX_SOURCE_TYPE_LENGTH) + + @field_validator("scope_id", "binding_id", "connector_name", "connector_version") + @classmethod + def validate_identity(cls, value: str) -> str: + if not value or not value.strip(): + raise ValueError("Connector identity must be non-empty") # noqa: TRY003 + if value != value.strip(): + raise ValueError("Connector identity must be trimmed") # noqa: TRY003 + return value + + +class ConnectorSubmissionResult(BaseModel): + """Sink result after one item has reached a durable acceptance boundary.""" + + model_config = ConfigDict(frozen=True) + + status: ConnectorSubmissionStatus + source_ref: SourceRef | None = None + detail: str | None = None + + @model_validator(mode="after") + def validate_source_ref(self) -> ConnectorSubmissionResult: + accepted = self.status in {ConnectorSubmissionStatus.ACCEPTED, ConnectorSubmissionStatus.REPLAYED} + if accepted != (self.source_ref is not None): + raise ValueError("accepted and replayed submissions require exactly one SourceRef") # noqa: TRY003 + return self + + +class ConnectorItemOutcome(BaseModel): + """Visible result for every item a Connector submitted during one run.""" + + model_config = ConfigDict(frozen=True) + + item_id: str + definition_name: str + status: ConnectorSubmissionStatus + source_ref: SourceRef | None = None + detail: str | None = None + + +class ConnectorRunCompletion(BaseModel): + """Connector-owned completion signal and next opaque checkpoint.""" + + model_config = ConfigDict(frozen=True) + + status: ConnectorRunStatus + checkpoint: JsonValue | None = None + + +class ConnectorRunResult(BaseModel): + """Observable lifecycle result after any safe checkpoint commit.""" + + model_config = ConfigDict(frozen=True) + + binding: ConnectorBinding + status: ConnectorRunStatus + previous_checkpoint: JsonValue | None + proposed_checkpoint: JsonValue | None + committed_checkpoint: JsonValue | None + items: tuple[ConnectorItemOutcome, ...] + + +class ConnectorSourceSink(Protocol): + """Accept definition-native input and return its durable local SourceRef.""" + + async def submit( + self, + binding: ConnectorBinding, + item_id: str, + definition_name: str, + value: object, + /, + ) -> ConnectorSubmissionResult: ... + + +class ConnectorCheckpointStore(Protocol): + """Persist opaque binding checkpoints using optimistic comparison.""" + + async def load(self, binding: ConnectorBinding, /) -> JsonValue | None: ... + + async def save( + self, + binding: ConnectorBinding, + checkpoint: JsonValue | None, + /, + *, + expected: JsonValue | None, + ) -> None: ... + + +class Connector(Protocol): + """Acquire provider items through one lifecycle session.""" + + name: str + version: str + source_definitions: frozenset[str] + capabilities: frozenset[ConnectorCapability] + + async def run(self, session: ConnectorRunSession, /) -> ConnectorRunCompletion: ... + + +class ConnectorRunSession: + """Constrain one Connector run to declared Definitions and visible outcomes.""" + + def __init__( + self, + *, + binding: ConnectorBinding, + checkpoint: JsonValue | None, + source_definitions: frozenset[str], + sink: ConnectorSourceSink, + ) -> None: + self.binding = binding + self.checkpoint = checkpoint + self._source_definitions = source_definitions + self._sink = sink + self._outcomes: list[ConnectorItemOutcome] = [] + self._item_ids: set[str] = set() + + @property + def outcomes(self) -> tuple[ConnectorItemOutcome, ...]: + return tuple(self._outcomes) + + async def submit( + self, + item_id: str, + definition_name: str, + value: object, + /, + ) -> ConnectorSubmissionResult: + """Submit one item and record success, rejection, or sink failure exactly once.""" + + _require_trimmed("item_id", item_id) + if item_id in self._item_ids: + raise InvalidConnectorRunError("duplicate-item", f"item {item_id!r} was submitted more than once") + if definition_name not in self._source_definitions: + raise InvalidConnectorRunError( + "undeclared-definition", + f"Connector did not declare Source Definition {definition_name!r}", + ) + self._item_ids.add(item_id) + try: + result = await self._sink.submit(self.binding, item_id, definition_name, value) + except Exception as error: + result = ConnectorSubmissionResult( + status=ConnectorSubmissionStatus.FAILED, + detail=type(error).__name__, + ) + if result.source_ref is not None and result.source_ref.source_type != definition_name: + raise InvalidConnectorRunError( + "definition-mismatch", + f"sink returned {result.source_ref.source_type!r} for {definition_name!r}", + ) + self._outcomes.append( + ConnectorItemOutcome( + item_id=item_id, + definition_name=definition_name, + status=result.status, + source_ref=result.source_ref, + detail=result.detail, + ) + ) + return result + + +class ConnectorLifecycle: + """Run Connectors while enforcing durable checkpoint ordering.""" + + def __init__(self, *, sink: ConnectorSourceSink, checkpoints: ConnectorCheckpointStore) -> None: + self._sink = sink + self._checkpoints = checkpoints + + async def run(self, connector: Connector, binding: ConnectorBinding, /) -> ConnectorRunResult: + source_definitions, capabilities = _validate_connector(connector, binding) + previous = await self._checkpoints.load(binding) + if previous is not None and ConnectorCapability.CHECKPOINT_RESUME not in capabilities: + raise InvalidConnectorRunError( + "unsupported-resume", + "binding has a checkpoint but Connector does not advertise checkpoint resume", + ) + session = ConnectorRunSession( + binding=binding, + checkpoint=previous, + source_definitions=source_definitions, + sink=self._sink, + ) + completion = await connector.run(session) + if not isinstance(completion, ConnectorRunCompletion): + raise InvalidConnectorRunError("completion", "Connector must return ConnectorRunCompletion") + + unsafe = tuple( + outcome + for outcome in session.outcomes + if outcome.status in {ConnectorSubmissionStatus.REJECTED, ConnectorSubmissionStatus.FAILED} + ) + committed = previous + if completion.checkpoint != previous and not unsafe: + await self._checkpoints.save(binding, completion.checkpoint, expected=previous) + committed = completion.checkpoint + return ConnectorRunResult( + binding=binding, + status=ConnectorRunStatus.INCOMPLETE if unsafe else completion.status, + previous_checkpoint=previous, + proposed_checkpoint=completion.checkpoint, + committed_checkpoint=committed, + items=session.outcomes, + ) + + +class CatalogConnectorSourceSink: + """Bridge lifecycle submissions to one scope-bound catalog and Source store.""" + + def __init__(self, *, scope_id: str, catalog: SourceCatalog, store: SourceStore[Source]) -> None: + _require_trimmed("scope_id", scope_id) + self._scope_id = scope_id + self._catalog = catalog + self._store = store + + async def submit( + self, + binding: ConnectorBinding, + item_id: str, + definition_name: str, + value: object, + /, + ) -> ConnectorSubmissionResult: + del item_id + if binding.scope_id != self._scope_id: + raise InvalidConnectorRunError( + "scope-mismatch", + f"sink is bound to {self._scope_id!r}, got {binding.scope_id!r}", + ) + source = await self._catalog.resolve(value) + source_ref = self._catalog.as_ref(source) + if source_ref.source_type != definition_name: + raise InvalidConnectorRunError( + "definition-mismatch", + f"input resolved as {source_ref.source_type!r}, expected {definition_name!r}", + ) + stored = await self._store.add(source) + stored_ref = self._catalog.as_ref(stored) + if stored_ref != source_ref: + raise InvalidConnectorRunError("identity-mismatch", "Source store changed the accepted identity") + return ConnectorSubmissionResult(status=ConnectorSubmissionStatus.ACCEPTED, source_ref=stored_ref) + + +def _validate_connector( + connector: Connector, + binding: ConnectorBinding, +) -> tuple[frozenset[str], frozenset[ConnectorCapability]]: + name = getattr(connector, "name", None) + version = getattr(connector, "version", None) + if name != binding.connector_name: + raise InvalidConnectorError("name", f"binding expects {binding.connector_name!r}, got {name!r}") + if version != binding.connector_version: + raise InvalidConnectorError("version", f"binding expects {binding.connector_version!r}, got {version!r}") + source_definitions = getattr(connector, "source_definitions", None) + if not isinstance(source_definitions, frozenset) or not source_definitions: + raise InvalidConnectorError("source_definitions", "must be a non-empty frozenset") + if not all(isinstance(value, str) and value.strip() == value and value for value in source_definitions): + raise InvalidConnectorError("source_definitions", "must contain non-empty trimmed names") + capabilities = getattr(connector, "capabilities", None) + if not isinstance(capabilities, frozenset) or not all( + isinstance(value, ConnectorCapability) for value in capabilities + ): + raise InvalidConnectorError("capabilities", "must be a frozenset of ConnectorCapability values") + if not callable(getattr(connector, "run", None)): + raise InvalidConnectorError("run", "must be callable") + return source_definitions, capabilities + + +def _require_trimmed(field: str, value: object) -> None: + if not isinstance(value, str) or not value or value.strip() != value: + raise InvalidConnectorRunError(field, "must be a non-empty trimmed string") diff --git a/src/powercontext/sources/definitions.py b/src/powercontext/sources/definitions.py new file mode 100644 index 000000000..a67fcd2a9 --- /dev/null +++ b/src/powercontext/sources/definitions.py @@ -0,0 +1,246 @@ +# Copyright (c) 2026 OceanBase. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Explicit Source Definition registration and named projection routing.""" + +from __future__ import annotations + +from collections.abc import Iterable, Mapping +from dataclasses import dataclass +from typing import Any, Generic, Protocol, TypeVar, cast + +from pydantic import BaseModel, JsonValue, TypeAdapter + +from powercontext.errors import ( + InvalidSourceAdapterError, + InvalidSourceDefinitionError, + InvalidSourceEntryError, + InvalidSourceProjectionError, + InvalidSourceResultError, + SourceAdapterNotFoundError, + SourceConflictError, + SourceDefinitionNotFoundError, + SourceProjectionNotFoundError, +) +from powercontext.sources.adapters import SourceAdapter +from powercontext.sources.models import Source, SourceProjectionKey + +InputT = TypeVar("InputT") +SourceT = TypeVar("SourceT", bound=Source) +ValueT_co = TypeVar("ValueT_co", covariant=True) + +_JSON_VALUE = TypeAdapter(JsonValue) +_AnySourceAdapter = SourceAdapter[Any, Any, Any] + + +class SourceProjection(Protocol[SourceT]): + """Project one exact Source value through a named, versioned capability.""" + + name: str + version: str + source_class: type[SourceT] + output_class: type[BaseModel] + + def project(self, source: SourceT, /) -> object: ... + + +class SourceDefinition(SourceAdapter[InputT, SourceT, ValueT_co], Protocol[InputT, SourceT, ValueT_co]): + """Bind one adapter contract to a durable version and optional projections.""" + + version: str + projections: tuple[SourceProjection[SourceT], ...] + + +@dataclass(frozen=True, slots=True) +class AdapterSourceDefinition(Generic[InputT, SourceT, ValueT_co]): + """Promote an existing typed Source adapter into an explicit Definition.""" + + adapter: SourceAdapter[InputT, SourceT, ValueT_co] + version: str = "1" + projections: tuple[SourceProjection[SourceT], ...] = () + + @property + def input_class(self) -> type[InputT]: + return self.adapter.input_class + + @property + def name(self) -> str: + return self.adapter.name + + @property + def source_class(self) -> type[SourceT]: + return self.adapter.source_class + + async def resolve(self, value: InputT, /) -> SourceT: + return await self.adapter.resolve(value) + + async def read(self, source: SourceT, /) -> ValueT_co: + return await self.adapter.read(source) + + +_AnySourceDefinition = SourceDefinition[Any, Any, Any] + + +class SourceDefinitionRegistry: + """Provide one immutable routing view for Source persistence and consumers.""" + + def __init__(self, definitions: Iterable[_AnySourceDefinition], /) -> None: + by_input: dict[type[object], _AnySourceDefinition] = {} + by_source: dict[type[Source], _AnySourceDefinition] = {} + by_name: dict[str, _AnySourceDefinition] = {} + projections: dict[type[Source], Mapping[SourceProjectionKey, SourceProjection[Any]]] = {} + registered: list[_AnySourceDefinition] = [] + + for definition in definitions: + input_class, source_class = _validate_definition(definition) + if input_class in by_input: + raise SourceConflictError("input_class", input_class) + if source_class in by_source: + raise SourceConflictError("source_class", source_class) + if definition.name in by_name: + raise SourceConflictError("name", definition.name) + + projection_routes: dict[SourceProjectionKey, SourceProjection[Any]] = {} + for projection in definition.projections: + key = _validate_projection(definition, projection) + if key in projection_routes: + raise SourceConflictError("projection", (definition.name, key)) + projection_routes[key] = projection + + by_input[input_class] = definition + by_source[source_class] = definition + by_name[definition.name] = definition + projections[source_class] = projection_routes + registered.append(definition) + + self._definitions = tuple(registered) + self._by_input = by_input + self._by_source = by_source + self._by_name = by_name + self._projections = projections + + @classmethod + def from_adapters(cls, adapters: Iterable[_AnySourceAdapter], /) -> SourceDefinitionRegistry: + """Wrap legacy adapters as version ``1`` Definitions without projections.""" + + return cls(AdapterSourceDefinition(adapter) for adapter in adapters) + + @property + def definitions(self) -> tuple[_AnySourceDefinition, ...]: + return self._definitions + + def definition_for_name(self, name: str, /) -> _AnySourceDefinition: + try: + return self._by_name[name] + except KeyError: + raise SourceDefinitionNotFoundError(name) from None + + def definition_for_source(self, source: object, /) -> _AnySourceDefinition: + if not isinstance(source, Source): + raise InvalidSourceEntryError(type(source)) + try: + definition = self._by_source[type(source)] + except KeyError: + raise SourceAdapterNotFoundError("source", type(source)) from None + if source.definition_version != definition.version: + raise InvalidSourceDefinitionError( + type(definition), + "version", + f"Source declares {source.definition_version!r}, expected {definition.version!r}", + ) + return definition + + async def resolve(self, value: object, /) -> Source: + input_class = type(value) + try: + definition = self._by_input[input_class] + except KeyError: + raise SourceAdapterNotFoundError("input", input_class) from None + source = await definition.resolve(value) + if type(source) is not definition.source_class: + raise InvalidSourceResultError(definition.name, "resolve", definition.source_class, type(source)) + self.definition_for_source(source) + return cast(Source, source) + + async def read(self, source: Source, /) -> object: + definition = self.definition_for_source(source) + return await definition.read(source) + + def projection_keys(self, source: Source, /) -> tuple[SourceProjectionKey, ...]: + self.definition_for_source(source) + return tuple(self._projections[type(source)]) + + def project(self, source: Source, key: SourceProjectionKey, /) -> JsonValue: + definition = self.definition_for_source(source) + try: + projection = self._projections[type(source)][key] + except KeyError: + raise SourceProjectionNotFoundError(definition.name, key.name, key.version) from None + value = projection.project(source) + try: + validated = projection.output_class.model_validate(value) + return _JSON_VALUE.validate_python(validated.model_dump(mode="json")) + except (TypeError, ValueError) as error: + raise InvalidSourceProjectionError(key.name, "result", "must match the declared output schema") from error + + +def _validate_definition(definition: object) -> tuple[type[object], type[Source]]: + definition_type = type(definition) + input_class = getattr(definition, "input_class", None) + if not isinstance(input_class, type): + raise InvalidSourceAdapterError(definition_type, "input_class", "must be a type") + name = getattr(definition, "name", None) + if not isinstance(name, str) or not name.strip() or name != name.strip(): + raise InvalidSourceDefinitionError(definition_type, "name", "must be a non-empty trimmed string") + version = getattr(definition, "version", None) + if not isinstance(version, str) or not version.strip() or version != version.strip(): + raise InvalidSourceDefinitionError(definition_type, "version", "must be a non-empty trimmed string") + source_class = getattr(definition, "source_class", None) + if not isinstance(source_class, type) or not issubclass(source_class, Source): + raise InvalidSourceAdapterError(definition_type, "source_class", "must be a Source subclass") + projections = getattr(definition, "projections", None) + if not isinstance(projections, tuple): + raise InvalidSourceDefinitionError(definition_type, "projections", "must be a tuple") + for method_name in ("resolve", "read"): + if not callable(getattr(definition, method_name, None)): + raise InvalidSourceAdapterError(definition_type, method_name, "must be callable") + return cast(type[object], input_class), cast(type[Source], source_class) + + +def _validate_projection( + definition: _AnySourceDefinition, + projection: object, +) -> SourceProjectionKey: + projection_type = type(projection) + name = getattr(projection, "name", None) + version = getattr(projection, "version", None) + if not isinstance(name, str) or not isinstance(version, str): + raise InvalidSourceProjectionError(str(name), "key", "must contain string name and version") + try: + key = SourceProjectionKey(name=name, version=version) + except (TypeError, ValueError) as error: + raise InvalidSourceProjectionError(str(name), "key", "must contain valid name and version") from error + source_class = getattr(projection, "source_class", None) + if source_class is not definition.source_class: + raise InvalidSourceProjectionError( + key.name, + "source_class", + f"must be {definition.source_class.__module__}.{definition.source_class.__qualname__}", + ) + output_class = getattr(projection, "output_class", None) + if not isinstance(output_class, type) or not issubclass(output_class, BaseModel): + raise InvalidSourceProjectionError(key.name, "output_class", "must be a BaseModel subclass") + if not callable(getattr(projection, "project", None)): + raise InvalidSourceProjectionError(key.name, "project", f"must be callable on {projection_type.__name__}") + return key diff --git a/src/powercontext/sources/models.py b/src/powercontext/sources/models.py index 9987e0620..6e2cd48df 100644 --- a/src/powercontext/sources/models.py +++ b/src/powercontext/sources/models.py @@ -16,7 +16,7 @@ from enum import StrEnum -from pydantic import BaseModel, field_validator +from pydantic import BaseModel, ConfigDict, field_validator from powercontext.errors import InvalidSourceReferenceError from powercontext.limits import MAX_SOURCE_ID_LENGTH, MAX_SOURCE_TYPE_LENGTH @@ -42,13 +42,35 @@ def validate_reference_part(cls, value: str, info) -> str: return value +class SourceProjectionKey(BaseModel): + """Select one independently versioned named projection capability.""" + + model_config = ConfigDict(frozen=True) + + name: str + version: str + + @field_validator("name", "version") + @classmethod + def validate_key_part(cls, value: str, info) -> str: + _validate_reference_part(info.field_name, value) + return value + + class Source(BaseModel): """Base value for an adapter-owned Source description.""" name: str + definition_version: str = "1" materialization: SourceMaterialization description: str | None = None + @field_validator("definition_version") + @classmethod + def validate_definition_version(cls, value: str) -> str: + _validate_reference_part("definition_version", value) + return value + def _validate_reference_part(field: str, value: object) -> None: if not isinstance(value, str) or not value.strip(): diff --git a/tests/builtin/persistence/test_provider.py b/tests/builtin/persistence/test_provider.py index 46a146c79..01de310d6 100644 --- a/tests/builtin/persistence/test_provider.py +++ b/tests/builtin/persistence/test_provider.py @@ -17,12 +17,45 @@ import asyncio import pytest - -from powercontext import ArtifactNotFoundError, SourceConflictError +from pydantic import BaseModel + +from powercontext import ( + AdapterSourceDefinition, + ArtifactNotFoundError, + Source, + SourceConflictError, + SourceDefinitionRegistry, + SourceMaterialization, +) from powercontext.builtin.artifacts.memory import MemoryCandidateRequest, MemoryEntryInput from powercontext.builtin.persistence.sqlite import SQLiteConfig from powercontext.builtin.runtime import BuiltinConfig, open_builtin_contexts -from powercontext.builtin.sources import ContentCapture, ContentSource, SourceCursor +from powercontext.builtin.sources import BUILTIN_SOURCE_REGISTRY, ContentCapture, ContentSource, SourceCursor + + +class CustomCapture(BaseModel): + source_id: str + value: str + + +class CustomSource(Source): + value: str + + +class CustomSourceAdapter: + input_class = CustomCapture + name = "test-custom" + source_class = CustomSource + + async def resolve(self, value: CustomCapture, /) -> CustomSource: + return CustomSource( + name=value.source_id, + materialization=SourceMaterialization.CAPTURED, + value=value.value, + ) + + async def read(self, source: CustomSource, /) -> str: + return source.value class EchoCandidatePipeline: @@ -54,6 +87,27 @@ class StateSaveFailure(RuntimeError): pass +def test_provider_uses_one_injected_source_registry_for_routing_and_persistence() -> None: + async def scenario() -> None: + registry = SourceDefinitionRegistry(( + *BUILTIN_SOURCE_REGISTRY.definitions, + AdapterSourceDefinition(CustomSourceAdapter()), + )) + async with open_builtin_contexts( + BuiltinConfig(database=SQLiteConfig()), + source_registry=registry, + ) as contexts: + context = await contexts.get("project") + source = await context.sources.resolve(CustomCapture(source_id="custom-1", value="typed value")) + stored = await context.sources.add(source) + + assert isinstance(stored, CustomSource) + assert await context.sources.read(stored) == "typed value" + assert await context.sources.list() == (stored,) + + asyncio.run(scenario()) + + def test_provider_translates_repository_source_identity_conflicts() -> None: async def scenario() -> None: async with open_builtin_contexts(BuiltinConfig(database=SQLiteConfig())) as contexts: diff --git a/tests/test_connectors.py b/tests/test_connectors.py new file mode 100644 index 000000000..1812569a9 --- /dev/null +++ b/tests/test_connectors.py @@ -0,0 +1,238 @@ +# Copyright (c) 2026 OceanBase. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from __future__ import annotations + +import asyncio +import builtins +from copy import deepcopy + +import pytest +from pydantic import JsonValue + +from powercontext import ( + CatalogConnectorSourceSink, + ConnectorBinding, + ConnectorCapability, + ConnectorLifecycle, + ConnectorRunCompletion, + ConnectorRunSession, + ConnectorRunStatus, + ConnectorSubmissionResult, + ConnectorSubmissionStatus, + InvalidConnectorError, + InvalidConnectorRunError, + Source, + SourceCatalog, + SourceConflictError, + SourceRef, +) +from powercontext.builtin.sources import ( + BUILTIN_SOURCE_REGISTRY, + CONTENT_SOURCE_NAME, + TEXT_EVIDENCE_PROJECTION_KEY, + ContentCapture, +) + + +class IdempotentSourceStore: + def __init__(self, events: builtins.list[str]) -> None: + self.events = events + self.sources: dict[tuple[str, str], Source] = {} + + async def add(self, source: Source, /) -> Source: + definition = BUILTIN_SOURCE_REGISTRY.definition_for_source(source) + ref = SourceRef(source_type=definition.name, source_id=source.name) + key = (ref.source_type, ref.source_id) + existing = self.sources.get(key) + if existing is not None and existing != source: + raise SourceConflictError("identity", ref) + self.events.append(f"source:{ref.source_id}") + self.sources.setdefault(key, deepcopy(source)) + return self.sources[key] + + async def get(self, source: Source, /) -> Source: + definition = BUILTIN_SOURCE_REGISTRY.definition_for_source(source) + return self.sources[(definition.name, source.name)] + + async def list(self) -> tuple[Source, ...]: + return tuple(self.sources.values()) + + +class MemoryCheckpointStore: + def __init__(self, events: list[str]) -> None: + self.events = events + self.values: dict[str, JsonValue | None] = {} + + async def load(self, binding: ConnectorBinding, /) -> JsonValue | None: + return deepcopy(self.values.get(binding.binding_id)) + + async def save( + self, + binding: ConnectorBinding, + checkpoint: JsonValue | None, + /, + *, + expected: JsonValue | None, + ) -> None: + assert self.values.get(binding.binding_id) == expected + self.events.append("checkpoint") + self.values[binding.binding_id] = deepcopy(checkpoint) + + +class ContentConnector: + name = "test-content" + version = "1" + source_definitions = frozenset({CONTENT_SOURCE_NAME}) + capabilities = frozenset({ConnectorCapability.CHECKPOINT_RESUME}) + + def __init__(self, capture: ContentCapture) -> None: + self.capture = capture + + async def run(self, session: ConnectorRunSession, /) -> ConnectorRunCompletion: + await session.submit(self.capture.source_id, CONTENT_SOURCE_NAME, self.capture) + return ConnectorRunCompletion(status=ConnectorRunStatus.COMPLETE, checkpoint={"cursor": 1}) + + +def _binding() -> ConnectorBinding: + return ConnectorBinding( + scope_id="scope-a", + binding_id="content-a", + connector_name="test-content", + connector_version="1", + ) + + +def test_connector_commits_checkpoint_after_durable_source_acceptance() -> None: + async def scenario() -> None: + events: list[str] = [] + store = IdempotentSourceStore(events) + catalog = SourceCatalog(backend=store, registry=BUILTIN_SOURCE_REGISTRY) + checkpoints = MemoryCheckpointStore(events) + lifecycle = ConnectorLifecycle( + sink=CatalogConnectorSourceSink(scope_id="scope-a", catalog=catalog, store=store), + checkpoints=checkpoints, + ) + connector = ContentConnector(ContentCapture(source_id="note-1", content="Remember this.")) + + result = await lifecycle.run(connector, _binding()) + + assert events == ["source:note-1", "checkpoint"] + assert result.status is ConnectorRunStatus.COMPLETE + assert result.previous_checkpoint is None + assert result.committed_checkpoint == {"cursor": 1} + assert result.items[0].status is ConnectorSubmissionStatus.ACCEPTED + assert result.items[0].source_ref == SourceRef(source_type=CONTENT_SOURCE_NAME, source_id="note-1") + assert len(store.sources) == 1 + stored = next(iter(store.sources.values())) + assert catalog.project(stored, TEXT_EVIDENCE_PROJECTION_KEY) == { + "source_type": CONTENT_SOURCE_NAME, + "source_id": "note-1", + "content": "Remember this.", + "metadata": {}, + } + assert catalog.project(stored, TEXT_EVIDENCE_PROJECTION_KEY) == catalog.project( + stored, + TEXT_EVIDENCE_PROJECTION_KEY, + ) + + replay = await lifecycle.run(connector, _binding()) + assert replay.previous_checkpoint == {"cursor": 1} + assert len(store.sources) == 1 + assert events == ["source:note-1", "checkpoint", "source:note-1"] + + asyncio.run(scenario()) + + +def test_connector_exposes_failed_items_and_does_not_advance_checkpoint() -> None: + class RejectingSink: + async def submit(self, binding, item_id, definition_name, value, /) -> ConnectorSubmissionResult: + return ConnectorSubmissionResult(status=ConnectorSubmissionStatus.REJECTED, detail="unsupported value") + + async def scenario() -> None: + events: list[str] = [] + checkpoints = MemoryCheckpointStore(events) + lifecycle = ConnectorLifecycle(sink=RejectingSink(), checkpoints=checkpoints) + + result = await lifecycle.run( + ContentConnector(ContentCapture(source_id="note-1", content="Remember this.")), + _binding(), + ) + + assert result.status is ConnectorRunStatus.INCOMPLETE + assert result.proposed_checkpoint == {"cursor": 1} + assert result.committed_checkpoint is None + assert result.items[0].status is ConnectorSubmissionStatus.REJECTED + assert events == [] + + asyncio.run(scenario()) + + +def test_connector_rejects_duplicate_items_and_binding_mismatches() -> None: + class DuplicateConnector(ContentConnector): + async def run(self, session: ConnectorRunSession, /) -> ConnectorRunCompletion: + await session.submit("note-1", CONTENT_SOURCE_NAME, self.capture) + await session.submit("note-1", CONTENT_SOURCE_NAME, self.capture) + return ConnectorRunCompletion(status=ConnectorRunStatus.COMPLETE) + + async def scenario() -> None: + events: list[str] = [] + store = IdempotentSourceStore(events) + lifecycle = ConnectorLifecycle( + sink=CatalogConnectorSourceSink( + scope_id="scope-a", + catalog=SourceCatalog(backend=store, registry=BUILTIN_SOURCE_REGISTRY), + store=store, + ), + checkpoints=MemoryCheckpointStore(events), + ) + capture = ContentCapture(source_id="note-1", content="Remember this.") + + with pytest.raises(InvalidConnectorRunError) as duplicate: + await lifecycle.run(DuplicateConnector(capture), _binding()) + assert duplicate.value.issue == "duplicate-item" + + mismatched = _binding().model_copy(update={"connector_version": "2"}) + with pytest.raises(InvalidConnectorError) as binding_error: + await lifecycle.run(ContentConnector(capture), mismatched) + assert binding_error.value.field == "version" + + asyncio.run(scenario()) + + +def test_catalog_connector_sink_rejects_a_different_scope_before_storage() -> None: + async def scenario() -> None: + events: list[str] = [] + store = IdempotentSourceStore(events) + lifecycle = ConnectorLifecycle( + sink=CatalogConnectorSourceSink( + scope_id="scope-b", + catalog=SourceCatalog(backend=store, registry=BUILTIN_SOURCE_REGISTRY), + store=store, + ), + checkpoints=MemoryCheckpointStore(events), + ) + + result = await lifecycle.run( + ContentConnector(ContentCapture(source_id="note-1", content="Remember this.")), + _binding(), + ) + + assert result.status is ConnectorRunStatus.INCOMPLETE + assert result.items[0].status is ConnectorSubmissionStatus.FAILED + assert result.items[0].detail == "InvalidConnectorRunError" + assert store.sources == {} + assert events == [] + + asyncio.run(scenario()) diff --git a/tests/test_sources.py b/tests/test_sources.py index b757b7b88..b9707f751 100644 --- a/tests/test_sources.py +++ b/tests/test_sources.py @@ -20,12 +20,17 @@ from typing import TypeVar import pytest +from pydantic import BaseModel from powercontext import ( + AdapterSourceDefinition, + InvalidSourceDefinitionError, InvalidSourceEntryError, + InvalidSourceProjectionError, InvalidSourceResultError, SourceAdapterNotFoundError, SourceNotFoundError, + SourceProjectionNotFoundError, ) from powercontext.context import Sources from powercontext.sources import ( @@ -33,7 +38,9 @@ SourceAdapter, SourceCatalog, SourceCatalogBackend, + SourceDefinitionRegistry, SourceMaterialization, + SourceProjectionKey, SourceStore, ) @@ -105,6 +112,24 @@ async def read(self, source: TranscriptExportSource) -> object: return source +class ConversationSummary(BaseModel): + session_id: str + message_count: int + + +class ConversationSummaryProjection: + name = "test.conversation-summary" + version = "1" + source_class = ConversationSource + output_class: type[BaseModel] = ConversationSummary + + def project(self, source: ConversationSource, /) -> ConversationSummary: + return ConversationSummary( + session_id=source.session_id, + message_count=0 if source.captured_value is None else len(source.captured_value.messages), + ) + + StoredSourceT = TypeVar("StoredSourceT", bound=Source) @@ -282,3 +307,54 @@ async def scenario() -> None: assert backend.sources == [] asyncio.run(scenario()) + + +def test_definition_registry_routes_named_projections_without_source_type_checks() -> None: + async def scenario() -> None: + adapter = ConversationAdapter({"session-42": Conversation(("one", "two"))}) + registry = SourceDefinitionRegistry(( + AdapterSourceDefinition( + adapter, + version="1", + projections=(ConversationSummaryProjection(),), + ), + )) + catalog = SourceCatalog(backend=InMemorySourceStore(), registry=registry) + source = await catalog.resolve(ConversationCapture("snapshot", "session-42", capture=True)) + key = SourceProjectionKey(name="test.conversation-summary", version="1") + + assert catalog.projection_keys(source) == (key,) + assert catalog.project(source, key) == {"session_id": "session-42", "message_count": 2} + with pytest.raises(SourceProjectionNotFoundError): + catalog.project(source, SourceProjectionKey(name=key.name, version="2")) + + asyncio.run(scenario()) + + +def test_definition_registry_rejects_version_and_projection_contract_violations() -> None: + class InvalidProjection: + name = "test.invalid-json" + version = "1" + source_class = ConversationSource + output_class: type[BaseModel] = ConversationSummary + + def project(self, source: ConversationSource, /) -> object: + return object() + + adapter = ConversationAdapter({"session-42": Conversation(("one",))}) + registry = SourceDefinitionRegistry(( + AdapterSourceDefinition(adapter, version="2", projections=(InvalidProjection(),)), + )) + source = ConversationSource( + name="snapshot", + materialization=SourceMaterialization.CAPTURED, + session_id="session-42", + captured_value=Conversation(("one",)), + ) + + with pytest.raises(InvalidSourceDefinitionError): + registry.definition_for_source(source) + + compatible = source.model_copy(update={"definition_version": "2"}) + with pytest.raises(InvalidSourceProjectionError): + registry.project(compatible, SourceProjectionKey(name="test.invalid-json", version="1")) From b6dc87ba47ab8939f139b26d8407eaf8fa599a26 Mon Sep 17 00:00:00 2001 From: Chojan Shang Date: Thu, 27 Aug 2026 17:41:22 +0800 Subject: [PATCH 4/9] feat(connectors): add OpenDAL text file ingestion --- .../how-to/ingest-text-files-with-opendal.md | 90 ++++++ .../how-to/ingest-text-files-with-opendal.md | 87 ++++++ pyproject.toml | 3 + .../builtin/connectors/__init__.py | 27 ++ .../builtin/connectors/opendal.py | 295 ++++++++++++++++++ .../builtin/persistence/__init__.py | 8 + .../builtin/persistence/connectors.py | 180 +++++++++++ .../builtin/persistence/tables.py | 11 + .../builtin/runtime/relational.py | 34 ++ src/powercontext/builtin/sources/__init__.py | 22 +- src/powercontext/builtin/sources/content.py | 19 +- .../builtin/sources/projections.py | 33 ++ src/powercontext/builtin/sources/text_file.py | 174 +++++++++++ src/powercontext/sources/connectors.py | 65 +++- tests/builtin/connectors/test_opendal.py | 234 ++++++++++++++ tests/test_connectors.py | 34 ++ uv.lock | 56 +++- zensical.toml | 2 + 18 files changed, 1345 insertions(+), 29 deletions(-) create mode 100644 docs/en/docs/how-to/ingest-text-files-with-opendal.md create mode 100644 docs/zh/docs/how-to/ingest-text-files-with-opendal.md create mode 100644 src/powercontext/builtin/connectors/__init__.py create mode 100644 src/powercontext/builtin/connectors/opendal.py create mode 100644 src/powercontext/builtin/persistence/connectors.py create mode 100644 src/powercontext/builtin/sources/projections.py create mode 100644 src/powercontext/builtin/sources/text_file.py create mode 100644 tests/builtin/connectors/test_opendal.py diff --git a/docs/en/docs/how-to/ingest-text-files-with-opendal.md b/docs/en/docs/how-to/ingest-text-files-with-opendal.md new file mode 100644 index 000000000..1ff5069ff --- /dev/null +++ b/docs/en/docs/how-to/ingest-text-files-with-opendal.md @@ -0,0 +1,90 @@ +--- +title: Ingest text files with OpenDAL +description: Capture UTF-8 files as typed Sources through an OpenDAL storage backend. +--- + +# Ingest text files with OpenDAL + +Use `OpenDALTextFileConnector` to capture bounded UTF-8 files from a storage backend supported by OpenDAL. Each accepted +file becomes an immutable `text-file-snapshot` Source with its path, namespace, content digest, and available provider +annotations. + +## Before you begin + +The OpenDAL integration requires Python 3.12 or later. Install the optional dependency: + +```bash +uv add "powercontext[opendal]" +``` + +Choose a stable `source_namespace` for the storage location. It distinguishes identical paths and bytes captured from +different authorities. Do not put credentials in the namespace. + +## Run a local filesystem binding + +The following binding scans the `docs` directory below `/absolute/path/to/project` and persists its checkpoint in the +same PowerContext database as the captured Sources: + +```python +import asyncio + +from powercontext.builtin.connectors import OpenDALTextFileConnector +from powercontext.builtin.persistence.sqlite import SQLiteConfig +from powercontext.builtin.runtime import BuiltinConfig, open_builtin_contexts +from powercontext.sources import ConnectorBinding + + +async def main() -> None: + connector = OpenDALTextFileConnector.from_service( + "fs", + source_namespace="project-docs", + root="docs", + storage_options={"root": "/absolute/path/to/project"}, + ) + binding = ConnectorBinding( + scope_id="project:example", + binding_id="project-docs", + connector_name=connector.name, + connector_version=connector.version, + ) + config = BuiltinConfig( + database=SQLiteConfig(url="sqlite+aiosqlite:///powercontext.db"), + ) + + async with open_builtin_contexts(config) as contexts: + result = await contexts.run_connector(connector, binding) + print(result.model_dump_json(indent=2)) + + +asyncio.run(main()) +``` + +Use a different OpenDAL service and its backend options for remote storage. `storage_options` are runtime-only and are +not copied into Source payloads or checkpoints. + +## Interpret the result + +An item outcome reports one of four states: + +- `accepted`: the Source was durably stored; +- `replayed`: the sink recognized an already accepted Source; +- `rejected`: the provider item could not satisfy the Source Definition, such as invalid UTF-8; +- `failed`: the item could not be read or stored safely. + +The checkpoint advances only after every selected item is accepted and the run completes. A rejected or failed item +leaves the previous checkpoint in place, so the next run safely retries the scan. Files whose digest matches the +committed checkpoint are skipped. + +Accepted Sources enter the same scoped Source journal used by Memory extraction. When the runtime has a Memory +candidate pipeline, its normal source-window flush or schedule can consume these Sources through the shared +`powercontext.builtin.text-evidence` projection. Connector completion does not itself create Memory. + +## Current limits + +- The default patterns select Markdown, text, reStructuredText, and AsciiDoc files. +- A run selects at most 10,000 files and reads at most 2 MiB per file unless configured otherwise. +- Only UTF-8 content is accepted. +- Changed bytes produce a new exact snapshot Source; earlier snapshots remain readable for lineage. +- A full scan removes missing paths from the next checkpoint but does not delete Sources or claim authoritative + deletion. +- The Connector does not provide a change feed. Schedule repeated runs to observe later changes. diff --git a/docs/zh/docs/how-to/ingest-text-files-with-opendal.md b/docs/zh/docs/how-to/ingest-text-files-with-opendal.md new file mode 100644 index 000000000..d5440ae1f --- /dev/null +++ b/docs/zh/docs/how-to/ingest-text-files-with-opendal.md @@ -0,0 +1,87 @@ +--- +title: 使用 OpenDAL 采集文本文件 +description: 通过 OpenDAL 存储后端把 UTF-8 文件捕获为类型化 Source。 +--- + +# 使用 OpenDAL 采集文本文件 + +使用 `OpenDALTextFileConnector` 从 OpenDAL 支持的存储后端捕获有界 UTF-8 文件。每个接受的文件都会成为不可变的 +`text-file-snapshot` Source，保留 path、namespace、content digest 和后端能够提供的 annotation。 + +## 前置条件 + +OpenDAL 集成要求 Python 3.12 或更高版本。安装可选依赖： + +```bash +uv add "powercontext[opendal]" +``` + +为存储位置选择稳定的 `source_namespace`。它用于区分来自不同 authority、但 path 和内容相同的文件。不要把凭据写进 +namespace。 + +## 运行本地文件系统 binding + +下面的 binding 扫描 `/absolute/path/to/project` 下的 `docs` 目录，并把 checkpoint 与捕获的 Source 持久化到同一个 +PowerContext 数据库： + +```python +import asyncio + +from powercontext.builtin.connectors import OpenDALTextFileConnector +from powercontext.builtin.persistence.sqlite import SQLiteConfig +from powercontext.builtin.runtime import BuiltinConfig, open_builtin_contexts +from powercontext.sources import ConnectorBinding + + +async def main() -> None: + connector = OpenDALTextFileConnector.from_service( + "fs", + source_namespace="project-docs", + root="docs", + storage_options={"root": "/absolute/path/to/project"}, + ) + binding = ConnectorBinding( + scope_id="project:example", + binding_id="project-docs", + connector_name=connector.name, + connector_version=connector.version, + ) + config = BuiltinConfig( + database=SQLiteConfig(url="sqlite+aiosqlite:///powercontext.db"), + ) + + async with open_builtin_contexts(config) as contexts: + result = await contexts.run_connector(connector, binding) + print(result.model_dump_json(indent=2)) + + +asyncio.run(main()) +``` + +访问远端存储时，换用对应的 OpenDAL service 及其 backend option。`storage_options` 只在运行期使用，不会复制进 Source +payload 或 checkpoint。 + +## 理解运行结果 + +每个 item outcome 有四种状态： + +- `accepted`：Source 已持久化； +- `replayed`：sink 识别到已接受的 Source； +- `rejected`：provider item 无法满足 Source Definition，例如不是有效 UTF-8； +- `failed`：无法安全读取或存储该 item。 + +只有所有选中 item 都被接受且本轮完整结束后，checkpoint 才会前移。出现 rejected 或 failed item 时保留旧 checkpoint， +下一轮会安全重试扫描。digest 与已提交 checkpoint 相同的文件会被跳过。 + +接受的 Source 会进入同一 scope 的 Source journal。Runtime 配置了 Memory candidate pipeline 后，常规 source-window flush +或调度任务可以通过共享的 `powercontext.builtin.text-evidence` projection 消费这些 Source。Connector 完成采集并不直接创建 +Memory。 + +## 当前限制 + +- 默认 pattern 选择 Markdown、纯文本、reStructuredText 和 AsciiDoc 文件。 +- 除非显式调整，每轮最多选择 10,000 个文件，每个文件最多读取 2 MiB。 +- 只接受 UTF-8 内容。 +- 文件内容变化会生成新的精确 snapshot Source；旧 snapshot 仍然可读，以保留 lineage。 +- 全量扫描会从下一 checkpoint 移除已消失的 path，但不会删除 Source，也不声明 authoritative deletion。 +- Connector 不提供 change feed。需要通过周期运行观察后续变化。 diff --git a/pyproject.toml b/pyproject.toml index 89cd74e0e..64c45a1ca 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -55,6 +55,9 @@ client = [ "opentelemetry-api>=1.30,<2", "pydantic-settings>=2.7,<3", ] +opendal = [ + "opendalfs>=0.1,<0.2; python_version >= '3.12'", +] server = [ "fastapi>=0.115,<1", "fastmcp>=3.4,<4", diff --git a/src/powercontext/builtin/connectors/__init__.py b/src/powercontext/builtin/connectors/__init__.py new file mode 100644 index 000000000..7bd21584d --- /dev/null +++ b/src/powercontext/builtin/connectors/__init__.py @@ -0,0 +1,27 @@ +# Copyright (c) 2026 OceanBase. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Supported built-in Connector implementations.""" + +from powercontext.builtin.connectors.opendal import ( + OPENDAL_TEXT_FILE_CONNECTOR_NAME, + OpenDALTextFileCheckpoint, + OpenDALTextFileConnector, +) + +__all__ = [ + "OPENDAL_TEXT_FILE_CONNECTOR_NAME", + "OpenDALTextFileCheckpoint", + "OpenDALTextFileConnector", +] diff --git a/src/powercontext/builtin/connectors/opendal.py b/src/powercontext/builtin/connectors/opendal.py new file mode 100644 index 000000000..7d0858dbd --- /dev/null +++ b/src/powercontext/builtin/connectors/opendal.py @@ -0,0 +1,295 @@ +# Copyright (c) 2026 OceanBase. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Capture UTF-8 text files through the OpenDAL fsspec implementation.""" + +from __future__ import annotations + +import asyncio +import fnmatch +import hashlib +import mimetypes +import posixpath +from collections.abc import Mapping, Sequence +from datetime import UTC, datetime +from typing import Any, Literal, Protocol + +from pydantic import BaseModel, ConfigDict, Field, JsonValue, ValidationError, field_validator + +from powercontext.builtin.sources import ( + TEXT_FILE_SNAPSHOT_SOURCE_NAME, + TextFileSnapshotCapture, +) +from powercontext.errors import InvalidConnectorRunError +from powercontext.sources import ( + ConnectorCapability, + ConnectorRunCompletion, + ConnectorRunSession, + ConnectorRunStatus, +) + +OPENDAL_TEXT_FILE_CONNECTOR_NAME = "opendal-text-files" +_DEFAULT_PATTERNS = ("**/*.md", "**/*.markdown", "**/*.txt", "**/*.rst", "**/*.adoc") + + +class _FsspecFileSystem(Protocol): + def find(self, path: str, *, detail: bool) -> Mapping[str, Mapping[str, object]]: ... + + def cat_file(self, path: str) -> bytes: ... + + +class OpenDALTextFileCheckpoint(BaseModel): + """Opaque content-digest checkpoint for one Connector binding.""" + + model_config = ConfigDict(frozen=True) + + schema_version: Literal["1"] = "1" + files: dict[str, str] = Field(default_factory=dict) + + @field_validator("files") + @classmethod + def validate_files(cls, value: dict[str, str]) -> dict[str, str]: + for path, digest in value.items(): + _validate_relative_path(path) + if not digest.startswith("sha256:") or len(digest) != 71: + raise ValueError("checkpoint digests must use sha256:") # noqa: TRY003 + try: + int(digest.removeprefix("sha256:"), 16) + except ValueError as error: + raise ValueError("checkpoint digest must contain lowercase hexadecimal") from error # noqa: TRY003 + if digest != digest.lower(): + raise ValueError("checkpoint digest must contain lowercase hexadecimal") # noqa: TRY003 + return value + + +class OpenDALTextFileConnector: + """Perform bounded full scans through an OpenDAL-backed fsspec filesystem.""" + + name = OPENDAL_TEXT_FILE_CONNECTOR_NAME + version = "1" + source_definitions = frozenset({TEXT_FILE_SNAPSHOT_SOURCE_NAME}) + capabilities = frozenset({ConnectorCapability.CHECKPOINT_RESUME}) + + def __init__( + self, + filesystem: _FsspecFileSystem, + *, + source_namespace: str, + root: str = "", + patterns: Sequence[str] = _DEFAULT_PATTERNS, + max_files: int = 10_000, + max_file_size: int = 2 * 1024 * 1024, + ) -> None: + if not source_namespace or source_namespace.strip() != source_namespace: + raise ValueError("source_namespace must be a non-empty trimmed string") # noqa: TRY003 + if max_files < 1: + raise ValueError("max_files must be positive") # noqa: TRY003 + if max_file_size < 1: + raise ValueError("max_file_size must be positive") # noqa: TRY003 + if isinstance(patterns, str): + raise TypeError("patterns must be a sequence of glob patterns") # noqa: TRY003 + normalized_patterns = tuple(patterns) + if not normalized_patterns or any(not pattern or pattern.strip() != pattern for pattern in normalized_patterns): + raise ValueError("patterns must contain non-empty trimmed values") # noqa: TRY003 + self._filesystem = filesystem + self._source_namespace = source_namespace + self._root = _normalize_root(root) + self._patterns = normalized_patterns + self._max_files = max_files + self._max_file_size = max_file_size + + @classmethod + def from_service( + cls, + service: str, + *, + source_namespace: str, + root: str = "", + storage_options: Mapping[str, object] | None = None, + patterns: Sequence[str] = _DEFAULT_PATTERNS, + max_files: int = 10_000, + max_file_size: int = 2 * 1024 * 1024, + ) -> OpenDALTextFileConnector: + """Create a Connector from one OpenDAL service and its runtime-only options.""" + + try: + from opendalfs import OpendalFileSystem + except ImportError as error: + raise ImportError( # noqa: TRY003 + "OpenDALTextFileConnector.from_service requires powercontext[opendal] on Python 3.12+" + ) from error + backend_options: dict[str, Any] = dict(storage_options or {}) + filesystem = OpendalFileSystem( + scheme=service, + asynchronous=False, + skip_instance_cache=True, + **backend_options, + ) + return cls( + filesystem, + source_namespace=source_namespace, + root=root, + patterns=patterns, + max_files=max_files, + max_file_size=max_file_size, + ) + + async def run(self, session: ConnectorRunSession, /) -> ConnectorRunCompletion: + previous = _checkpoint(session.checkpoint) + entries = await asyncio.to_thread(self._filesystem.find, self._root, detail=True) + files = self._selected_files(entries) + if len(files) > self._max_files: + raise InvalidConnectorRunError( + "file-limit", + f"scan selected {len(files)} files, maximum is {self._max_files}", + ) + + current_files: dict[str, str] = {} + for relative_path, storage_path, info in files: + size = _non_negative_int(info.get("size")) + if size is not None and size > self._max_file_size: + session.reject( + relative_path, + TEXT_FILE_SNAPSHOT_SOURCE_NAME, + f"file exceeds {self._max_file_size} bytes", + ) + continue + try: + content_bytes = await asyncio.to_thread(self._filesystem.cat_file, storage_path) + except Exception as error: + session.fail(relative_path, TEXT_FILE_SNAPSHOT_SOURCE_NAME, type(error).__name__) + continue + if not isinstance(content_bytes, bytes): + session.fail(relative_path, TEXT_FILE_SNAPSHOT_SOURCE_NAME, "filesystem returned non-bytes content") + continue + if len(content_bytes) > self._max_file_size: + session.reject( + relative_path, + TEXT_FILE_SNAPSHOT_SOURCE_NAME, + f"file exceeds {self._max_file_size} bytes", + ) + continue + + content_digest = f"sha256:{hashlib.sha256(content_bytes).hexdigest()}" + current_files[relative_path] = content_digest + if previous.files.get(relative_path) == content_digest: + continue + try: + content = content_bytes.decode("utf-8") + except UnicodeDecodeError: + session.reject(relative_path, TEXT_FILE_SNAPSHOT_SOURCE_NAME, "file is not valid UTF-8") + continue + capture = TextFileSnapshotCapture( + namespace=self._source_namespace, + path=relative_path, + content=content, + media_type=mimetypes.guess_type(relative_path)[0] or "text/plain", + etag=_optional_string(info.get("etag")), + provider_version=_optional_string(info.get("version")), + modified_at=_optional_datetime(info.get("mtime")), + ) + await session.submit(relative_path, TEXT_FILE_SNAPSHOT_SOURCE_NAME, capture) + + checkpoint = OpenDALTextFileCheckpoint(files=current_files) + return ConnectorRunCompletion( + status=ConnectorRunStatus.COMPLETE, + checkpoint=checkpoint.model_dump(mode="json"), + ) + + def _selected_files( + self, + entries: Mapping[str, Mapping[str, object]], + ) -> tuple[tuple[str, str, Mapping[str, object]], ...]: + selected: list[tuple[str, str, Mapping[str, object]]] = [] + for storage_path, info in entries.items(): + if info.get("type") != "file": + continue + relative_path = _relative_path(storage_path, self._root) + if not _matches(relative_path, self._patterns): + continue + selected.append((relative_path, storage_path, info)) + selected.sort(key=lambda item: item[0]) + return tuple(selected) + + +def _checkpoint(value: JsonValue | None) -> OpenDALTextFileCheckpoint: + if value is None: + return OpenDALTextFileCheckpoint() + try: + return OpenDALTextFileCheckpoint.model_validate(value) + except ValidationError as error: + raise InvalidConnectorRunError("checkpoint", "does not match OpenDALTextFileCheckpoint") from error + + +def _normalize_root(value: str) -> str: + if value != value.strip() or "\\" in value: + raise ValueError("root must be a normalized POSIX path") # noqa: TRY003 + normalized = posixpath.normpath(value).strip("/") + if normalized in {"", "."}: + return "" + _validate_relative_path(normalized) + return normalized + + +def _relative_path(storage_path: str, root: str) -> str: + normalized = posixpath.normpath(storage_path).strip("/") + relative = posixpath.relpath(normalized, root) if root else normalized + _validate_relative_path(relative) + return relative + + +def _validate_relative_path(value: str) -> None: + if not value or value.startswith("/") or "\\" in value: + raise ValueError("file path must be a relative POSIX path") # noqa: TRY003 + normalized = posixpath.normpath(value) + if normalized != value or normalized == ".." or normalized.startswith("../"): + raise ValueError("file path escapes the configured root") # noqa: TRY003 + + +def _matches(path: str, patterns: tuple[str, ...]) -> bool: + return any( + fnmatch.fnmatchcase(path, pattern) + or (pattern.startswith("**/") and fnmatch.fnmatchcase(path, pattern.removeprefix("**/"))) + for pattern in patterns + ) + + +def _non_negative_int(value: object) -> int | None: + if not isinstance(value, int) or isinstance(value, bool) or value < 0: + return None + return value + + +def _optional_string(value: object) -> str | None: + return value if isinstance(value, str) and value and value.strip() == value else None + + +def _optional_datetime(value: object) -> datetime | None: + if isinstance(value, datetime): + return value + if isinstance(value, int | float) and not isinstance(value, bool): + return datetime.fromtimestamp(value, tz=UTC) + if not isinstance(value, str): + return None + try: + return datetime.fromisoformat(value.replace("Z", "+00:00")) + except ValueError: + return None + + +__all__ = [ + "OPENDAL_TEXT_FILE_CONNECTOR_NAME", + "OpenDALTextFileCheckpoint", + "OpenDALTextFileConnector", +] diff --git a/src/powercontext/builtin/persistence/__init__.py b/src/powercontext/builtin/persistence/__init__.py index 9f75b5f30..9267564f4 100644 --- a/src/powercontext/builtin/persistence/__init__.py +++ b/src/powercontext/builtin/persistence/__init__.py @@ -15,6 +15,11 @@ """SQLAlchemy-backed relational persistence building blocks.""" from powercontext.builtin.persistence.candidates import CandidateRepository +from powercontext.builtin.persistence.connectors import ( + ConnectorCheckpointRepository, + RelationalConnectorCheckpointStore, + StoredConnectorCheckpoint, +) from powercontext.builtin.persistence.database import AsyncDatabase from powercontext.builtin.persistence.errors import ( DatabaseClosedError, @@ -39,6 +44,7 @@ __all__ = ( "AsyncDatabase", "CandidateRepository", + "ConnectorCheckpointRepository", "DatabaseClosedError", "ExternalSkillRepository", "GenerationConflictError", @@ -47,9 +53,11 @@ "InvalidStoredColumnError", "InvalidStoredPayloadError", "PersistenceError", + "RelationalConnectorCheckpointStore", "RepositoryError", "RepositoryNotFoundError", "StatisticsRepository", + "StoredConnectorCheckpoint", "StoredInventoryCounts", "StoredModelUsage", "StoredPayloadConflictError", diff --git a/src/powercontext/builtin/persistence/connectors.py b/src/powercontext/builtin/persistence/connectors.py new file mode 100644 index 000000000..edb901617 --- /dev/null +++ b/src/powercontext/builtin/persistence/connectors.py @@ -0,0 +1,180 @@ +# Copyright (c) 2026 OceanBase. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Durable Connector checkpoints and their runtime store adapter.""" + +from __future__ import annotations + +from collections.abc import Mapping +from typing import Any + +from pydantic import BaseModel, ConfigDict, JsonValue +from sqlalchemy import insert, select, update +from sqlalchemy.exc import IntegrityError +from sqlalchemy.ext.asyncio import AsyncConnection + +from powercontext.builtin.persistence.codec import dump_model, load_model, stored_bytes +from powercontext.builtin.persistence.database import AsyncDatabase +from powercontext.builtin.persistence.tables import CONNECTOR_CHECKPOINTS_TABLE +from powercontext.errors import InvalidConnectorRunError +from powercontext.sources import ConnectorBinding + + +class _CheckpointPayload(BaseModel): + model_config = ConfigDict(frozen=True) + + value: JsonValue | None + + +class StoredConnectorCheckpoint(BaseModel): + """One decoded checkpoint bound to an exact Connector identity.""" + + model_config = ConfigDict(frozen=True) + + binding: ConnectorBinding + checkpoint: JsonValue | None + + +class ConnectorCheckpointRepository: + """Persist opaque Connector checkpoints with value-based comparison.""" + + async def load( + self, + connection: AsyncConnection, + binding: ConnectorBinding, + /, + *, + for_update: bool = False, + ) -> StoredConnectorCheckpoint | None: + statement = select(CONNECTOR_CHECKPOINTS_TABLE).where( + CONNECTOR_CHECKPOINTS_TABLE.c.scope_id == binding.scope_id, + CONNECTOR_CHECKPOINTS_TABLE.c.binding_id == binding.binding_id, + ) + if for_update: + statement = statement.with_for_update() + row = (await connection.execute(statement)).mappings().one_or_none() + if row is None: + return None + stored = _decode_row(row) + if stored.binding != binding: + raise InvalidConnectorRunError( + "binding-conflict", + f"checkpoint {binding.binding_id!r} belongs to a different Connector identity", + ) + return stored + + async def save( + self, + connection: AsyncConnection, + binding: ConnectorBinding, + checkpoint: JsonValue | None, + /, + *, + expected: JsonValue | None, + ) -> StoredConnectorCheckpoint: + existing = await self.load(connection, binding, for_update=True) + actual = None if existing is None else existing.checkpoint + if actual != expected: + raise _checkpoint_conflict(binding) + + payload = _dump_checkpoint(binding, checkpoint) + if existing is None: + try: + async with connection.begin_nested(): + await connection.execute( + insert(CONNECTOR_CHECKPOINTS_TABLE).values( + scope_id=binding.scope_id, + binding_id=binding.binding_id, + connector_name=binding.connector_name, + connector_version=binding.connector_version, + checkpoint=payload, + ) + ) + except IntegrityError: + raise _checkpoint_conflict(binding) from None + else: + result = await connection.execute( + update(CONNECTOR_CHECKPOINTS_TABLE) + .where( + CONNECTOR_CHECKPOINTS_TABLE.c.scope_id == binding.scope_id, + CONNECTOR_CHECKPOINTS_TABLE.c.binding_id == binding.binding_id, + CONNECTOR_CHECKPOINTS_TABLE.c.checkpoint == _dump_checkpoint(binding, expected), + ) + .values(checkpoint=payload) + ) + if result.rowcount != 1: + raise _checkpoint_conflict(binding) + return StoredConnectorCheckpoint(binding=binding, checkpoint=checkpoint) + + +class RelationalConnectorCheckpointStore: + """Adapt the Connector checkpoint protocol to an ``AsyncDatabase``.""" + + def __init__(self, database: AsyncDatabase, repository: ConnectorCheckpointRepository, /) -> None: + self._database = database + self._repository = repository + + async def load(self, binding: ConnectorBinding, /) -> JsonValue | None: + async with self._database.transaction() as connection: + stored = await self._repository.load(connection, binding) + return None if stored is None else stored.checkpoint + + async def save( + self, + binding: ConnectorBinding, + checkpoint: JsonValue | None, + /, + *, + expected: JsonValue | None, + ) -> None: + async with self._database.transaction() as connection: + await self._repository.save(connection, binding, checkpoint, expected=expected) + + +def _dump_checkpoint(binding: ConnectorBinding, checkpoint: JsonValue | None) -> bytes: + return dump_model( + _CheckpointPayload(value=checkpoint), + kind="connector-checkpoint", + name=binding.binding_id, + ) + + +def _checkpoint_conflict(binding: ConnectorBinding) -> InvalidConnectorRunError: + return InvalidConnectorRunError( + "checkpoint-conflict", + f"binding {binding.binding_id!r} changed during the run", + ) + + +def _decode_row(row: Mapping[Any, Any]) -> StoredConnectorCheckpoint: + binding = ConnectorBinding( + scope_id=str(row["scope_id"]), + binding_id=str(row["binding_id"]), + connector_name=str(row["connector_name"]), + connector_version=str(row["connector_version"]), + ) + payload = load_model( + _CheckpointPayload, + stored_bytes(row["checkpoint"], column="checkpoint"), + kind="connector-checkpoint", + name=binding.binding_id, + ) + return StoredConnectorCheckpoint(binding=binding, checkpoint=payload.value) + + +__all__ = [ + "ConnectorCheckpointRepository", + "RelationalConnectorCheckpointStore", + "StoredConnectorCheckpoint", +] diff --git a/src/powercontext/builtin/persistence/tables.py b/src/powercontext/builtin/persistence/tables.py index 99e1013a6..cfbcf2d58 100644 --- a/src/powercontext/builtin/persistence/tables.py +++ b/src/powercontext/builtin/persistence/tables.py @@ -274,6 +274,16 @@ def _entry_text_type(): CheckConstraint("generation >= 0", name="ck_pc_source_cursors_generation_nonnegative"), ) +CONNECTOR_CHECKPOINTS_TABLE = Table( + "pc_connector_checkpoints", + SHARED_METADATA, + Column("scope_id", identity_string(MAX_SCOPE_ID_LENGTH), primary_key=True), + Column("binding_id", identity_string(MAX_SOURCE_ID_LENGTH), primary_key=True), + Column("connector_name", identity_string(MAX_SOURCE_TYPE_LENGTH), nullable=False), + Column("connector_version", identity_string(MAX_SOURCE_TYPE_LENGTH), nullable=False), + Column("checkpoint", _canonical_payload_type(), nullable=False), +) + EXTERNAL_SKILL_REGISTRATIONS_TABLE = Table( "pc_external_skill_registrations", SHARED_METADATA, @@ -357,6 +367,7 @@ def _entry_text_type(): ARTIFACT_CANDIDATE_VERSIONS_TABLE, ARTIFACT_CANDIDATE_HEADS_TABLE, SOURCE_CURSORS_TABLE, + CONNECTOR_CHECKPOINTS_TABLE, EXTERNAL_SKILL_REGISTRATIONS_TABLE, ) diff --git a/src/powercontext/builtin/runtime/relational.py b/src/powercontext/builtin/runtime/relational.py index 120dda956..1ed68cf56 100644 --- a/src/powercontext/builtin/runtime/relational.py +++ b/src/powercontext/builtin/runtime/relational.py @@ -63,6 +63,10 @@ from powercontext.builtin.inference import EmbeddingModel, InvalidInferenceOutputError, TokenEstimator from powercontext.builtin.persistence.artifacts import ArtifactRepository from powercontext.builtin.persistence.candidates import CandidateRepository +from powercontext.builtin.persistence.connectors import ( + ConnectorCheckpointRepository, + RelationalConnectorCheckpointStore, +) from powercontext.builtin.persistence.cursors import SourceCursorRepository from powercontext.builtin.persistence.database import AsyncDatabase from powercontext.builtin.persistence.errors import RepositoryNotFoundError, StoredPayloadConflictError @@ -111,6 +115,11 @@ from powercontext.context import PowerContext from powercontext.errors import ArtifactNotFoundError, SourceConflictError, SourceNotFoundError from powercontext.sources import ( + CatalogConnectorSourceSink, + Connector, + ConnectorBinding, + ConnectorLifecycle, + ConnectorRunResult, Source, SourceCatalog, SourceDefinitionRegistry, @@ -127,6 +136,7 @@ class _Repositories: sources: SourceRepository artifacts: ArtifactRepository candidates: CandidateRepository + connector_checkpoints: ConnectorCheckpointRepository cursors: SourceCursorRepository external_skills: ExternalSkillRepository statistics: StatisticsRepository @@ -313,6 +323,7 @@ def __init__( Experience.family: ExperienceContent, Skill.family: SkillContent, }), + connector_checkpoints=ConnectorCheckpointRepository(), cursors=SourceCursorRepository(), external_skills=ExternalSkillRepository(), statistics=StatisticsRepository(), @@ -368,6 +379,29 @@ def statistics(self, scope_id: str, /) -> RelationalScopedStatistics: return self._services_for(scope_id).statistics() + async def run_connector( + self, + connector: Connector, + binding: ConnectorBinding, + /, + ) -> ConnectorRunResult: + """Run one Connector binding with durable Sources and checkpoint ordering.""" + + services = self._services_for(binding.scope_id) + source_store, source_catalog = services.sources() + lifecycle = ConnectorLifecycle( + sink=CatalogConnectorSourceSink( + scope_id=services.scope_id, + catalog=source_catalog, + store=source_store, + ), + checkpoints=RelationalConnectorCheckpointStore( + self.database, + self.repositories.connector_checkpoints, + ), + ) + return await lifecycle.run(connector, binding) + async def estimate_recall_tokens( self, scope_id: str, diff --git a/src/powercontext/builtin/sources/__init__.py b/src/powercontext/builtin/sources/__init__.py index 109a21316..5ca040828 100644 --- a/src/powercontext/builtin/sources/__init__.py +++ b/src/powercontext/builtin/sources/__init__.py @@ -18,11 +18,9 @@ CONTENT_SOURCE_ADAPTER, CONTENT_SOURCE_DEFINITION, CONTENT_SOURCE_NAME, - TEXT_EVIDENCE_PROJECTION_KEY, ContentCapture, ContentSource, ContentSourceAdapter, - ContentTextEvidence, ContentTextEvidenceProjection, ) from powercontext.builtin.sources.external_skill import ( @@ -40,11 +38,22 @@ SourceJournalEntry, validate_scope_id, ) +from powercontext.builtin.sources.projections import TEXT_EVIDENCE_PROJECTION_KEY, TextEvidence +from powercontext.builtin.sources.text_file import ( + TEXT_FILE_SNAPSHOT_SOURCE_ADAPTER, + TEXT_FILE_SNAPSHOT_SOURCE_DEFINITION, + TEXT_FILE_SNAPSHOT_SOURCE_NAME, + TextFileEvidenceProjection, + TextFileSnapshotCapture, + TextFileSnapshotSource, + TextFileSnapshotSourceAdapter, +) from powercontext.sources import SourceDefinitionRegistry BUILTIN_SOURCE_REGISTRY = SourceDefinitionRegistry(( CONTENT_SOURCE_DEFINITION, EXTERNAL_SKILL_SNAPSHOT_SOURCE_DEFINITION, + TEXT_FILE_SNAPSHOT_SOURCE_DEFINITION, )) __all__ = [ @@ -56,10 +65,12 @@ "EXTERNAL_SKILL_SNAPSHOT_SOURCE_DEFINITION", "EXTERNAL_SKILL_SNAPSHOT_SOURCE_NAME", "TEXT_EVIDENCE_PROJECTION_KEY", + "TEXT_FILE_SNAPSHOT_SOURCE_ADAPTER", + "TEXT_FILE_SNAPSHOT_SOURCE_DEFINITION", + "TEXT_FILE_SNAPSHOT_SOURCE_NAME", "ContentCapture", "ContentSource", "ContentSourceAdapter", - "ContentTextEvidence", "ContentTextEvidenceProjection", "ExternalSkillImportMode", "ExternalSkillSnapshotCapture", @@ -68,5 +79,10 @@ "SourceCursor", "SourceJournal", "SourceJournalEntry", + "TextEvidence", + "TextFileEvidenceProjection", + "TextFileSnapshotCapture", + "TextFileSnapshotSource", + "TextFileSnapshotSourceAdapter", "validate_scope_id", ] diff --git a/src/powercontext/builtin/sources/content.py b/src/powercontext/builtin/sources/content.py index c444b8020..f008777bb 100644 --- a/src/powercontext/builtin/sources/content.py +++ b/src/powercontext/builtin/sources/content.py @@ -20,11 +20,11 @@ from pydantic import BaseModel, Field, JsonValue, field_validator -from powercontext.sources import AdapterSourceDefinition, SourceProjectionKey +from powercontext.builtin.sources.projections import TEXT_EVIDENCE_PROJECTION_KEY, TextEvidence +from powercontext.sources import AdapterSourceDefinition from powercontext.sources.models import Source, SourceMaterialization CONTENT_SOURCE_NAME = "content" -TEXT_EVIDENCE_PROJECTION_KEY = SourceProjectionKey(name="powercontext.builtin.text-evidence", version="1") NonEmptyText = Annotated[str, Field(min_length=1)] @@ -50,15 +50,6 @@ class ContentSource(Source): metadata: dict[str, JsonValue] = Field(default_factory=dict) -class ContentTextEvidence(BaseModel): - """Schema for the built-in text evidence projection.""" - - source_type: str - source_id: str - content: str - metadata: dict[str, JsonValue] = Field(default_factory=dict) - - class ContentSourceAdapter: """Resolve and read the runtime's built-in captured-text Source.""" @@ -88,10 +79,10 @@ class ContentTextEvidenceProjection: name = TEXT_EVIDENCE_PROJECTION_KEY.name version = TEXT_EVIDENCE_PROJECTION_KEY.version source_class = ContentSource - output_class: type[BaseModel] = ContentTextEvidence + output_class: type[BaseModel] = TextEvidence - def project(self, source: ContentSource, /) -> ContentTextEvidence: - return ContentTextEvidence( + def project(self, source: ContentSource, /) -> TextEvidence: + return TextEvidence( source_type=CONTENT_SOURCE_NAME, source_id=source.name, content=source.content, diff --git a/src/powercontext/builtin/sources/projections.py b/src/powercontext/builtin/sources/projections.py new file mode 100644 index 000000000..2ee6657b6 --- /dev/null +++ b/src/powercontext/builtin/sources/projections.py @@ -0,0 +1,33 @@ +# Copyright (c) 2026 OceanBase. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Built-in named projection schemas shared by Source Definitions.""" + +from pydantic import BaseModel, Field, JsonValue + +from powercontext.sources import SourceProjectionKey + +TEXT_EVIDENCE_PROJECTION_KEY = SourceProjectionKey(name="powercontext.builtin.text-evidence", version="1") + + +class TextEvidence(BaseModel): + """Canonical JSON shape consumed as textual Artifact evidence.""" + + source_type: str + source_id: str + content: str + metadata: dict[str, JsonValue] = Field(default_factory=dict) + + +__all__ = ["TEXT_EVIDENCE_PROJECTION_KEY", "TextEvidence"] diff --git a/src/powercontext/builtin/sources/text_file.py b/src/powercontext/builtin/sources/text_file.py new file mode 100644 index 000000000..659254c05 --- /dev/null +++ b/src/powercontext/builtin/sources/text_file.py @@ -0,0 +1,174 @@ +# Copyright (c) 2026 OceanBase. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Typed captured text-file snapshots for filesystem Connectors.""" + +from __future__ import annotations + +import hashlib +from datetime import datetime +from pathlib import PurePosixPath +from typing import Literal + +from pydantic import BaseModel, JsonValue, field_validator + +from powercontext.builtin.sources.projections import TEXT_EVIDENCE_PROJECTION_KEY, TextEvidence +from powercontext.sources import AdapterSourceDefinition, Source, SourceMaterialization + +TEXT_FILE_SNAPSHOT_SOURCE_NAME = "text-file-snapshot" + + +class TextFileSnapshotCapture(BaseModel): + """One UTF-8 file value captured with non-authoritative provider annotations.""" + + namespace: str + path: str + content: str + media_type: str = "text/plain" + encoding: Literal["utf-8"] = "utf-8" + etag: str | None = None + provider_version: str | None = None + modified_at: datetime | None = None + + @field_validator("namespace", "media_type") + @classmethod + def validate_trimmed_text(cls, value: str) -> str: + if not value or value.strip() != value: + raise ValueError("value must be a non-empty trimmed string") # noqa: TRY003 + return value + + @field_validator("path") + @classmethod + def validate_path(cls, value: str) -> str: + if not value or value.strip() != value or "\\" in value: + raise ValueError("path must be a non-empty normalized POSIX path") # noqa: TRY003 + path = PurePosixPath(value) + if path.is_absolute() or value != path.as_posix() or any(part in {"", ".", ".."} for part in path.parts): + raise ValueError("path must be a relative normalized POSIX path") # noqa: TRY003 + return value + + @field_validator("etag", "provider_version") + @classmethod + def validate_optional_annotation(cls, value: str | None) -> str | None: + if value is not None and (not value or value.strip() != value): + raise ValueError("annotation must be a non-empty trimmed string") # noqa: TRY003 + return value + + +class TextFileSnapshotSource(Source): + """Captured text-file bytes with explicit filesystem provenance.""" + + namespace: str + path: str + content: str + media_type: str + encoding: Literal["utf-8"] + content_digest: str + size: int + etag: str | None = None + provider_version: str | None = None + modified_at: datetime | None = None + + +class TextFileSnapshotSourceAdapter: + """Canonicalize UTF-8 file captures into immutable snapshot Sources.""" + + input_class = TextFileSnapshotCapture + name = TEXT_FILE_SNAPSHOT_SOURCE_NAME + source_class = TextFileSnapshotSource + + async def resolve(self, value: TextFileSnapshotCapture, /) -> TextFileSnapshotSource: + content_bytes = value.content.encode(value.encoding) + content_digest = f"sha256:{hashlib.sha256(content_bytes).hexdigest()}" + source_id = _snapshot_id(value.namespace, value.path, content_digest) + return TextFileSnapshotSource( + name=source_id, + materialization=SourceMaterialization.CAPTURED, + description=f"Captured text file {value.path}", + namespace=value.namespace, + path=value.path, + content=value.content, + media_type=value.media_type, + encoding=value.encoding, + content_digest=content_digest, + size=len(content_bytes), + etag=value.etag, + provider_version=value.provider_version, + modified_at=value.modified_at, + ) + + async def read(self, source: TextFileSnapshotSource, /) -> TextFileSnapshotCapture: + return TextFileSnapshotCapture( + namespace=source.namespace, + path=source.path, + content=source.content, + media_type=source.media_type, + encoding=source.encoding, + etag=source.etag, + provider_version=source.provider_version, + modified_at=source.modified_at, + ) + + +class TextFileEvidenceProjection: + """Expose one file snapshot through the shared text-evidence capability.""" + + name = TEXT_EVIDENCE_PROJECTION_KEY.name + version = TEXT_EVIDENCE_PROJECTION_KEY.version + source_class = TextFileSnapshotSource + output_class: type[BaseModel] = TextEvidence + + def project(self, source: TextFileSnapshotSource, /) -> TextEvidence: + metadata: dict[str, JsonValue] = { + "namespace": source.namespace, + "path": source.path, + "media_type": source.media_type, + "encoding": source.encoding, + "content_digest": source.content_digest, + "size": source.size, + } + if source.etag is not None: + metadata["etag"] = source.etag + if source.provider_version is not None: + metadata["provider_version"] = source.provider_version + if source.modified_at is not None: + metadata["modified_at"] = source.modified_at.isoformat() + return TextEvidence( + source_type=TEXT_FILE_SNAPSHOT_SOURCE_NAME, + source_id=source.name, + content=source.content, + metadata=metadata, + ) + + +def _snapshot_id(namespace: str, path: str, content_digest: str) -> str: + identity = "\0".join((namespace, path, content_digest)) + return f"text_file_{hashlib.sha256(identity.encode()).hexdigest()}" + + +TEXT_FILE_SNAPSHOT_SOURCE_ADAPTER = TextFileSnapshotSourceAdapter() +TEXT_FILE_SNAPSHOT_SOURCE_DEFINITION = AdapterSourceDefinition( + TEXT_FILE_SNAPSHOT_SOURCE_ADAPTER, + projections=(TextFileEvidenceProjection(),), +) + +__all__ = [ + "TEXT_FILE_SNAPSHOT_SOURCE_ADAPTER", + "TEXT_FILE_SNAPSHOT_SOURCE_DEFINITION", + "TEXT_FILE_SNAPSHOT_SOURCE_NAME", + "TextFileEvidenceProjection", + "TextFileSnapshotCapture", + "TextFileSnapshotSource", + "TextFileSnapshotSourceAdapter", +] diff --git a/src/powercontext/sources/connectors.py b/src/powercontext/sources/connectors.py index 774bc85d8..c3973fa96 100644 --- a/src/powercontext/sources/connectors.py +++ b/src/powercontext/sources/connectors.py @@ -194,15 +194,7 @@ async def submit( ) -> ConnectorSubmissionResult: """Submit one item and record success, rejection, or sink failure exactly once.""" - _require_trimmed("item_id", item_id) - if item_id in self._item_ids: - raise InvalidConnectorRunError("duplicate-item", f"item {item_id!r} was submitted more than once") - if definition_name not in self._source_definitions: - raise InvalidConnectorRunError( - "undeclared-definition", - f"Connector did not declare Source Definition {definition_name!r}", - ) - self._item_ids.add(item_id) + self._claim_item(item_id, definition_name) try: result = await self._sink.submit(self.binding, item_id, definition_name, value) except Exception as error: @@ -226,6 +218,59 @@ async def submit( ) return result + def reject(self, item_id: str, definition_name: str, detail: str, /) -> ConnectorSubmissionResult: + """Record one provider item that cannot satisfy its Source Definition.""" + + return self._record_provider_outcome( + item_id, + definition_name, + ConnectorSubmissionStatus.REJECTED, + detail, + ) + + def fail(self, item_id: str, definition_name: str, detail: str, /) -> ConnectorSubmissionResult: + """Record one provider item that could not be acquired safely.""" + + return self._record_provider_outcome( + item_id, + definition_name, + ConnectorSubmissionStatus.FAILED, + detail, + ) + + def _record_provider_outcome( + self, + item_id: str, + definition_name: str, + status: ConnectorSubmissionStatus, + detail: str, + ) -> ConnectorSubmissionResult: + _require_trimmed("detail", detail) + if status not in {ConnectorSubmissionStatus.REJECTED, ConnectorSubmissionStatus.FAILED}: + raise InvalidConnectorRunError("provider-outcome", "must be rejected or failed") + self._claim_item(item_id, definition_name) + result = ConnectorSubmissionResult(status=status, detail=detail) + self._outcomes.append( + ConnectorItemOutcome( + item_id=item_id, + definition_name=definition_name, + status=status, + detail=detail, + ) + ) + return result + + def _claim_item(self, item_id: str, definition_name: str) -> None: + _require_trimmed("item_id", item_id) + if item_id in self._item_ids: + raise InvalidConnectorRunError("duplicate-item", f"item {item_id!r} was submitted more than once") + if definition_name not in self._source_definitions: + raise InvalidConnectorRunError( + "undeclared-definition", + f"Connector did not declare Source Definition {definition_name!r}", + ) + self._item_ids.add(item_id) + class ConnectorLifecycle: """Run Connectors while enforcing durable checkpoint ordering.""" @@ -258,7 +303,7 @@ async def run(self, connector: Connector, binding: ConnectorBinding, /) -> Conne if outcome.status in {ConnectorSubmissionStatus.REJECTED, ConnectorSubmissionStatus.FAILED} ) committed = previous - if completion.checkpoint != previous and not unsafe: + if completion.status is ConnectorRunStatus.COMPLETE and completion.checkpoint != previous and not unsafe: await self._checkpoints.save(binding, completion.checkpoint, expected=previous) committed = completion.checkpoint return ConnectorRunResult( diff --git a/tests/builtin/connectors/test_opendal.py b/tests/builtin/connectors/test_opendal.py new file mode 100644 index 000000000..033fa547f --- /dev/null +++ b/tests/builtin/connectors/test_opendal.py @@ -0,0 +1,234 @@ +# Copyright (c) 2026 OceanBase. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from __future__ import annotations + +import asyncio + +import pytest + +from powercontext.builtin.artifacts.memory import MemoryCandidateRequest, MemoryEntryInput +from powercontext.builtin.connectors import ( + OPENDAL_TEXT_FILE_CONNECTOR_NAME, + OpenDALTextFileConnector, +) +from powercontext.builtin.persistence.sqlite import SQLiteConfig +from powercontext.builtin.runtime import BuiltinConfig, open_builtin_contexts +from powercontext.builtin.sources import ( + TEXT_EVIDENCE_PROJECTION_KEY, + TextFileSnapshotSource, +) +from powercontext.sources import ( + ConnectorBinding, + ConnectorCapability, + ConnectorRunStatus, + ConnectorSubmissionStatus, +) + + +def _binding() -> ConnectorBinding: + return ConnectorBinding( + scope_id="project-a", + binding_id="documents-a", + connector_name=OPENDAL_TEXT_FILE_CONNECTOR_NAME, + connector_version="1", + ) + + +class MemoryFileSystem: + def __init__(self) -> None: + self.files: dict[str, bytes] = {} + + def pipe_file(self, path: str, content: bytes) -> None: + self.files[path] = content + + def find(self, path: str, *, detail: bool) -> dict[str, dict[str, object]]: + assert detail + prefix = f"{path.rstrip('/')}/" if path else "" + return { + name: {"name": name, "size": len(content), "type": "file"} + for name, content in self.files.items() + if not prefix or name.startswith(prefix) + } + + def cat_file(self, path: str) -> bytes: + return self.files[path] + + +def _filesystem() -> MemoryFileSystem: + return MemoryFileSystem() + + +class TextFileCandidatePipeline: + async def extract(self, request: MemoryCandidateRequest, /) -> tuple[MemoryEntryInput, ...]: + return tuple( + MemoryEntryInput( + kind="document", + text=source.content, + sources=(source,), + ) + for source in request.sources + if isinstance(source, TextFileSnapshotSource) + ) + + +def test_opendal_connector_persists_incremental_snapshots_across_runtime_restart(tmp_path) -> None: + async def scenario() -> None: + filesystem = _filesystem() + filesystem.pipe_file("docs/readme.md", b"First value") + filesystem.pipe_file("docs/nested/note.txt", b"Nested value") + filesystem.pipe_file("docs/image.bin", b"\x00\x01") + connector = OpenDALTextFileConnector( + filesystem, + source_namespace="workspace-a", + root="docs", + ) + config = BuiltinConfig(database=SQLiteConfig(url=f"sqlite+aiosqlite:///{tmp_path / 'powercontext.db'}")) + + async with open_builtin_contexts(config) as contexts: + first = await contexts.run_connector(connector, _binding()) + context = await contexts.get("project-a") + first_sources = await context.sources.list() + + assert first.status is ConnectorRunStatus.COMPLETE + assert [item.item_id for item in first.items] == ["nested/note.txt", "readme.md"] + assert all(item.status is ConnectorSubmissionStatus.ACCEPTED for item in first.items) + assert len(first_sources) == 2 + readme = next( + source + for source in first_sources + if isinstance(source, TextFileSnapshotSource) and source.path == "readme.md" + ) + assert context.sources.catalog.project(readme, TEXT_EVIDENCE_PROJECTION_KEY) == { + "source_type": "text-file-snapshot", + "source_id": readme.name, + "content": "First value", + "metadata": { + "namespace": "workspace-a", + "path": "readme.md", + "media_type": "text/markdown", + "encoding": "utf-8", + "content_digest": readme.content_digest, + "size": 11, + }, + } + + async with open_builtin_contexts(config) as contexts: + unchanged = await contexts.run_connector(connector, _binding()) + assert unchanged.previous_checkpoint == first.committed_checkpoint + assert unchanged.items == () + + filesystem.pipe_file("docs/readme.md", b"Second value") + changed = await contexts.run_connector(connector, _binding()) + context = await contexts.get("project-a") + sources = await context.sources.list() + + assert [item.item_id for item in changed.items] == ["readme.md"] + assert len(sources) == 3 + readme_snapshots = [ + source + for source in sources + if isinstance(source, TextFileSnapshotSource) and source.path == "readme.md" + ] + assert {source.content for source in readme_snapshots} == {"First value", "Second value"} + assert len({source.name for source in readme_snapshots}) == 2 + + asyncio.run(scenario()) + + +def test_opendal_connector_keeps_checkpoint_before_a_rejected_item() -> None: + async def scenario() -> None: + filesystem = _filesystem() + filesystem.pipe_file("good.md", b"Good value") + filesystem.pipe_file("invalid.txt", b"\xff") + connector = OpenDALTextFileConnector(filesystem, source_namespace="workspace-a") + + async with open_builtin_contexts(BuiltinConfig(database=SQLiteConfig())) as contexts: + rejected = await contexts.run_connector(connector, _binding()) + context = await contexts.get("project-a") + + assert rejected.status is ConnectorRunStatus.INCOMPLETE + assert rejected.committed_checkpoint is None + assert [(item.item_id, item.status) for item in rejected.items] == [ + ("good.md", ConnectorSubmissionStatus.ACCEPTED), + ("invalid.txt", ConnectorSubmissionStatus.REJECTED), + ] + assert len(await context.sources.list()) == 1 + + filesystem.pipe_file("invalid.txt", b"Recovered value") + recovered = await contexts.run_connector(connector, _binding()) + + assert recovered.status is ConnectorRunStatus.COMPLETE + assert recovered.committed_checkpoint is not None + assert len(await context.sources.list()) == 2 + + asyncio.run(scenario()) + + +def test_opendal_connector_sources_complete_the_memory_ingestion_loop() -> None: + async def scenario() -> None: + filesystem = _filesystem() + filesystem.pipe_file("decision.md", b"Use exact snapshot references.") + connector = OpenDALTextFileConnector(filesystem, source_namespace="workspace-a") + + async with open_builtin_contexts( + BuiltinConfig(database=SQLiteConfig()), + candidate_pipeline=TextFileCandidatePipeline(), + ) as contexts: + connector_result = await contexts.run_connector(connector, _binding()) + context = await contexts.get("project-a") + flush_result = await context.triggers.flush(limit=10) + memory = await context.artifacts.memory.head("memory") + entries = await context.artifacts.memory.entries(memory) + + assert flush_result.source_count == 1 + assert len(entries) == 1 + assert entries[0].text == "Use exact snapshot references." + assert entries[0].sources == (connector_result.items[0].source_ref,) + + asyncio.run(scenario()) + + +def test_opendal_connector_does_not_claim_authoritative_deletion() -> None: + connector = OpenDALTextFileConnector(_filesystem(), source_namespace="workspace-a") + + assert ConnectorCapability.CHECKPOINT_RESUME in connector.capabilities + assert ConnectorCapability.AUTHORITATIVE_DELETION not in connector.capabilities + assert ConnectorCapability.CHANGE_FEED not in connector.capabilities + + +def test_opendal_connector_reads_the_real_opendalfs_memory_backend() -> None: + opendalfs = pytest.importorskip("opendalfs") + + async def scenario() -> None: + filesystem = opendalfs.OpendalFileSystem( + scheme="memory", + asynchronous=False, + skip_instance_cache=True, + ) + filesystem.pipe_file("docs/readme.md", b"OpenDAL value") + connector = OpenDALTextFileConnector( + filesystem, + source_namespace="opendal-memory", + root="docs", + ) + + async with open_builtin_contexts(BuiltinConfig(database=SQLiteConfig())) as contexts: + result = await contexts.run_connector(connector, _binding()) + + assert result.status is ConnectorRunStatus.COMPLETE + assert result.items[0].item_id == "readme.md" + assert result.items[0].status is ConnectorSubmissionStatus.ACCEPTED + + asyncio.run(scenario()) diff --git a/tests/test_connectors.py b/tests/test_connectors.py index 1812569a9..b1708e908 100644 --- a/tests/test_connectors.py +++ b/tests/test_connectors.py @@ -179,6 +179,40 @@ async def scenario() -> None: asyncio.run(scenario()) +def test_connector_does_not_advance_an_incomplete_run_checkpoint() -> None: + class IncompleteConnector(ContentConnector): + async def run(self, session: ConnectorRunSession, /) -> ConnectorRunCompletion: + await session.submit(self.capture.source_id, CONTENT_SOURCE_NAME, self.capture) + return ConnectorRunCompletion( + status=ConnectorRunStatus.INCOMPLETE, + checkpoint={"cursor": 1}, + ) + + async def scenario() -> None: + events: list[str] = [] + store = IdempotentSourceStore(events) + lifecycle = ConnectorLifecycle( + sink=CatalogConnectorSourceSink( + scope_id="scope-a", + catalog=SourceCatalog(backend=store, registry=BUILTIN_SOURCE_REGISTRY), + store=store, + ), + checkpoints=MemoryCheckpointStore(events), + ) + + result = await lifecycle.run( + IncompleteConnector(ContentCapture(source_id="note-1", content="Remember this.")), + _binding(), + ) + + assert result.status is ConnectorRunStatus.INCOMPLETE + assert result.proposed_checkpoint == {"cursor": 1} + assert result.committed_checkpoint is None + assert events == ["source:note-1"] + + asyncio.run(scenario()) + + def test_connector_rejects_duplicate_items_and_binding_mismatches() -> None: class DuplicateConnector(ContentConnector): async def run(self, session: ConnectorRunSession, /) -> ConnectorRunCompletion: diff --git a/uv.lock b/uv.lock index ab5b5a3c3..8688c1996 100644 --- a/uv.lock +++ b/uv.lock @@ -761,6 +761,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/4a/e3/f1fae3647d170919c2cf2a898e77e7d1a4e5c7cae0aed7bb4bd3f5ebff6f/filelock-3.29.5-py3-none-any.whl", hash = "sha256:8af830889ba3a0ffcefbd6c7d2af8a54012058103771f2e10848222f476a1693", size = 45073, upload-time = "2026-07-03T03:50:30.445Z" }, ] +[[package]] +name = "fsspec" +version = "2026.7.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/00/78/f34251dadb8f3921264a1d9b8946f5e542014ee2614b285261b4e40e6775/fsspec-2026.7.0.tar.gz", hash = "sha256:c803c40f4cf860b49dea58ee3e1c33cb9c790520e233537e1340049f89b82a88", size = 317040, upload-time = "2026-07-28T16:34:51.052Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/fd/3c/6a2bf344106328fd04963664a60b9bb6496fc25df8e962fcdc1367285fb9/fsspec-2026.7.0-py3-none-any.whl", hash = "sha256:b57ddbafedfaef7018c1ecab32aa200a9d7ca26b77965f64e48b70061249d279", size = 206583, upload-time = "2026-07-28T16:34:49.538Z" }, +] + [[package]] name = "genai-prices" version = "0.1.2" @@ -987,7 +996,7 @@ name = "importlib-metadata" version = "9.0.0" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "zipp" }, + { name = "zipp", marker = "python_full_version < '3.12'" }, ] sdist = { url = "https://files.pythonhosted.org/packages/a9/01/15bb152d77b21318514a96f43af312635eb2500c96b55398d020c93d86ea/importlib_metadata-9.0.0.tar.gz", hash = "sha256:a4f57ab599e6a2e3016d7595cfd72eb4661a5106e787a95bcc90c7105b831efc", size = 56405, upload-time = "2026-03-20T06:42:56.999Z" } wheels = [ @@ -1817,6 +1826,45 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/12/cf/03675d8bd8ecbf4445504d8071adab19f5f993676795708e36402ab38263/openapi_pydantic-0.5.1-py3-none-any.whl", hash = "sha256:a3a09ef4586f5bd760a8df7f43028b60cafb6d9f61de2acba9574766255ab146", size = 96381, upload-time = "2025-01-08T19:29:25.275Z" }, ] +[[package]] +name = "opendal" +version = "0.47.6" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/97/a3/898b9097795c4015a3329dc3f99a96350570769367ddea7a9d087ffb1a05/opendal-0.47.6.tar.gz", hash = "sha256:297b876ab44162490d4e38041ecb712800ad14814ee35c4c3c196af4b26ffd61", size = 1801677, upload-time = "2026-08-20T17:32:58.452Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f4/8e/51d2c1cfabb4052cf7bbf3aef80a49a699f3e5d55f7ebd67b9de3830ad15/opendal-0.47.6-cp311-abi3-macosx_10_12_x86_64.whl", hash = "sha256:ce28bee2ef19c04d16033b602604d4d26eedb295c7868ced2c55ef1255aa695b", size = 17705330, upload-time = "2026-08-20T17:32:19.061Z" }, + { url = "https://files.pythonhosted.org/packages/24/fa/84e3bb3e2ff046fd9b55c870badf929a51cf6e399843c125aa7da9f958f5/opendal-0.47.6-cp311-abi3-macosx_11_0_arm64.whl", hash = "sha256:94a481ba0931df63e361347efb02bb4e715c63b12ac5f3cb3bfb8f2f6bbed527", size = 16330575, upload-time = "2026-08-20T17:32:20.956Z" }, + { url = "https://files.pythonhosted.org/packages/28/d9/315ce33d51500222ad5e47769d423d9eff51b2b67275872b3afc03bf6a86/opendal-0.47.6-cp311-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:586dd31f6b0e337f5037d6f80035a5b8c615ed56fe0a13b72ece68c6675ed749", size = 16923201, upload-time = "2026-08-20T17:32:23.465Z" }, + { url = "https://files.pythonhosted.org/packages/1a/04/ddb50902fef4ac5f9a201f16cf1041ba7b4cf351bdce5fcf3dd26cb91012/opendal-0.47.6-cp311-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ee1443cbab2e95fd55ad35c890284693ced5132e74ab63e9953ecabb15efe4f5", size = 18146763, upload-time = "2026-08-20T17:32:25.583Z" }, + { url = "https://files.pythonhosted.org/packages/d4/0b/b0a993271eb0745e428c1358401af52d1293dddc7d758e5b3bde64ea2fac/opendal-0.47.6-cp311-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:a9c457824b161f8351801c03632c9b617f32dd597b2a6fd9c88083c8c9428a1a", size = 17220213, upload-time = "2026-08-20T17:32:27.878Z" }, + { url = "https://files.pythonhosted.org/packages/40/4b/cd2cd193a6cb250699cc4b816baebac9cf389d34bbaaac4dba9d21dedca2/opendal-0.47.6-cp311-abi3-musllinux_1_1_aarch64.whl", hash = "sha256:8be0535b4cbd28115458a9bc194030fde49ce9a626f0c225b6f900203a942cfc", size = 17561196, upload-time = "2026-08-20T17:32:30.343Z" }, + { url = "https://files.pythonhosted.org/packages/f9/e6/d2061fdef84f173e580dd42fedfc99e7c43b7bbe9bea4d0f271464cdbdb1/opendal-0.47.6-cp311-abi3-musllinux_1_1_armv7l.whl", hash = "sha256:4e815a0120f95d9ae37312f5c6b2e8da2010314c0c55b18cfb4c90384bcf2743", size = 17227288, upload-time = "2026-08-20T17:32:32.469Z" }, + { url = "https://files.pythonhosted.org/packages/57/09/51f24e3540f0777d0433f2559e8158e2297eb120265544b4be2d831f280a/opendal-0.47.6-cp311-abi3-musllinux_1_1_x86_64.whl", hash = "sha256:ac3519eb30648d5f8d159dd29e1c54dac1a5730c8fe656ccfd1664fd042e8b8a", size = 18373732, upload-time = "2026-08-20T17:32:34.66Z" }, + { url = "https://files.pythonhosted.org/packages/85/cd/45e8484b5b141e9605b49a8fe3b92c747d486d59a04da868dc5ae62ebb9a/opendal-0.47.6-cp311-abi3-win_amd64.whl", hash = "sha256:81d52f5919b0845f747eeef16bf41a968ad11a16ede3a686556c75b1d0903298", size = 19421219, upload-time = "2026-08-20T17:32:36.991Z" }, + { url = "https://files.pythonhosted.org/packages/75/c9/ce51d2f5cda6f961d77dd376c3a7db9c74fac8769a36a2d91f3a09c1f03e/opendal-0.47.6-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:8142f90419334550cc42ea655184e3a5744909d8cf8a53462cf22849b5448a06", size = 17713841, upload-time = "2026-08-20T17:32:39.279Z" }, + { url = "https://files.pythonhosted.org/packages/25/c2/8daa8ff57682104992fce97e3f776e3289516538e1267b8fcbc94eb79939/opendal-0.47.6-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:19ca7f2c9751fc1279d4803b65cf27c0ea55d035e29601e263cfcc4b31c6435f", size = 16314577, upload-time = "2026-08-20T17:32:41.397Z" }, + { url = "https://files.pythonhosted.org/packages/e7/0d/bc5a06ddca9d8d30bc5ca2fd1c5fd16c411ecf15e3523376760a93d9a972/opendal-0.47.6-cp314-cp314t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:dfa9b490c327745032ea865dd28cc5b2282b1a5e1ed6fe03e2e3320a74677479", size = 16917195, upload-time = "2026-08-20T17:32:43.532Z" }, + { url = "https://files.pythonhosted.org/packages/9c/61/c70cec20351311caf33c2a261e4e615798f0e733c68f84b81983b329bb46/opendal-0.47.6-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:43a0a12144066fff5246a8e34116724e8248bc5c3c34c6998fcea5d01586bfa1", size = 18136350, upload-time = "2026-08-20T17:32:45.701Z" }, + { url = "https://files.pythonhosted.org/packages/ad/57/15358c7e9455f34a5abe6dc156024d7681b78d64c4e2f2db9300e3a1ca18/opendal-0.47.6-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:71e99848b40787e9505f5423473488bc818499c92707dc168037a047d297be1b", size = 17208049, upload-time = "2026-08-20T17:32:47.801Z" }, + { url = "https://files.pythonhosted.org/packages/d4/f5/72cea960b30c643cd3b216d2ad4af3c14ce503c21045b75a7ea23905d1de/opendal-0.47.6-cp314-cp314t-musllinux_1_1_aarch64.whl", hash = "sha256:64ee2b40a4f457f60ddfb822a5263962ff1bc07399403ceff0e90a175defafef", size = 17558165, upload-time = "2026-08-20T17:32:49.903Z" }, + { url = "https://files.pythonhosted.org/packages/0c/6f/ae82bedcc1acc2d1047931c4aefd750afecc2f66b2e7967a3afdbffdbe48/opendal-0.47.6-cp314-cp314t-musllinux_1_1_armv7l.whl", hash = "sha256:0e3532a848e3473e1787ed29a745b85b699938a1e01eeea390ace833405fd545", size = 17221105, upload-time = "2026-08-20T17:32:52.026Z" }, + { url = "https://files.pythonhosted.org/packages/24/5b/de96032981f74e0e473c4e81efd4357657aacf4673c3e09b8c3cd665d5d5/opendal-0.47.6-cp314-cp314t-musllinux_1_1_x86_64.whl", hash = "sha256:0ab143680d79a189deae5cb9ed2dbd7adbe967bfd82947317f09f02709553a21", size = 18367917, upload-time = "2026-08-20T17:32:54.371Z" }, + { url = "https://files.pythonhosted.org/packages/57/c3/da0429e7da22bbddd2708259e134f24d228e862404f8c430fa4bdcd4f4b9/opendal-0.47.6-cp314-cp314t-win_amd64.whl", hash = "sha256:60673bbf72aad5f1b1be37be5c25014a852349b1e395671731cd49c6792aae04", size = 19402260, upload-time = "2026-08-20T17:32:56.707Z" }, +] + +[[package]] +name = "opendalfs" +version = "0.1.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "fsspec", marker = "python_full_version >= '3.12'" }, + { name = "opendal", marker = "python_full_version >= '3.12'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/0d/23/fdb4d34f08ac68a27b2d553256e13b15ce139916d36bab33f9a0e71e8770/opendalfs-0.1.0.tar.gz", hash = "sha256:5ceaeccc0852ef10c8ef2ce20e5927dbc18083909c06550ad36f81bcacb5774d", size = 75112, upload-time = "2026-08-26T05:30:33.497Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ec/7d/baac5ab7f0fcd4d4f06484531e30dbee2eaf924cc0bf007572bc54bbc14a/opendalfs-0.1.0-py3-none-any.whl", hash = "sha256:8b180c9dae767053023ea4923250ce66d40a9bcad2e6f947df5ec8152208365b", size = 16278, upload-time = "2026-08-26T05:30:32.22Z" }, +] + [[package]] name = "opentelemetry-api" version = "1.43.0" @@ -2085,6 +2133,9 @@ client = [ { name = "opentelemetry-api" }, { name = "pydantic-settings" }, ] +opendal = [ + { name = "opendalfs", marker = "python_full_version >= '3.12'" }, +] seekdb = [ { name = "aiosqlite" }, { name = "apscheduler" }, @@ -2150,6 +2201,7 @@ requires-dist = [ { name = "httpx", extras = ["socks"], marker = "extra == 'cli'", specifier = ">=0.28,<1" }, { name = "httpx", extras = ["socks"], marker = "extra == 'client'", specifier = ">=0.28,<1" }, { name = "jinja2", marker = "extra == 'server'", specifier = ">=3.1,<4" }, + { name = "opendalfs", marker = "python_full_version >= '3.12' and extra == 'opendal'", specifier = ">=0.1,<0.2" }, { name = "opentelemetry-api", marker = "extra == 'cli'", specifier = ">=1.30,<2" }, { name = "opentelemetry-api", marker = "extra == 'client'", specifier = ">=1.30,<2" }, { name = "opentelemetry-api", marker = "extra == 'server'", specifier = ">=1.30,<2" }, @@ -2182,7 +2234,7 @@ requires-dist = [ { name = "typing-extensions", specifier = ">=4.12,<5" }, { name = "uvicorn", marker = "extra == 'server'", specifier = ">=0.34,<1" }, ] -provides-extras = ["builtin", "cli", "client", "seekdb", "server", "tracing-otlp"] +provides-extras = ["builtin", "cli", "client", "opendal", "seekdb", "server", "tracing-otlp"] [package.metadata.requires-dev] dev = [ diff --git a/zensical.toml b/zensical.toml index d79245750..b420a477d 100644 --- a/zensical.toml +++ b/zensical.toml @@ -37,6 +37,7 @@ nav = [ { "Configure WorkBuddy" = "en/docs/how-to/configure-workbuddy.md" }, { "Configure OpenClaw" = "en/docs/how-to/configure-openclaw.md" }, { "Configure OpenCode" = "en/docs/how-to/configure-opencode.md" }, + { "Ingest text files with OpenDAL" = "en/docs/how-to/ingest-text-files-with-opendal.md" }, { "Trace with Phoenix" = "en/docs/how-to/trace-with-phoenix.md" }, ] }, { "Reference" = [ @@ -115,6 +116,7 @@ nav = [ { "配置 WorkBuddy" = "zh/docs/how-to/configure-workbuddy.md" }, { "配置 OpenClaw" = "zh/docs/how-to/configure-openclaw.md" }, { "配置 OpenCode" = "zh/docs/how-to/configure-opencode.md" }, + { "使用 OpenDAL 采集文本文件" = "zh/docs/how-to/ingest-text-files-with-opendal.md" }, { "用 Phoenix 查看 trace" = "zh/docs/how-to/trace-with-phoenix.md" }, ] }, { "参考" = [ From b4885bdc131a0c10d4c928c4a361569493c8d13f Mon Sep 17 00:00:00 2001 From: Chojan Shang Date: Thu, 27 Aug 2026 19:48:20 +0800 Subject: [PATCH 5/9] feat(connectors): add remote worker ingestion --- .../how-to/ingest-text-files-with-opendal.md | 127 ++++---- ...source_definition_and_observation_model.md | 80 +++-- .../how-to/ingest-text-files-with-opendal.md | 128 ++++---- ...source_definition_and_observation_model.md | 65 ++-- .../powercontext/src/operations.generated.ts | 4 + .../powercontext/src/operations.generated.ts | 4 + integrations/opendal/README.md | 27 ++ integrations/opendal/pyproject.toml | 29 ++ .../__init__.py | 20 +- .../src/powercontext_connector_opendal/cli.py | 95 ++++++ .../connector.py | 10 +- .../powercontext_connector_opendal/source.py | 11 +- .../powercontext/src/operations.generated.ts | 4 + openapi/powercontext.yaml | 273 ++++++++++++++++ pyproject.toml | 7 +- src/powercontext/__init__.py | 4 + .../builtin/persistence/__init__.py | 6 + .../builtin/persistence/source_definitions.py | 130 ++++++++ .../builtin/persistence/sources.py | 39 ++- .../builtin/persistence/tables.py | 11 + src/powercontext/builtin/runtime/__init__.py | 11 +- .../builtin/runtime/application.py | 38 ++- .../builtin/runtime/composition.py | 1 + src/powercontext/builtin/runtime/models.py | 30 +- src/powercontext/builtin/runtime/protocols.py | 29 +- .../builtin/runtime/relational.py | 175 +++++++++-- src/powercontext/builtin/sources/__init__.py | 20 +- src/powercontext/builtin/sources/content.py | 3 +- src/powercontext/client/__init__.py | 8 + src/powercontext/client/client.py | 34 ++ src/powercontext/client/ingestion.py | 176 +++++++++++ src/powercontext/errors.py | 9 + src/powercontext/http/__init__.py | 24 ++ src/powercontext/http/_generated/models.py | 111 +++++++ .../http/_generated/operations.py | 80 +++++ src/powercontext/http/_generated/schema.py | 231 ++++++++++++++ src/powercontext/server/app.py | 95 +++++- src/powercontext/server/mapping.py | 81 +++++ src/powercontext/sources/__init__.py | 19 ++ src/powercontext/sources/catalog.py | 9 + src/powercontext/sources/connectors.py | 6 +- src/powercontext/sources/observations.py | 228 ++++++++++++++ .../{builtin => }/sources/projections.py | 6 +- tests/builtin/connectors/test_opendal.py | 234 -------------- tests/integrations/test_opendal_connector.py | 296 ++++++++++++++++++ tests/test_source_observations.py | 71 +++++ uv.lock | 60 +--- 47 files changed, 2617 insertions(+), 542 deletions(-) create mode 100644 integrations/opendal/README.md create mode 100644 integrations/opendal/pyproject.toml rename {src/powercontext/builtin/connectors => integrations/opendal/src/powercontext_connector_opendal}/__init__.py (54%) create mode 100644 integrations/opendal/src/powercontext_connector_opendal/cli.py rename src/powercontext/builtin/connectors/opendal.py => integrations/opendal/src/powercontext_connector_opendal/connector.py (99%) rename src/powercontext/builtin/sources/text_file.py => integrations/opendal/src/powercontext_connector_opendal/source.py (96%) create mode 100644 src/powercontext/builtin/persistence/source_definitions.py create mode 100644 src/powercontext/client/ingestion.py create mode 100644 src/powercontext/sources/observations.py rename src/powercontext/{builtin => }/sources/projections.py (84%) delete mode 100644 tests/builtin/connectors/test_opendal.py create mode 100644 tests/integrations/test_opendal_connector.py create mode 100644 tests/test_source_observations.py diff --git a/docs/en/docs/how-to/ingest-text-files-with-opendal.md b/docs/en/docs/how-to/ingest-text-files-with-opendal.md index 1ff5069ff..7c349cf1e 100644 --- a/docs/en/docs/how-to/ingest-text-files-with-opendal.md +++ b/docs/en/docs/how-to/ingest-text-files-with-opendal.md @@ -1,90 +1,95 @@ --- title: Ingest text files with OpenDAL -description: Capture UTF-8 files as typed Sources through an OpenDAL storage backend. +description: Capture UTF-8 files as typed Sources with an independent OpenDAL Connector worker. --- # Ingest text files with OpenDAL -Use `OpenDALTextFileConnector` to capture bounded UTF-8 files from a storage backend supported by OpenDAL. Each accepted -file becomes an immutable `text-file-snapshot` Source with its path, namespace, content digest, and available provider -annotations. +`powercontext-connector-opendal` is deployed independently from PowerContext Server. It owns OpenDAL credentials, +provider configuration, the executable Source Definition, and file reads. The Server stores only a declarative +Definition manifest, materialized Source observations, named projections, and opaque checkpoints. ## Before you begin -The OpenDAL integration requires Python 3.12 or later. Install the optional dependency: +The integration requires Python 3.12 or later. Start PowerContext Server, then install the worker from a checkout: ```bash -uv add "powercontext[opendal]" +uv tool install ./integrations/opendal ``` -Choose a stable `source_namespace` for the storage location. It distinguishes identical paths and bytes captured from -different authorities. Do not put credentials in the namespace. +Choose a stable `source_namespace` that distinguishes storage authorities. Do not put credentials in the namespace, +Source payload, or checkpoint. If Server authentication is enabled, provide its bearer token through the +`POWERCONTEXT_TOKEN` environment variable. -## Run a local filesystem binding +## Run a binding -The following binding scans the `docs` directory below `/absolute/path/to/project` and persists its checkpoint in the -same PowerContext database as the captured Sources: +This independent process scans `/absolute/path/to/project/docs`. The `binding_id` identifies checkpoint continuity; +the `scope_id` determines which Scope owns accepted Sources: -```python -import asyncio - -from powercontext.builtin.connectors import OpenDALTextFileConnector -from powercontext.builtin.persistence.sqlite import SQLiteConfig -from powercontext.builtin.runtime import BuiltinConfig, open_builtin_contexts -from powercontext.sources import ConnectorBinding - - -async def main() -> None: - connector = OpenDALTextFileConnector.from_service( - "fs", - source_namespace="project-docs", - root="docs", - storage_options={"root": "/absolute/path/to/project"}, - ) - binding = ConnectorBinding( - scope_id="project:example", - binding_id="project-docs", - connector_name=connector.name, - connector_version=connector.version, - ) - config = BuiltinConfig( - database=SQLiteConfig(url="sqlite+aiosqlite:///powercontext.db"), - ) - - async with open_builtin_contexts(config) as contexts: - result = await contexts.run_connector(connector, binding) - print(result.model_dump_json(indent=2)) - - -asyncio.run(main()) +```bash +powercontext-connector-opendal \ + --base-url http://127.0.0.1:8765 \ + --scope-id project:example \ + --binding-id project-docs \ + --service fs \ + --storage-option root=/absolute/path/to/project \ + --root docs \ + --source-namespace project-docs ``` -Use a different OpenDAL service and its backend options for remote storage. `storage_options` are runtime-only and are -not copied into Source payloads or checkpoints. +For remote storage, replace the OpenDAL service and pass its `--storage-option KEY=VALUE` arguments. These options +remain inside the worker process and are never sent through the ingestion API. + +On every run, the worker idempotently registers the `text-file-snapshot` Definition manifest, reads the binding +checkpoint, submits changed Source observations, and compare-and-swaps the checkpoint after every durable receipt. +Use cron, a Kubernetes Job, or another external scheduler to run the command periodically. -## Interpret the result +## Embed the lifecycle in a worker -An item outcome reports one of four states: +Use the generic remote lifecycle when a deployment needs custom supervision or schedules multiple bindings: + +```python +from powercontext.client import PowerContextClient, RemoteConnectorWorker +from powercontext.sources import ConnectorBinding, SourceDefinitionRegistry +from powercontext_connector_opendal import ( + TEXT_FILE_SNAPSHOT_SOURCE_DEFINITION, + OpenDALTextFileConnector, +) + +connector = OpenDALTextFileConnector.from_service( + "fs", + source_namespace="project-docs", + root="docs", + storage_options={"root": "/absolute/path/to/project"}, +) +binding = ConnectorBinding( + scope_id="project:example", + binding_id="project-docs", + connector_name=connector.name, + connector_version=connector.version, +) +registry = SourceDefinitionRegistry((TEXT_FILE_SNAPSHOT_SOURCE_DEFINITION,)) + +async with PowerContextClient("http://127.0.0.1:8765") as client: + result = await RemoteConnectorWorker(client=client, registry=registry).run(connector, binding) +``` -- `accepted`: the Source was durably stored; -- `replayed`: the sink recognized an already accepted Source; -- `rejected`: the provider item could not satisfy the Source Definition, such as invalid UTF-8; -- `failed`: the item could not be read or stored safely. +## Runtime semantics -The checkpoint advances only after every selected item is accepted and the run completes. A rejected or failed item -leaves the previous checkpoint in place, so the next run safely retries the scan. Files whose digest matches the -committed checkpoint are skipped. +Each item outcome is `accepted`, `replayed`, `rejected`, or `failed`. The checkpoint advances only when the run +completes without rejected or failed items. Otherwise the prior checkpoint remains, and the next run safely retries +from it. Files whose digest matches the committed checkpoint are skipped. -Accepted Sources enter the same scoped Source journal used by Memory extraction. When the runtime has a Memory -candidate pipeline, its normal source-window flush or schedule can consume these Sources through the shared -`powercontext.builtin.text-evidence` projection. Connector completion does not itself create Memory. +Accepted Sources enter the target Scope's Source journal. The worker also computes the standard +`powercontext.text-evidence` projection, so Memory consumers need not understand the native `text-file-snapshot` +schema. A Connector run does not create Memory directly; the normal source-window flush or schedule still does that. -## Current limits +## Limits - The default patterns select Markdown, text, reStructuredText, and AsciiDoc files. -- A run selects at most 10,000 files and reads at most 2 MiB per file unless configured otherwise. +- A run selects at most 10,000 files and reads at most 2 MiB per file by default. - Only UTF-8 content is accepted. -- Changed bytes produce a new exact snapshot Source; earlier snapshots remain readable for lineage. +- Changed content creates a new exact snapshot Source; earlier snapshots remain available. - A full scan removes missing paths from the next checkpoint but does not delete Sources or claim authoritative deletion. -- The Connector does not provide a change feed. Schedule repeated runs to observe later changes. +- The Connector does not provide a change feed; an external scheduler must run the worker again to observe changes. diff --git a/docs/en/rfcs/0000_source_definition_and_observation_model.md b/docs/en/rfcs/0000_source_definition_and_observation_model.md index c0a4ce338..8600f3c3f 100644 --- a/docs/en/rfcs/0000_source_definition_and_observation_model.md +++ b/docs/en/rfcs/0000_source_definition_and_observation_model.md @@ -24,9 +24,10 @@ A Definition may advertise named projection capabilities for consumers that do n Each projection has an independently versioned schema and deterministic meaning over one exact observation. A consumer selects a projection by capability name and version, never by inspecting a concrete Source class. -A Connector lifecycle binds provider acquisition to a Scope, submits definition-native observations, records -per-item outcomes, and advances an opaque checkpoint only after accepted observations are durable. Connector runs -distinguish complete discovery from incomplete discovery so that absence is not silently converted into deletion. +A Connector lifecycle binds provider acquisition to a Scope, resolves definition-native inputs in its worker, +submits materialized observations, records per-item outcomes, and advances an opaque checkpoint only after accepted +observations are durable. Connector runs distinguish complete discovery from incomplete discovery so that absence +is not silently converted into deletion. Materialization identifies the authority used to resolve an exact observation. A captured observation is resolved from the canonical value retained by PowerContext. A referenced observation is resolved from an immutable external @@ -36,9 +37,9 @@ referenced contract. `ContentSource` remains a simple captured-text Source. Its caller-stable identity and immutable-payload conflict rule make it useful for one-shot content capture, but it is not the general external integration model. -This RFC defines Source, projection, and Connector lifecycle semantics and conformance. It does not define a hosting -runtime, plugin discovery mechanism, storage schema, public transport operation, synchronization algorithm, -scheduler, credential transport, concrete Source family, or Connector implementation. +This RFC defines Source, projection, Connector lifecycle, and the remote ingestion boundary between a worker and the +PowerContext Server. It does not define plugin discovery, a scheduler, credential transport, a concrete Source +family, or a Connector implementation. # Motivation @@ -309,21 +310,45 @@ advanced, or the Connector being unavailable. ## Definition registration contract -A composed Runtime has one explicit Definition registry. Registration validates stable Definition name and version, -declared value and provenance schemas, identity rules, materialization support, and read behavior. Two incompatible -Definitions cannot claim the same `(source_type, definition_version)`. +Executable Definitions belong to the worker that resolves definition-native inputs, canonicalizes Source values, +and computes named projections. The Server does not import Connector or Definition packages and does not execute +their Python classes. -Registration also validates each advertised projection name and version, its declared output schema, and its -canonicalization contract. Two incompatible projections cannot claim the same capability key within one Definition -version. +Before submitting an observation, the worker registers an immutable declarative manifest containing the stable +Definition name and version, the canonical Source JSON Schema, every projection key and output JSON Schema, and a +fingerprint over the complete declaration. The fingerprint is SHA-256 over RFC 8785 canonical JSON. Registration is +idempotent for an identical manifest and rejects a different declaration for an existing `(source_type, +definition_version)`. -Registration is fixed for the Runtime lifetime. Catalog decoding, Source reads, and Artifact validation use the same -registry view. A persisted observation whose Definition is unavailable remains stored but cannot be interpreted or -advertised as readable. It is not decoded into a base Source with discarded fields. +The Server validates the manifest's schemas and any named projection it recognizes as a shared standard. A manifest +does not transfer executable identity rules, canonicalization code, read behavior, credentials, or provider +configuration. Those remain worker-owned. The registered manifest is sufficient for the Server to validate and +retain an opaque canonical observation without loading plugin code. Definition discovery and registration are separate. A package entry point or another discovery mechanism may report -available Definitions, but installation does not imply activation. This RFC does not select entry -points, a central settings format, pluggy, or a Connector marketplace. +available Definitions, but installation does not imply activation. This RFC does not select entry points, a central +settings format, pluggy, or a Connector marketplace. + +## Remote worker ingestion contract + +A Connector runs in an independent worker process. The worker owns provider access and all executable Definition +behavior. The Server owns durable Source history, Artifact consumption, and checkpoint comparison. Their data-plane +interaction consists of four generic operations: + +1. register an immutable Source Definition manifest; +2. read the opaque checkpoint for one Connector binding; +3. submit a worker-materialized Source observation with all declared projections; and +4. compare-and-swap the binding checkpoint from the value read at run start. + +The observation envelope carries the Definition name, version and fingerprint, canonical Source payload, and one +value for every projection declared by the manifest. The Server validates envelope identity, payload schema, +projection-key equality, projection schemas, and standard projection invariants before durable acceptance. Provider +names, storage services, paths, credentials, or other Connector-specific configuration do not appear in this API +unless a Definition deliberately includes them in its canonical Source schema. + +The Server returns a durable Source receipt before the worker may commit a checkpoint. The checkpoint operation uses +optimistic comparison so concurrent runs of the same binding cannot silently overwrite each other. Submission is +idempotent for an identical Source identity and payload; conflicting content for an accepted identity is rejected. ## Definition compatibility contract @@ -369,10 +394,10 @@ A Connector binding activates one Connector configuration for exactly one Scope. for checkpoint and provider-namespace continuity, but it does not own Sources and does not replace `scope_id` or `source_type`. Credentials are resolved by the hosting environment and do not become Source value or provenance. -A Connector run begins from an opaque binding checkpoint, submits zero or more definition-native observations, and -records an outcome for every submitted item. An accepted or idempotently replayed observation returns its exact -SourceRef. A rejected or failed item remains visible in the run outcome and cannot be hidden by advancing the -checkpoint past work that is not safely replayable. +A Connector run begins from an opaque binding checkpoint, resolves zero or more definition-native inputs inside the +worker, and submits their materialized observations. It records an outcome for every item. An accepted or +idempotently replayed observation returns its exact SourceRef. A rejected or failed item remains visible in the run +outcome and cannot be hidden by advancing the checkpoint past work that is not safely replayable. A run finishes as complete or incomplete. A complete snapshot may produce positive deletion evidence for previously known provider objects that are absent. An incomplete listing, timeout, permission failure, cancellation, or lost @@ -384,9 +409,8 @@ from an earlier checkpoint is valid because Source observation submission is ide health, retry, and run-status records are operational state rather than Source observations or Artifact evidence. Installation, discovery, activation, and execution are separate concerns. Installing a Connector package does not -activate a binding. This contract does not require that a Connector run inside the PowerContext Server; direct tools, -hosted workers, and external synchronization services can follow the same lifecycle and submit the same -definition-native observations. +activate a binding. A Connector package executes outside the PowerContext Server and uses the remote worker +ingestion contract; scheduling and process supervision belong to the deployment environment. ## Artifact evidence and cross-Scope delivery @@ -425,6 +449,10 @@ visibility, durable checkpoint ordering, complete-versus-incomplete run behavior Provider-specific behavior is established by its implementation evidence rather than generalized into the standard contract. +A remote worker path additionally verifies manifest fingerprint and conflict handling, rejection of unregistered or +schema-invalid observations, exact projection-set validation, durable receipt ordering, and stale checkpoint CAS +rejection across a Server restart. + # Drawbacks - Separating SourceKey, SourceRef, Source head, and Definition version introduces more concepts than one immutable @@ -435,7 +463,7 @@ contract. - Named projections and Connector lifecycle state add contracts that must evolve independently from Source values. - Referenced Sources are unavailable for providers that expose only current values, so some integrations must retain captured data. -- Explicit registration requires deployment coordination before a persisted custom Source can be read. +- Explicit registration requires deployment coordination before a custom Source observation can be accepted. # Rationale and alternatives @@ -505,8 +533,6 @@ Connector replacement change Source identity. active exact head and leave deletion entirely to Connector state? - Which projection names and schemas have enough implementation evidence to become shared standards rather than namespaced capabilities? -- Which Connector hosting and scheduling contracts, if any, must be standardized beyond the lifecycle semantics in - this RFC? # Future possibilities diff --git a/docs/zh/docs/how-to/ingest-text-files-with-opendal.md b/docs/zh/docs/how-to/ingest-text-files-with-opendal.md index d5440ae1f..74412ac70 100644 --- a/docs/zh/docs/how-to/ingest-text-files-with-opendal.md +++ b/docs/zh/docs/how-to/ingest-text-files-with-opendal.md @@ -1,87 +1,93 @@ --- title: 使用 OpenDAL 采集文本文件 -description: 通过 OpenDAL 存储后端把 UTF-8 文件捕获为类型化 Source。 +description: 用独立 OpenDAL Connector worker 把 UTF-8 文件捕获为类型化 Source。 --- # 使用 OpenDAL 采集文本文件 -使用 `OpenDALTextFileConnector` 从 OpenDAL 支持的存储后端捕获有界 UTF-8 文件。每个接受的文件都会成为不可变的 -`text-file-snapshot` Source，保留 path、namespace、content digest 和后端能够提供的 annotation。 +`powercontext-connector-opendal` 是独立于 PowerContext Server 部署的 worker。它拥有 OpenDAL credential、 +provider configuration、可执行 Source Definition 和文件读取逻辑。Server 只保存声明式 Definition manifest、 +已经物化的 Source observation、named projection 与 opaque checkpoint。 ## 前置条件 -OpenDAL 集成要求 Python 3.12 或更高版本。安装可选依赖： +该集成要求 Python 3.12 或更高版本。先启动 PowerContext Server，再从 checkout 安装 worker： ```bash -uv add "powercontext[opendal]" +uv tool install ./integrations/opendal ``` -为存储位置选择稳定的 `source_namespace`。它用于区分来自不同 authority、但 path 和内容相同的文件。不要把凭据写进 -namespace。 +选择稳定的 `source_namespace` 来区分不同 storage authority。不要把 credential 写进 namespace、Source payload 或 +checkpoint。Server 启用 authentication 时，通过 `POWERCONTEXT_TOKEN` 环境变量提供 bearer token。 -## 运行本地文件系统 binding +## 运行一个 binding -下面的 binding 扫描 `/absolute/path/to/project` 下的 `docs` 目录，并把 checkpoint 与捕获的 Source 持久化到同一个 -PowerContext 数据库： +下面的独立进程扫描 `/absolute/path/to/project/docs`。`binding_id` 标识 checkpoint continuity，`scope_id` 决定 +接受后的 Source 属于哪个 Scope： -```python -import asyncio - -from powercontext.builtin.connectors import OpenDALTextFileConnector -from powercontext.builtin.persistence.sqlite import SQLiteConfig -from powercontext.builtin.runtime import BuiltinConfig, open_builtin_contexts -from powercontext.sources import ConnectorBinding - - -async def main() -> None: - connector = OpenDALTextFileConnector.from_service( - "fs", - source_namespace="project-docs", - root="docs", - storage_options={"root": "/absolute/path/to/project"}, - ) - binding = ConnectorBinding( - scope_id="project:example", - binding_id="project-docs", - connector_name=connector.name, - connector_version=connector.version, - ) - config = BuiltinConfig( - database=SQLiteConfig(url="sqlite+aiosqlite:///powercontext.db"), - ) - - async with open_builtin_contexts(config) as contexts: - result = await contexts.run_connector(connector, binding) - print(result.model_dump_json(indent=2)) - - -asyncio.run(main()) +```bash +powercontext-connector-opendal \ + --base-url http://127.0.0.1:8765 \ + --scope-id project:example \ + --binding-id project-docs \ + --service fs \ + --storage-option root=/absolute/path/to/project \ + --root docs \ + --source-namespace project-docs ``` -访问远端存储时，换用对应的 OpenDAL service 及其 backend option。`storage_options` 只在运行期使用，不会复制进 Source -payload 或 checkpoint。 +访问远端存储时，替换 OpenDAL service 与对应的 `--storage-option KEY=VALUE`。这些 option 只存在于 worker 进程， +不会通过摄取 API 发送给 Server。 + +Worker 每次运行都会幂等注册 `text-file-snapshot` Definition manifest，读取 binding checkpoint，提交本轮变化的 +Source observation，并在所有 durable receipt 返回后 compare-and-swap checkpoint。可以由 cron、Kubernetes Job +或其他外部 scheduler 周期执行该命令。 -## 理解运行结果 +## 嵌入自定义 worker -每个 item outcome 有四种状态： +需要自定义进程监管或多 binding 调度时，可直接使用通用远程 lifecycle： + +```python +from powercontext.client import PowerContextClient, RemoteConnectorWorker +from powercontext.sources import ConnectorBinding, SourceDefinitionRegistry +from powercontext_connector_opendal import ( + TEXT_FILE_SNAPSHOT_SOURCE_DEFINITION, + OpenDALTextFileConnector, +) + +connector = OpenDALTextFileConnector.from_service( + "fs", + source_namespace="project-docs", + root="docs", + storage_options={"root": "/absolute/path/to/project"}, +) +binding = ConnectorBinding( + scope_id="project:example", + binding_id="project-docs", + connector_name=connector.name, + connector_version=connector.version, +) +registry = SourceDefinitionRegistry((TEXT_FILE_SNAPSHOT_SOURCE_DEFINITION,)) + +async with PowerContextClient("http://127.0.0.1:8765") as client: + result = await RemoteConnectorWorker(client=client, registry=registry).run(connector, binding) +``` -- `accepted`：Source 已持久化； -- `replayed`：sink 识别到已接受的 Source； -- `rejected`：provider item 无法满足 Source Definition，例如不是有效 UTF-8； -- `failed`：无法安全读取或存储该 item。 +## 运行语义 -只有所有选中 item 都被接受且本轮完整结束后，checkpoint 才会前移。出现 rejected 或 failed item 时保留旧 checkpoint， -下一轮会安全重试扫描。digest 与已提交 checkpoint 相同的文件会被跳过。 +每个 item outcome 是 `accepted`、`replayed`、`rejected` 或 `failed`。只有本轮完整结束并且没有 rejected 或 failed +item 时 checkpoint 才会前移。否则保留旧 checkpoint，下一轮从同一位置安全重试。与已提交 checkpoint 中 digest +相同的文件会被跳过。 -接受的 Source 会进入同一 scope 的 Source journal。Runtime 配置了 Memory candidate pipeline 后，常规 source-window flush -或调度任务可以通过共享的 `powercontext.builtin.text-evidence` projection 消费这些 Source。Connector 完成采集并不直接创建 -Memory。 +接受的 Source 进入目标 Scope 的 Source journal。Worker 同时计算标准 `powercontext.text-evidence` projection， +因此不了解 `text-file-snapshot` native schema 的 Memory consumer 仍可消费文本。Connector run 不直接创建 Memory； +Memory 仍由常规 source-window flush 或调度任务生成。 -## 当前限制 +## 限制 -- 默认 pattern 选择 Markdown、纯文本、reStructuredText 和 AsciiDoc 文件。 -- 除非显式调整，每轮最多选择 10,000 个文件，每个文件最多读取 2 MiB。 +- 默认选择 Markdown、纯文本、reStructuredText 与 AsciiDoc 文件。 +- 默认每轮最多选择 10,000 个文件，每个文件最多读取 2 MiB。 - 只接受 UTF-8 内容。 -- 文件内容变化会生成新的精确 snapshot Source；旧 snapshot 仍然可读，以保留 lineage。 -- 全量扫描会从下一 checkpoint 移除已消失的 path，但不会删除 Source，也不声明 authoritative deletion。 -- Connector 不提供 change feed。需要通过周期运行观察后续变化。 +- 内容变化会生成新的精确 snapshot Source，旧 snapshot 继续保留。 +- 全量扫描会从下一 checkpoint 移除已消失 path，但不会删除 Source，也不声明 authoritative deletion。 +- Connector 不提供 change feed；后续变化依赖外部 scheduler 再次运行 worker。 diff --git a/docs/zh/rfcs/0000_source_definition_and_observation_model.md b/docs/zh/rfcs/0000_source_definition_and_observation_model.md index 0d248883c..7336d610a 100644 --- a/docs/zh/rfcs/0000_source_definition_and_observation_model.md +++ b/docs/zh/rfcs/0000_source_definition_and_observation_model.md @@ -22,9 +22,9 @@ Definition 可以为无法理解 native value 的 consumer 声明 named projecti 版本的 schema，并对一个精确 observation 具有确定语义。Consumer 按 capability name 与 version 选择 projection， 而不是检查具体 Source class。 -Connector lifecycle 将 provider acquisition 绑定到 Scope，提交 definition-native observation，记录 per-item -outcome，并且只在接受的 observation 已持久化后推进 opaque checkpoint。Connector run 区分 complete discovery -与 incomplete discovery，避免把缺失对象静默转换为删除。 +Connector lifecycle 将 provider acquisition 绑定到 Scope，在 worker 内解析 definition-native input，提交 +materialized observation，记录 per-item outcome，并且只在接受的 observation 已持久化后推进 opaque checkpoint。 +Connector run 区分 complete discovery 与 incomplete discovery，避免把缺失对象静默转换为删除。 Materialization 表达解析某个精确观察时所依赖的权威来源。Captured observation 从 PowerContext 保留的 canonical value 解析；referenced observation 从外部不可变 revision 解析。仅有外部 locator、修改时间、 @@ -33,9 +33,8 @@ ETag 或 provider 当前值读取，并不能满足 referenced 契约。 `ContentSource` 继续作为简单的 captured-text Source。调用方提供稳定身份，加上 immutable-payload 冲突规则， 适合一次性内容捕获，但它不是通用的外部集成模型。 -本 RFC 定义 Source、projection 与 Connector lifecycle 的语义和 conformance，不定义 hosting runtime、插件发现 -机制、存储 schema、公开 transport operation、同步算法、scheduler、credential transport、具体 Source family -或 Connector 实现。 +本 RFC 定义 Source、projection、Connector lifecycle，以及 worker 与 PowerContext Server 之间的远程摄取边界。 +它不定义插件发现、scheduler、credential transport、具体 Source family 或 Connector 实现。 # Motivation @@ -290,21 +289,42 @@ digest。无法解析精确 observation，不等同于 logical Source 已删除 ## Definition registration contract -组合后的 Runtime 拥有一个显式 Definition registry。注册时验证稳定的 Definition name 与 version、声明的 value -与 provenance schemas、identity rules、materialization support 和 read behavior。两个不兼容 Definition 不能 -声明同一个 `(source_type, definition_version)`。 +Executable Definition 属于 worker。Worker 用它解析 definition-native input、canonicalize Source value，并计算 +named projection。Server 不导入 Connector 或 Definition package，也不执行其中的 Python class。 -注册还会验证每个声明的 projection name 与 version、output schema 和 canonicalization contract。两个不兼容的 -projection 不能在同一个 Definition version 内声明相同 capability key。 +提交 observation 前，worker 注册不可变的声明式 manifest。Manifest 包含稳定的 Definition name 与 version、 +canonical Source JSON Schema、每个 projection key 与 output JSON Schema，以及覆盖完整声明的 fingerprint。 +Fingerprint 是 RFC 8785 canonical JSON 的 SHA-256。相同 manifest 的注册是幂等的；同一个 +`(source_type, definition_version)` 对应不同声明时必须拒绝。 -Registry 在 Runtime 生命周期内固定。Catalog decoding、Source reads 与 Artifact validation 使用同一个 registry -view。Definition 不可用时，已经持久化的 observation 仍保留，但不能被解释或宣称为 readable；不能把它解码成 -丢失字段的 base Source。 +Server 验证 manifest schema，以及其识别为 shared standard 的 named projection。Manifest 不传输可执行的 identity +rule、canonicalization code、read behavior、credential 或 provider configuration；这些仍由 worker 持有。 +注册后的 manifest 足以让 Server 在不加载 plugin code 的情况下验证并保存 opaque canonical observation。 Definition discovery 与 registration 相互独立。Package entry point 或其他 discovery mechanism 可以报告 可用 Definition，但安装不意味着激活。本 RFC 不选择 entry points、central settings format、pluggy 或 Connector marketplace。 +## Remote worker ingestion contract + +Connector 在独立 worker 进程中运行。Worker 拥有 provider access 与所有 executable Definition behavior；Server +拥有 durable Source history、Artifact consumption 与 checkpoint comparison。双方的数据面交互只有四个通用操作： + +1. 注册不可变的 Source Definition manifest； +2. 读取一个 Connector binding 的 opaque checkpoint； +3. 提交 worker 已物化的 Source observation 及其全部声明 projection； +4. 从 run 开始时读到的值 compare-and-swap binding checkpoint。 + +Observation envelope 携带 Definition name、version 与 fingerprint、canonical Source payload，以及 manifest 声明的 +每个 projection value。Server 在 durable acceptance 前验证 envelope identity、payload schema、projection key +集合相等、projection schema 与标准 projection invariant。Provider name、storage service、path、credential 或其他 +Connector-specific configuration 不出现在该 API 中；只有 Definition 刻意将其声明为 canonical Source schema 的 +一部分时才例外。 + +Server 必须先返回 durable Source receipt，worker 才能提交 checkpoint。Checkpoint operation 使用 optimistic +comparison，防止同一 binding 的并发 run 静默覆盖。相同 Source identity 与 payload 的提交是幂等的；已接受 identity +对应不同内容时必须拒绝。 + ## Definition compatibility contract Definition name 在兼容 schema 演进中保持稳定。每个 persisted observation 记录验证和 canonicalize 它时使用的 @@ -348,9 +368,9 @@ Connector binding 为一个 Scope 激活一份 Connector configuration。Binding namespace continuity 的稳定 identity，但不拥有 Source，也不替代 `scope_id` 或 `source_type`。Credential 由 hosting environment 解析，不会成为 Source value 或 provenance。 -Connector run 从 opaque binding checkpoint 开始，提交零个或多个 definition-native observation，并记录每个 -submitted item 的 outcome。Accepted 或 idempotently replayed observation 返回精确 SourceRef。Rejected 或 failed -item 会保留在 run outcome 中；如果尚不能安全重放，checkpoint 不能越过这些工作。 +Connector run 从 opaque binding checkpoint 开始，在 worker 内解析零个或多个 definition-native input，再提交其 +materialized observation，并记录每个 item 的 outcome。Accepted 或 idempotently replayed observation 返回精确 +SourceRef。Rejected 或 failed item 会保留在 run outcome 中；如果尚不能安全重放，checkpoint 不能越过这些工作。 Run 以 complete 或 incomplete 结束。Complete snapshot 可以为之前已知但本次缺失的 provider object 产生 positive deletion evidence。Incomplete listing、timeout、permission failure、cancellation 或 lost connection 不会产生 @@ -361,9 +381,8 @@ Completed checkpoint 只有在 accepted observation 与 deletion evidence 均已 observation submission 具有幂等性，从更早 checkpoint 重试是合法行为。Connector checkpoint、health、retry 与 run-status record 是 operational state，而不是 Source observation 或 Artifact evidence。 -Installation、discovery、activation 与 execution 相互独立。安装 Connector package 不会激活 binding。本契约不 -要求 Connector 运行在 PowerContext Server 内；direct tool、hosted worker 与 external synchronization service -都可以遵循相同 lifecycle，提交相同 definition-native observation。 +Installation、discovery、activation 与 execution 相互独立。安装 Connector package 不会激活 binding。Connector +package 在 PowerContext Server 之外执行，并使用 remote worker ingestion contract；调度与进程监管属于部署环境。 ## Artifact evidence and cross-Scope delivery @@ -400,6 +419,9 @@ Connector capability 只有在 conformance 验证 checkpoint replay、per-item o ordering、complete-versus-incomplete run behavior，以及其声明的 deletion evidence 后才能被声明。Provider-specific behavior 由对应实现证据确定，不会被直接推广为标准契约。 +Remote worker path 还必须验证 manifest fingerprint 与 conflict handling、拒绝未注册或 schema-invalid observation、 +projection set 精确校验、durable receipt ordering，以及跨 Server restart 的 stale checkpoint CAS rejection。 + # Drawbacks - 分离 SourceKey、SourceRef、Source head 与 Definition version，比一个不可变的 `(source_type, source_id)` pair @@ -408,7 +430,7 @@ behavior 由对应实现证据确定，不会被直接推广为标准契约。 - Definition author 必须声明 canonicalization、provenance 与 compatibility，而不能依赖任意 metadata。 - Named projection 与 Connector lifecycle state 增加了需要独立于 Source value 演进的契约。 - 只暴露当前值的 provider 无法使用 Referenced Source，因此部分集成必须保留 captured data。 -- Persisted custom Source 可读之前，显式 registration 需要部署协调。 +- Custom Source observation 被接受之前，显式 registration 需要部署协调。 # Rationale and alternatives @@ -473,7 +495,6 @@ replacement 改变 Source identity。 - Source head deletion 应是通用 catalog state，还是标准契约只暴露 active exact head，并把 deletion 完全留给 Connector state？ - 哪些 projection name 与 schema 已有足够实现证据，可以成为 shared standard 而不是 namespaced capability？ -- 除本 RFC 的 lifecycle semantics 外，是否还需要标准化 Connector hosting 与 scheduling contract？ # Future possibilities diff --git a/integrations/dsh/plugins/powercontext/src/operations.generated.ts b/integrations/dsh/plugins/powercontext/src/operations.generated.ts index 092c7108a..06178ba3b 100644 --- a/integrations/dsh/plugins/powercontext/src/operations.generated.ts +++ b/integrations/dsh/plugins/powercontext/src/operations.generated.ts @@ -21,6 +21,10 @@ export const OPERATIONS = { get_readiness: { method: 'GET', path: '/health/ready', location: null, scope: false }, get_capabilities: { method: 'GET', path: '/v1/capabilities', location: null, scope: false }, capture_content_source: { method: 'POST', path: '/v1/sources/content', location: "body", scope: true }, + register_source_definition: { method: 'POST', path: '/v1/source-definitions/register', location: "body", scope: false }, + get_connector_checkpoint: { method: 'POST', path: '/v1/connector-checkpoints/get', location: "body", scope: false }, + submit_source_observation: { method: 'POST', path: '/v1/source-observations', location: "body", scope: false }, + commit_connector_checkpoint: { method: 'POST', path: '/v1/connector-checkpoints/commit', location: "body", scope: false }, prepare_context: { method: 'POST', path: '/v1/context/prepare', location: "body", scope: true }, create_work_contract: { method: 'POST', path: '/v1/work/contracts/create', location: "body", scope: true }, handoff_current_work: { method: 'POST', path: '/v1/work/handoffs/prepare-current', location: "body", scope: true }, diff --git a/integrations/opencode/plugins/powercontext/src/operations.generated.ts b/integrations/opencode/plugins/powercontext/src/operations.generated.ts index 092c7108a..06178ba3b 100644 --- a/integrations/opencode/plugins/powercontext/src/operations.generated.ts +++ b/integrations/opencode/plugins/powercontext/src/operations.generated.ts @@ -21,6 +21,10 @@ export const OPERATIONS = { get_readiness: { method: 'GET', path: '/health/ready', location: null, scope: false }, get_capabilities: { method: 'GET', path: '/v1/capabilities', location: null, scope: false }, capture_content_source: { method: 'POST', path: '/v1/sources/content', location: "body", scope: true }, + register_source_definition: { method: 'POST', path: '/v1/source-definitions/register', location: "body", scope: false }, + get_connector_checkpoint: { method: 'POST', path: '/v1/connector-checkpoints/get', location: "body", scope: false }, + submit_source_observation: { method: 'POST', path: '/v1/source-observations', location: "body", scope: false }, + commit_connector_checkpoint: { method: 'POST', path: '/v1/connector-checkpoints/commit', location: "body", scope: false }, prepare_context: { method: 'POST', path: '/v1/context/prepare', location: "body", scope: true }, create_work_contract: { method: 'POST', path: '/v1/work/contracts/create', location: "body", scope: true }, handoff_current_work: { method: 'POST', path: '/v1/work/handoffs/prepare-current', location: "body", scope: true }, diff --git a/integrations/opendal/README.md b/integrations/opendal/README.md new file mode 100644 index 000000000..e23e26ab3 --- /dev/null +++ b/integrations/opendal/README.md @@ -0,0 +1,27 @@ +# OpenDAL Connector worker + +`powercontext-connector-opendal` is an independently deployed Connector worker. It owns the executable text-file +Source Definition and uses OpenDAL through `opendalfs` to acquire files. PowerContext Server only receives the +Definition manifest, projected Source observations, and opaque checkpoint comparisons. + +Install from a checkout: + +```bash +uv tool install ./integrations/opendal +``` + +Run one bounded scan against a filesystem backend: + +```bash +powercontext-connector-opendal \ + --base-url http://127.0.0.1:8765 \ + --scope-id project-a \ + --binding-id workspace-documents \ + --service fs \ + --storage-option root=/path/to/workspace \ + --source-namespace workspace-a +``` + +The process registers its immutable Definition manifest before each run. It advances the binding checkpoint only +after every accepted Source observation has a durable Server receipt and the scan completes without rejected or +failed items. Set `POWERCONTEXT_TOKEN` when the Server requires bearer authentication. diff --git a/integrations/opendal/pyproject.toml b/integrations/opendal/pyproject.toml new file mode 100644 index 000000000..5d08e8bb7 --- /dev/null +++ b/integrations/opendal/pyproject.toml @@ -0,0 +1,29 @@ +# Copyright (c) 2026 OceanBase. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 + +[project] +name = "powercontext-connector-opendal" +version = "0.0.1" +description = "OpenDAL file connector worker for PowerContext." +readme = "README.md" +license = "Apache-2.0" +requires-python = ">=3.12,<4.0" +dependencies = [ + "opendalfs>=0.1,<0.2", + "powercontext[client]>=0.0.3,<1", +] + +[project.scripts] +powercontext-connector-opendal = "powercontext_connector_opendal.cli:main" + +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[tool.hatch.build.targets.wheel] +packages = ["src/powercontext_connector_opendal"] diff --git a/src/powercontext/builtin/connectors/__init__.py b/integrations/opendal/src/powercontext_connector_opendal/__init__.py similarity index 54% rename from src/powercontext/builtin/connectors/__init__.py rename to integrations/opendal/src/powercontext_connector_opendal/__init__.py index 7bd21584d..634863194 100644 --- a/src/powercontext/builtin/connectors/__init__.py +++ b/integrations/opendal/src/powercontext_connector_opendal/__init__.py @@ -12,16 +12,32 @@ # See the License for the specific language governing permissions and # limitations under the License. -"""Supported built-in Connector implementations.""" +"""OpenDAL Connector worker for PowerContext.""" -from powercontext.builtin.connectors.opendal import ( +from powercontext_connector_opendal.connector import ( OPENDAL_TEXT_FILE_CONNECTOR_NAME, OpenDALTextFileCheckpoint, OpenDALTextFileConnector, ) +from powercontext_connector_opendal.source import ( + TEXT_FILE_SNAPSHOT_SOURCE_ADAPTER, + TEXT_FILE_SNAPSHOT_SOURCE_DEFINITION, + TEXT_FILE_SNAPSHOT_SOURCE_NAME, + TextFileEvidenceProjection, + TextFileSnapshotCapture, + TextFileSnapshotSource, + TextFileSnapshotSourceAdapter, +) __all__ = [ "OPENDAL_TEXT_FILE_CONNECTOR_NAME", + "TEXT_FILE_SNAPSHOT_SOURCE_ADAPTER", + "TEXT_FILE_SNAPSHOT_SOURCE_DEFINITION", + "TEXT_FILE_SNAPSHOT_SOURCE_NAME", "OpenDALTextFileCheckpoint", "OpenDALTextFileConnector", + "TextFileEvidenceProjection", + "TextFileSnapshotCapture", + "TextFileSnapshotSource", + "TextFileSnapshotSourceAdapter", ] diff --git a/integrations/opendal/src/powercontext_connector_opendal/cli.py b/integrations/opendal/src/powercontext_connector_opendal/cli.py new file mode 100644 index 000000000..ca5f1bf94 --- /dev/null +++ b/integrations/opendal/src/powercontext_connector_opendal/cli.py @@ -0,0 +1,95 @@ +# Copyright (c) 2026 OceanBase. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Command-line entry point for one independently scheduled Connector run.""" + +from __future__ import annotations + +import argparse +import asyncio +import os +from collections.abc import Sequence + +from powercontext.client import PowerContextClient, RemoteConnectorWorker +from powercontext.sources import ConnectorBinding, ConnectorRunStatus, SourceDefinitionRegistry +from powercontext_connector_opendal.connector import OPENDAL_TEXT_FILE_CONNECTOR_NAME, OpenDALTextFileConnector +from powercontext_connector_opendal.source import TEXT_FILE_SNAPSHOT_SOURCE_DEFINITION + + +def main() -> None: + """Run one scan and return a process status suitable for an external scheduler.""" + + raise SystemExit(asyncio.run(_run(_parser().parse_args()))) + + +async def _run(args: argparse.Namespace) -> int: + options = _storage_options(args.storage_option) + if args.pattern: + connector = OpenDALTextFileConnector.from_service( + args.service, + source_namespace=args.source_namespace, + root=args.root, + storage_options=options, + patterns=tuple(args.pattern), + max_files=args.max_files, + max_file_size=args.max_file_size, + ) + else: + connector = OpenDALTextFileConnector.from_service( + args.service, + source_namespace=args.source_namespace, + root=args.root, + storage_options=options, + max_files=args.max_files, + max_file_size=args.max_file_size, + ) + binding = ConnectorBinding( + scope_id=args.scope_id, + binding_id=args.binding_id, + connector_name=OPENDAL_TEXT_FILE_CONNECTOR_NAME, + connector_version=connector.version, + ) + registry = SourceDefinitionRegistry((TEXT_FILE_SNAPSHOT_SOURCE_DEFINITION,)) + async with PowerContextClient(args.base_url, token=os.environ.get("POWERCONTEXT_TOKEN")) as client: + result = await RemoteConnectorWorker(client=client, registry=registry).run(connector, binding) + return 0 if result.status is ConnectorRunStatus.COMPLETE else 1 + + +def _storage_options(values: Sequence[str]) -> dict[str, str]: + options: dict[str, str] = {} + for value in values: + key, separator, option = value.partition("=") + if not separator or not key or key.strip() != key: + raise ValueError("storage options must use KEY=VALUE") # noqa: TRY003 + options[key] = option + return options + + +def _parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--base-url", required=True) + parser.add_argument("--scope-id", required=True) + parser.add_argument("--binding-id", required=True) + parser.add_argument("--service", required=True) + parser.add_argument("--source-namespace", required=True) + parser.add_argument("--root", default="") + parser.add_argument("--storage-option", action="append", default=[]) + parser.add_argument("--pattern", action="append") + parser.add_argument("--max-files", type=int, default=10_000) + parser.add_argument("--max-file-size", type=int, default=2 * 1024 * 1024) + return parser + + +if __name__ == "__main__": + main() diff --git a/src/powercontext/builtin/connectors/opendal.py b/integrations/opendal/src/powercontext_connector_opendal/connector.py similarity index 99% rename from src/powercontext/builtin/connectors/opendal.py rename to integrations/opendal/src/powercontext_connector_opendal/connector.py index 7d0858dbd..4a9438c80 100644 --- a/src/powercontext/builtin/connectors/opendal.py +++ b/integrations/opendal/src/powercontext_connector_opendal/connector.py @@ -27,10 +27,6 @@ from pydantic import BaseModel, ConfigDict, Field, JsonValue, ValidationError, field_validator -from powercontext.builtin.sources import ( - TEXT_FILE_SNAPSHOT_SOURCE_NAME, - TextFileSnapshotCapture, -) from powercontext.errors import InvalidConnectorRunError from powercontext.sources import ( ConnectorCapability, @@ -38,6 +34,10 @@ ConnectorRunSession, ConnectorRunStatus, ) +from powercontext_connector_opendal.source import ( + TEXT_FILE_SNAPSHOT_SOURCE_NAME, + TextFileSnapshotCapture, +) OPENDAL_TEXT_FILE_CONNECTOR_NAME = "opendal-text-files" _DEFAULT_PATTERNS = ("**/*.md", "**/*.markdown", "**/*.txt", "**/*.rst", "**/*.adoc") @@ -127,7 +127,7 @@ def from_service( from opendalfs import OpendalFileSystem except ImportError as error: raise ImportError( # noqa: TRY003 - "OpenDALTextFileConnector.from_service requires powercontext[opendal] on Python 3.12+" + "OpenDALTextFileConnector.from_service requires powercontext-connector-opendal on Python 3.12+" ) from error backend_options: dict[str, Any] = dict(storage_options or {}) filesystem = OpendalFileSystem( diff --git a/src/powercontext/builtin/sources/text_file.py b/integrations/opendal/src/powercontext_connector_opendal/source.py similarity index 96% rename from src/powercontext/builtin/sources/text_file.py rename to integrations/opendal/src/powercontext_connector_opendal/source.py index 659254c05..99f2ddecc 100644 --- a/src/powercontext/builtin/sources/text_file.py +++ b/integrations/opendal/src/powercontext_connector_opendal/source.py @@ -12,7 +12,7 @@ # See the License for the specific language governing permissions and # limitations under the License. -"""Typed captured text-file snapshots for filesystem Connectors.""" +"""Typed captured text-file snapshots for the OpenDAL Connector.""" from __future__ import annotations @@ -23,8 +23,13 @@ from pydantic import BaseModel, JsonValue, field_validator -from powercontext.builtin.sources.projections import TEXT_EVIDENCE_PROJECTION_KEY, TextEvidence -from powercontext.sources import AdapterSourceDefinition, Source, SourceMaterialization +from powercontext.sources import ( + TEXT_EVIDENCE_PROJECTION_KEY, + AdapterSourceDefinition, + Source, + SourceMaterialization, + TextEvidence, +) TEXT_FILE_SNAPSHOT_SOURCE_NAME = "text-file-snapshot" diff --git a/integrations/pi/plugins/powercontext/src/operations.generated.ts b/integrations/pi/plugins/powercontext/src/operations.generated.ts index 092c7108a..06178ba3b 100644 --- a/integrations/pi/plugins/powercontext/src/operations.generated.ts +++ b/integrations/pi/plugins/powercontext/src/operations.generated.ts @@ -21,6 +21,10 @@ export const OPERATIONS = { get_readiness: { method: 'GET', path: '/health/ready', location: null, scope: false }, get_capabilities: { method: 'GET', path: '/v1/capabilities', location: null, scope: false }, capture_content_source: { method: 'POST', path: '/v1/sources/content', location: "body", scope: true }, + register_source_definition: { method: 'POST', path: '/v1/source-definitions/register', location: "body", scope: false }, + get_connector_checkpoint: { method: 'POST', path: '/v1/connector-checkpoints/get', location: "body", scope: false }, + submit_source_observation: { method: 'POST', path: '/v1/source-observations', location: "body", scope: false }, + commit_connector_checkpoint: { method: 'POST', path: '/v1/connector-checkpoints/commit', location: "body", scope: false }, prepare_context: { method: 'POST', path: '/v1/context/prepare', location: "body", scope: true }, create_work_contract: { method: 'POST', path: '/v1/work/contracts/create', location: "body", scope: true }, handoff_current_work: { method: 'POST', path: '/v1/work/handoffs/prepare-current', location: "body", scope: true }, diff --git a/openapi/powercontext.yaml b/openapi/powercontext.yaml index 2c8681f99..bed6472a9 100644 --- a/openapi/powercontext.yaml +++ b/openapi/powercontext.yaml @@ -111,6 +111,107 @@ paths: $ref: "#/components/responses/Unavailable" "500": $ref: "#/components/responses/InternalError" + /v1/source-definitions/register: + post: + tags: [source-ingestion] + summary: Register a worker-owned Source Definition manifest + description: Registers an immutable declarative manifest without loading worker plugin code. + operationId: register_source_definition + requestBody: + required: true + content: + application/json: + schema: + $ref: "#/components/schemas/RegisterSourceDefinitionRequest" + responses: + "200": + description: The exact manifest is registered or was already registered identically. + content: + application/json: + schema: + $ref: "#/components/schemas/SourceDefinitionManifest" + "409": + $ref: "#/components/responses/Conflict" + "401": + $ref: "#/components/responses/Unauthorized" + "422": + $ref: "#/components/responses/InvalidRequest" + /v1/connector-checkpoints/get: + post: + tags: [source-ingestion] + summary: Read a Connector binding checkpoint + operationId: get_connector_checkpoint + requestBody: + required: true + content: + application/json: + schema: + $ref: "#/components/schemas/GetConnectorCheckpointRequest" + responses: + "200": + description: The current opaque checkpoint, including a normal null initial value. + content: + application/json: + schema: + $ref: "#/components/schemas/ConnectorCheckpointState" + "401": + $ref: "#/components/responses/Unauthorized" + "409": + $ref: "#/components/responses/Conflict" + "422": + $ref: "#/components/responses/InvalidRequest" + /v1/source-observations: + post: + tags: [source-ingestion] + summary: Submit a worker-materialized Source observation + description: Validates the observation against its registered manifest and durably appends it before receipt. + operationId: submit_source_observation + requestBody: + required: true + content: + application/json: + schema: + $ref: "#/components/schemas/SubmitSourceObservationRequest" + responses: + "202": + description: The observation is durably accepted and can be referenced exactly. + content: + application/json: + schema: + $ref: "#/components/schemas/SourceObservationReceipt" + "401": + $ref: "#/components/responses/Unauthorized" + "409": + $ref: "#/components/responses/Conflict" + "404": + $ref: "#/components/responses/NotFound" + "422": + $ref: "#/components/responses/InvalidRequest" + /v1/connector-checkpoints/commit: + post: + tags: [source-ingestion] + summary: Commit a Connector binding checkpoint + description: Replaces the checkpoint only when its expected starting value still matches. + operationId: commit_connector_checkpoint + requestBody: + required: true + content: + application/json: + schema: + $ref: "#/components/schemas/CommitConnectorCheckpointRequest" + responses: + "200": + description: The new opaque checkpoint is durable. + content: + application/json: + schema: + $ref: "#/components/schemas/ConnectorCheckpointState" + "401": + $ref: "#/components/responses/Unauthorized" + "409": + $ref: "#/components/responses/Conflict" + "422": + $ref: "#/components/responses/InvalidRequest" /v1/context/prepare: post: tags: [context] @@ -2663,6 +2764,178 @@ components: position: type: integer minimum: 1 + SourceProjectionKey: + type: object + additionalProperties: false + required: [name, version] + properties: + name: + type: string + minLength: 1 + maxLength: 128 + pattern: '.*\S.*' + version: + type: string + minLength: 1 + maxLength: 128 + pattern: '.*\S.*' + SourceProjectionManifest: + type: object + additionalProperties: false + required: [key, schema] + properties: + key: + $ref: "#/components/schemas/SourceProjectionKey" + schema: + type: object + additionalProperties: true + SourceDefinitionManifest: + type: object + additionalProperties: false + required: [name, version, fingerprint, source_schema, projections] + properties: + name: + type: string + minLength: 1 + maxLength: 128 + pattern: '.*\S.*' + version: + type: string + minLength: 1 + maxLength: 128 + pattern: '.*\S.*' + fingerprint: + type: string + pattern: '^sha256:[0-9a-f]{64}$' + source_schema: + type: object + additionalProperties: true + projections: + type: array + maxItems: 16 + items: + $ref: "#/components/schemas/SourceProjectionManifest" + RegisterSourceDefinitionRequest: + type: object + additionalProperties: false + required: [manifest] + properties: + manifest: + $ref: "#/components/schemas/SourceDefinitionManifest" + ConnectorBinding: + type: object + additionalProperties: false + required: [scope_id, binding_id, connector_name, connector_version] + properties: + scope_id: + type: string + minLength: 1 + maxLength: 256 + pattern: '.*\S.*' + binding_id: + type: string + minLength: 1 + maxLength: 256 + pattern: '.*\S.*' + connector_name: + type: string + minLength: 1 + maxLength: 128 + pattern: '.*\S.*' + connector_version: + type: string + minLength: 1 + maxLength: 128 + pattern: '.*\S.*' + GetConnectorCheckpointRequest: + type: object + additionalProperties: false + required: [binding] + properties: + binding: + $ref: "#/components/schemas/ConnectorBinding" + ConnectorCheckpointState: + type: object + additionalProperties: false + required: [binding, checkpoint] + properties: + binding: + $ref: "#/components/schemas/ConnectorBinding" + checkpoint: + nullable: true + SourceProjectionValue: + type: object + additionalProperties: false + required: [key, value] + properties: + key: + $ref: "#/components/schemas/SourceProjectionKey" + value: {} + ProjectedSource: + type: object + additionalProperties: false + required: + [name, definition_version, materialization, source_type, definition_fingerprint, payload, projections] + properties: + name: + type: string + minLength: 1 + maxLength: 256 + definition_version: + type: string + minLength: 1 + maxLength: 128 + materialization: + type: string + enum: [captured, referenced] + description: + type: string + nullable: true + source_type: + type: string + minLength: 1 + maxLength: 128 + definition_fingerprint: + type: string + pattern: '^sha256:[0-9a-f]{64}$' + payload: + type: object + additionalProperties: true + projections: + type: array + maxItems: 16 + items: + $ref: "#/components/schemas/SourceProjectionValue" + SubmitSourceObservationRequest: + type: object + additionalProperties: false + required: [binding, source] + properties: + binding: + $ref: "#/components/schemas/ConnectorBinding" + source: + $ref: "#/components/schemas/ProjectedSource" + SourceObservationReceipt: + type: object + additionalProperties: false + required: [source, position] + properties: + source: + $ref: "#/components/schemas/SourceReference" + position: + type: integer + minimum: 1 + CommitConnectorCheckpointRequest: + type: object + additionalProperties: false + required: [binding, expected, checkpoint] + properties: + binding: + $ref: "#/components/schemas/ConnectorBinding" + expected: + nullable: true + checkpoint: + nullable: true CommitHandoffRequest: type: object additionalProperties: false diff --git a/pyproject.toml b/pyproject.toml index 64c45a1ca..ef3dccdf5 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -40,6 +40,7 @@ classifiers = [ builtin = [ "aiosqlite>=0.22,<1", "apscheduler>=3.11,<4", + "jsonschema>=4.23,<5", "pydantic-ai-slim[anthropic,openai]>=2.27.1,<3", "pydantic-settings>=2.7,<3", "pyobvector>=0.2.28,<0.3", @@ -55,9 +56,6 @@ client = [ "opentelemetry-api>=1.30,<2", "pydantic-settings>=2.7,<3", ] -opendal = [ - "opendalfs>=0.1,<0.2; python_version >= '3.12'", -] server = [ "fastapi>=0.115,<1", "fastmcp>=3.4,<4", @@ -136,6 +134,7 @@ extra-paths = [ "./integrations/workbuddy/plugins/powercontext", "./integrations/workbuddy/plugins/powercontext/hooks", "./integrations/pydantic-ai/src", + "./integrations/opendal/src", ] [tool.ty.src] @@ -166,7 +165,7 @@ missing-override-decorator = "ignore" [tool.pytest.ini_options] testpaths = ["tests"] -pythonpath = ["integrations/pydantic-ai/src"] +pythonpath = ["integrations/pydantic-ai/src", "integrations/opendal/src"] markers = [ "real_e2e: uses real Codex, external model providers, and the configured database", ] diff --git a/src/powercontext/__init__.py b/src/powercontext/__init__.py index 2c68383b2..5a4abf7ff 100644 --- a/src/powercontext/__init__.py +++ b/src/powercontext/__init__.py @@ -34,6 +34,7 @@ InvalidSourceAdapterError, InvalidSourceDefinitionError, InvalidSourceEntryError, + InvalidSourceObservationError, InvalidSourceProjectionError, InvalidSourceReferenceError, InvalidSourceResultError, @@ -73,6 +74,7 @@ SourceProjectionKey, SourceRef, SourceStore, + validate_connector, ) from powercontext.triggers import PolicyTransition, Trigger @@ -109,6 +111,7 @@ "InvalidSourceAdapterError", "InvalidSourceDefinitionError", "InvalidSourceEntryError", + "InvalidSourceObservationError", "InvalidSourceProjectionError", "InvalidSourceReferenceError", "InvalidSourceResultError", @@ -135,4 +138,5 @@ "SourceStore", "Sources", "Trigger", + "validate_connector", ] diff --git a/src/powercontext/builtin/persistence/__init__.py b/src/powercontext/builtin/persistence/__init__.py index 9267564f4..703f204f4 100644 --- a/src/powercontext/builtin/persistence/__init__.py +++ b/src/powercontext/builtin/persistence/__init__.py @@ -34,6 +34,10 @@ StoredPayloadConflictError, ) from powercontext.builtin.persistence.external_skills import ExternalSkillRepository +from powercontext.builtin.persistence.source_definitions import ( + SourceDefinitionManifestRepository, + StoredSourceDefinitionManifest, +) from powercontext.builtin.persistence.statistics import ( StatisticsRepository, StoredInventoryCounts, @@ -56,10 +60,12 @@ "RelationalConnectorCheckpointStore", "RepositoryError", "RepositoryNotFoundError", + "SourceDefinitionManifestRepository", "StatisticsRepository", "StoredConnectorCheckpoint", "StoredInventoryCounts", "StoredModelUsage", "StoredPayloadConflictError", "StoredRecallTokenUsage", + "StoredSourceDefinitionManifest", ) diff --git a/src/powercontext/builtin/persistence/source_definitions.py b/src/powercontext/builtin/persistence/source_definitions.py new file mode 100644 index 000000000..eae333ab1 --- /dev/null +++ b/src/powercontext/builtin/persistence/source_definitions.py @@ -0,0 +1,130 @@ +# Copyright (c) 2026 OceanBase. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Persistence for worker-owned Source Definition manifests.""" + +from __future__ import annotations + +from collections.abc import Mapping +from typing import Any + +from pydantic import BaseModel, ConfigDict +from sqlalchemy import insert, select +from sqlalchemy.exc import IntegrityError +from sqlalchemy.ext.asyncio import AsyncConnection + +from powercontext.builtin.persistence.codec import dump_model, load_model, stored_bytes +from powercontext.builtin.persistence.errors import ( + IdentityMismatchError, + RepositoryNotFoundError, + StoredPayloadConflictError, +) +from powercontext.builtin.persistence.tables import SOURCE_DEFINITION_MANIFESTS_TABLE +from powercontext.sources import SourceDefinitionManifest + + +class StoredSourceDefinitionManifest(BaseModel): + """One exact declarative Source Definition registration.""" + + model_config = ConfigDict(frozen=True) + + manifest: SourceDefinitionManifest + + +class SourceDefinitionManifestRepository: + """Register immutable worker-owned Definition manifests by name and version.""" + + async def register( + self, + connection: AsyncConnection, + manifest: SourceDefinitionManifest, + /, + ) -> StoredSourceDefinitionManifest: + payload = dump_model(manifest, kind="source-definition-manifest", name=manifest.name) + existing = await self.find(connection, manifest.name, manifest.version) + if existing is not None: + if existing.manifest != manifest: + raise StoredPayloadConflictError("source-definition-manifest", (manifest.name, manifest.version)) + return existing + try: + await connection.execute( + insert(SOURCE_DEFINITION_MANIFESTS_TABLE).values( + definition_name=manifest.name, + definition_version=manifest.version, + fingerprint=manifest.fingerprint, + manifest=payload, + ) + ) + except IntegrityError: + existing = await self.find(connection, manifest.name, manifest.version) + if existing is None or existing.manifest != manifest: + raise StoredPayloadConflictError( + "source-definition-manifest", + (manifest.name, manifest.version), + ) from None + return existing + return StoredSourceDefinitionManifest(manifest=manifest) + + async def get( + self, + connection: AsyncConnection, + name: str, + version: str, + /, + ) -> StoredSourceDefinitionManifest: + stored = await self.find(connection, name, version) + if stored is None: + raise RepositoryNotFoundError("source-definition-manifest", (name, version)) + return stored + + async def find( + self, + connection: AsyncConnection, + name: str, + version: str, + /, + ) -> StoredSourceDefinitionManifest | None: + row = ( + ( + await connection.execute( + select(SOURCE_DEFINITION_MANIFESTS_TABLE).where( + SOURCE_DEFINITION_MANIFESTS_TABLE.c.definition_name == name, + SOURCE_DEFINITION_MANIFESTS_TABLE.c.definition_version == version, + ) + ) + ) + .mappings() + .one_or_none() + ) + return None if row is None else _decode_row(row) + + +def _decode_row(row: Mapping[Any, Any]) -> StoredSourceDefinitionManifest: + name = str(row["definition_name"]) + version = str(row["definition_version"]) + fingerprint = str(row["fingerprint"]) + manifest = load_model( + SourceDefinitionManifest, + stored_bytes(row["manifest"], column="manifest"), + kind="source-definition-manifest", + name=name, + ) + indexed = (name, version, fingerprint) + decoded = (manifest.name, manifest.version, manifest.fingerprint) + if indexed != decoded: + raise IdentityMismatchError("source-definition-manifest", indexed, decoded) + return StoredSourceDefinitionManifest(manifest=manifest) + + +__all__ = ["SourceDefinitionManifestRepository", "StoredSourceDefinitionManifest"] diff --git a/src/powercontext/builtin/persistence/sources.py b/src/powercontext/builtin/persistence/sources.py index cc1154ee0..9a09b3006 100644 --- a/src/powercontext/builtin/persistence/sources.py +++ b/src/powercontext/builtin/persistence/sources.py @@ -36,7 +36,7 @@ from powercontext.builtin.persistence.tables import SOURCE_JOURNAL_HEADS_TABLE, SOURCES_TABLE from powercontext.errors import SourceDefinitionNotFoundError from powercontext.limits import MAX_SCOPE_ID_LENGTH -from powercontext.sources import Source, SourceAdapter, SourceDefinitionRegistry, SourceRef +from powercontext.sources import ProjectedSource, Source, SourceAdapter, SourceDefinitionRegistry, SourceRef _AnySourceAdapter = SourceAdapter[Any, Any, Any] @@ -73,9 +73,12 @@ async def add( """Add one stable Source or return an identical existing capture.""" _require_identity("scope_id", scope_id, MAX_SCOPE_ID_LENGTH) - definition = self._registry.definition_for_source(source) - ref = SourceRef(source_type=definition.name, source_id=source.name) - payload = dump_model(source, kind="source", name=definition.name) + if isinstance(source, ProjectedSource): + ref = SourceRef(source_type=source.source_type, source_id=source.name) + else: + definition = self._registry.definition_for_source(source) + ref = SourceRef(source_type=definition.name, source_id=source.name) + payload = dump_model(source, kind="source", name=ref.source_type) await _lock_journal_head(connection, scope_id) existing = await self._find_row(connection, scope_id, ref) if existing is not None: @@ -192,16 +195,26 @@ async def _find_row( def _decode_row(self, row: Mapping[Any, Any]) -> StoredSource: source_type = str(row["source_type"]) source_id = str(row["source_id"]) - definition = self._definition_by_name(source_type) - source = load_model( - definition.source_class, - stored_bytes(row["payload"], column="payload"), - kind="source", - name=source_type, - ) + try: + definition = self._definition_by_name(source_type) + except RepositoryNotFoundError: + source = load_model( + ProjectedSource, + stored_bytes(row["payload"], column="payload"), + kind="projected-source", + name=source_type, + ) + decoded = SourceRef(source_type=source.source_type, source_id=source.name) + else: + source = load_model( + definition.source_class, + stored_bytes(row["payload"], column="payload"), + kind="source", + name=source_type, + ) + self._registry.definition_for_source(source) + decoded = SourceRef(source_type=definition.name, source_id=source.name) indexed = SourceRef(source_type=source_type, source_id=source_id) - self._registry.definition_for_source(source) - decoded = SourceRef(source_type=definition.name, source_id=source.name) if indexed != decoded: raise IdentityMismatchError("source", indexed, decoded) return StoredSource( diff --git a/src/powercontext/builtin/persistence/tables.py b/src/powercontext/builtin/persistence/tables.py index cfbcf2d58..f81ed8b18 100644 --- a/src/powercontext/builtin/persistence/tables.py +++ b/src/powercontext/builtin/persistence/tables.py @@ -284,6 +284,16 @@ def _entry_text_type(): Column("checkpoint", _canonical_payload_type(), nullable=False), ) +SOURCE_DEFINITION_MANIFESTS_TABLE = Table( + "pc_source_definition_manifests", + SHARED_METADATA, + Column("definition_name", identity_string(MAX_SOURCE_TYPE_LENGTH), primary_key=True), + Column("definition_version", identity_string(MAX_SOURCE_TYPE_LENGTH), primary_key=True), + Column("fingerprint", identity_string(71), nullable=False), + Column("manifest", _canonical_payload_type(), nullable=False), + UniqueConstraint("definition_name", "fingerprint", name="uq_pc_source_definition_manifest_fingerprint"), +) + EXTERNAL_SKILL_REGISTRATIONS_TABLE = Table( "pc_external_skill_registrations", SHARED_METADATA, @@ -368,6 +378,7 @@ def _entry_text_type(): ARTIFACT_CANDIDATE_HEADS_TABLE, SOURCE_CURSORS_TABLE, CONNECTOR_CHECKPOINTS_TABLE, + SOURCE_DEFINITION_MANIFESTS_TABLE, EXTERNAL_SKILL_REGISTRATIONS_TABLE, ) diff --git a/src/powercontext/builtin/runtime/__init__.py b/src/powercontext/builtin/runtime/__init__.py index c13587b1b..de332fe75 100644 --- a/src/powercontext/builtin/runtime/__init__.py +++ b/src/powercontext/builtin/runtime/__init__.py @@ -45,6 +45,7 @@ ExternalSkillApplication, HandoffApplication, MemoryApplication, + RemoteIngestionApplication, ReviewApplication, ScheduledExperienceProcessor, ScheduledSourceProcessor, @@ -79,6 +80,8 @@ from powercontext.builtin.runtime.models import ( ApproveArtifactCandidateRequest, CaptureSource, + CommitConnectorCheckpoint, + ConnectorCheckpointState, ExperienceCandidate, ExperienceCandidatePage, ExperienceIncubationResult, @@ -120,8 +123,9 @@ SearchMemoryRequest, SkillCandidate, SourceReceipt, + SubmitSourceObservation, ) -from powercontext.builtin.runtime.protocols import PowerContextProvider +from powercontext.builtin.runtime.protocols import PowerContextProvider, RemoteIngestion from powercontext.builtin.runtime.readiness import ( ReadinessCheckStatus, ReadinessProbeDefinition, @@ -165,6 +169,8 @@ "CandidateFamilyCount", "CandidateInventoryStatistics", "CaptureSource", + "CommitConnectorCheckpoint", + "ConnectorCheckpointState", "DatabaseConfig", "ExperienceApplication", "ExperienceCandidate", @@ -242,6 +248,8 @@ "RecallTokenValue", "RejectArtifactCandidateRequest", "RememberMemoryRequest", + "RemoteIngestion", + "RemoteIngestionApplication", "ResolveExternalSkillRequest", "ResolvedUsagePeriod", "RetireMemoryEntryRequest", @@ -276,6 +284,7 @@ "Statistics", "StatisticsApplication", "StatisticsPeriod", + "SubmitSourceObservation", "UsageStatistics", "WorkApplication", "open_builtin_contexts", diff --git a/src/powercontext/builtin/runtime/application.py b/src/powercontext/builtin/runtime/application.py index 94eb7e8ee..9c5fa9f7a 100644 --- a/src/powercontext/builtin/runtime/application.py +++ b/src/powercontext/builtin/runtime/application.py @@ -83,6 +83,8 @@ from powercontext.builtin.runtime.models import ( ApproveArtifactCandidateRequest, CaptureSource, + CommitConnectorCheckpoint, + ConnectorCheckpointState, ExperienceCandidate, ExperienceIncubationResult, ExternalSkillList, @@ -118,11 +120,13 @@ SearchMemoryRequest, SkillCandidate, SourceReceipt, + SubmitSourceObservation, ) from powercontext.builtin.runtime.prepared_context import PreparedContextBuild, PreparedContextBuilder from powercontext.builtin.runtime.protocols import ( BuiltinTriggers, PowerContextProvider, + RemoteIngestion, RuntimeSpan, RuntimeTracing, TraceAttribute, @@ -170,7 +174,7 @@ ) from powercontext.context import PowerContext from powercontext.errors import ArtifactNotFoundError, RevisionConflictError -from powercontext.sources import SourceRef +from powercontext.sources import ConnectorBinding, SourceDefinitionManifest, SourceRef if TYPE_CHECKING: from apscheduler.schedulers.asyncio import AsyncIOScheduler @@ -214,6 +218,7 @@ def __init__(self, code: str) -> None: "empty-write": "explicit Memory write did not produce a Memory", "experience-incubation": "Experience incubation is not configured", "external-skill-registry": "External Skill Registry is not configured", + "remote-ingestion": "Remote Source ingestion is not configured", "review": "Candidate Review services are not configured", "scheduler": "Built-in Runtime scheduler is already started", "statistics": "Statistics services are not configured", @@ -250,6 +255,35 @@ def for_scope(self, scope_id: str, /) -> ScopedSourceApplication: return ScopedSourceApplication(self._runtime, scope_id) +class RemoteIngestionApplication: + """Expose worker-owned Definition and observation operations.""" + + def __init__(self, runtime: BuiltinRuntime, service: RemoteIngestion | None) -> None: + self._runtime = runtime + self._service = service + + def _require_service(self) -> RemoteIngestion: + if self._service is None: + raise _RuntimeStateError("remote-ingestion") + return self._service + + async def register(self, manifest: SourceDefinitionManifest, /) -> SourceDefinitionManifest: + async with self._runtime._operation(): + return await self._require_service().register_source_definition(manifest) + + async def checkpoint(self, binding: ConnectorBinding, /) -> ConnectorCheckpointState: + async with self._runtime._operation(): + return await self._require_service().connector_checkpoint(binding) + + async def submit(self, request: SubmitSourceObservation, /) -> SourceReceipt: + async with self._runtime._operation(): + return await self._require_service().submit_source_observation(request) + + async def commit(self, request: CommitConnectorCheckpoint, /) -> ConnectorCheckpointState: + async with self._runtime._operation(): + return await self._require_service().commit_connector_checkpoint(request) + + class ScopedStatisticsApplication: """Read product statistics and record model usage for one scope.""" @@ -1194,6 +1228,7 @@ def __init__( readiness: RuntimeReadinessChecks | None = None, clock: Clock | None = None, tracing: RuntimeTracing | None = None, + remote_ingestion: RemoteIngestion | None = None, ) -> None: if source_window_limit < 1: raise _RuntimeConfigurationError("source_window_limit") @@ -1228,6 +1263,7 @@ def __init__( self._scheduler: AsyncIOScheduler | None = None self._scheduler_runtime_key: str | None = None self.sources = SourceApplication(self) + self.ingestion = RemoteIngestionApplication(self, remote_ingestion) self.context = ContextApplication(self) self.experience = ExperienceApplication(self) self.external_skills = ExternalSkillApplication(self) diff --git a/src/powercontext/builtin/runtime/composition.py b/src/powercontext/builtin/runtime/composition.py index c8419a80a..bb35ae7a2 100644 --- a/src/powercontext/builtin/runtime/composition.py +++ b/src/powercontext/builtin/runtime/composition.py @@ -291,6 +291,7 @@ async def open_builtin_runtime( recall_token_estimator=contexts.estimate_recall_tokens, readiness=RuntimeReadinessChecks(readiness_probes), tracing=tracing, + remote_ingestion=contexts, ) ) if config.handoff_report.enabled: diff --git a/src/powercontext/builtin/runtime/models.py b/src/powercontext/builtin/runtime/models.py index 95e8c4cf1..42dd8decf 100644 --- a/src/powercontext/builtin/runtime/models.py +++ b/src/powercontext/builtin/runtime/models.py @@ -49,7 +49,7 @@ ) from powercontext.builtin.review.generation import SkillGenerationOrigin from powercontext.builtin.sources import ExternalSkillImportMode -from powercontext.sources import SourceRef +from powercontext.sources import ConnectorBinding, ProjectedSource, SourceDefinitionManifest, SourceRef PreparedContextSchema: TypeAlias = Literal["powercontext.prepared-context.v1"] PreparedContextStatus: TypeAlias = Literal["ready", "empty"] @@ -77,6 +77,34 @@ class SourceReceipt(BaseModel): sequence: int +class RegisterSourceDefinition(BaseModel): + """Register one immutable worker-owned Source Definition manifest.""" + + manifest: SourceDefinitionManifest + + +class SubmitSourceObservation(BaseModel): + """Submit one worker-materialized observation for durable acceptance.""" + + binding: ConnectorBinding + source: ProjectedSource + + +class ConnectorCheckpointState(BaseModel): + """Current opaque checkpoint for one exact Connector binding.""" + + binding: ConnectorBinding + checkpoint: JsonValue | None + + +class CommitConnectorCheckpoint(BaseModel): + """Compare and replace one binding checkpoint after durable submissions.""" + + binding: ConnectorBinding + expected: JsonValue | None + checkpoint: JsonValue | None + + class RuntimeCapabilities(BaseModel): """Behavior available from the assembled Source-to-Memory Runtime.""" diff --git a/src/powercontext/builtin/runtime/protocols.py b/src/powercontext/builtin/runtime/protocols.py index af3a66e3b..3366e6a42 100644 --- a/src/powercontext/builtin/runtime/protocols.py +++ b/src/powercontext/builtin/runtime/protocols.py @@ -21,9 +21,16 @@ from typing import Protocol, TypeVar from powercontext.builtin.artifacts.handoff import ActivateHandoff, HandoffActivation -from powercontext.builtin.runtime.models import MemoryFlushResult +from powercontext.builtin.runtime.models import ( + CommitConnectorCheckpoint, + ConnectorCheckpointState, + MemoryFlushResult, + SourceReceipt, + SubmitSourceObservation, +) from powercontext.builtin.sources import SourceCursor from powercontext.context import PowerContext +from powercontext.sources import ConnectorBinding, SourceDefinitionManifest SourcesT = TypeVar("SourcesT", covariant=True) ArtifactsT = TypeVar("ArtifactsT", covariant=True) @@ -54,6 +61,26 @@ class PowerContextProvider(Protocol[SourcesT, ArtifactsT, TriggersT]): async def get(self, scope_id: str, /) -> PowerContext[SourcesT, ArtifactsT, TriggersT]: ... +class RemoteIngestion(Protocol): + """Server-side authority used by independent Connector workers.""" + + async def register_source_definition( + self, + manifest: SourceDefinitionManifest, + /, + ) -> SourceDefinitionManifest: ... + + async def connector_checkpoint(self, binding: ConnectorBinding, /) -> ConnectorCheckpointState: ... + + async def submit_source_observation(self, request: SubmitSourceObservation, /) -> SourceReceipt: ... + + async def commit_connector_checkpoint( + self, + request: CommitConnectorCheckpoint, + /, + ) -> ConnectorCheckpointState: ... + + class BuiltinTriggers(Protocol): """Atomically execute the built-in Trigger policies for one scope.""" diff --git a/src/powercontext/builtin/runtime/relational.py b/src/powercontext/builtin/runtime/relational.py index 1ed68cf56..7cb187b4f 100644 --- a/src/powercontext/builtin/runtime/relational.py +++ b/src/powercontext/builtin/runtime/relational.py @@ -17,11 +17,15 @@ from __future__ import annotations import asyncio -from collections.abc import Callable +from collections.abc import Callable, Mapping from dataclasses import dataclass -from typing import cast +from typing import Any, cast from uuid import uuid4 +from jsonschema import Draft202012Validator +from jsonschema.exceptions import SchemaError +from jsonschema.exceptions import ValidationError as JsonSchemaValidationError +from jsonschema.protocols import Validator from sqlalchemy import select from sqlalchemy.ext.asyncio import AsyncConnection @@ -78,6 +82,7 @@ ) from powercontext.builtin.persistence.memory import RelationalMemoryBackend from powercontext.builtin.persistence.memory_index import MemoryIndex, NoMemoryIndex +from powercontext.builtin.persistence.source_definitions import SourceDefinitionManifestRepository from powercontext.builtin.persistence.sources import SourceRepository, StoredSource from powercontext.builtin.persistence.statistics import StatisticsRepository from powercontext.builtin.persistence.tables import ARTIFACT_HEADS_TABLE, SOURCE_JOURNAL_HEADS_TABLE @@ -88,7 +93,14 @@ SkillGenerationOrigin, ) from powercontext.builtin.review.service import ReviewService -from powercontext.builtin.runtime.models import ExperienceIncubationResult, MemoryFlushResult +from powercontext.builtin.runtime.models import ( + CommitConnectorCheckpoint, + ConnectorCheckpointState, + ExperienceIncubationResult, + MemoryFlushResult, + SourceReceipt, + SubmitSourceObservation, +) from powercontext.builtin.runtime.prepared_context import PreparedContextBuild from powercontext.builtin.runtime.protocols import BuiltinTriggers from powercontext.builtin.runtime.recall import RelationalRecallTokenEstimator @@ -113,17 +125,24 @@ SourceWindowTrigger, ) from powercontext.context import PowerContext -from powercontext.errors import ArtifactNotFoundError, SourceConflictError, SourceNotFoundError +from powercontext.errors import ( + ArtifactNotFoundError, + InvalidSourceDefinitionError, + InvalidSourceObservationError, + SourceConflictError, + SourceDefinitionNotFoundError, + SourceNotFoundError, +) from powercontext.sources import ( - CatalogConnectorSourceSink, - Connector, + TEXT_EVIDENCE_PROJECTION_KEY, ConnectorBinding, - ConnectorLifecycle, - ConnectorRunResult, + ProjectedSource, Source, SourceCatalog, + SourceDefinitionManifest, SourceDefinitionRegistry, SourceRef, + TextEvidence, ) IdFactory = Callable[[str], str] @@ -137,6 +156,7 @@ class _Repositories: artifacts: ArtifactRepository candidates: CandidateRepository connector_checkpoints: ConnectorCheckpointRepository + source_definitions: SourceDefinitionManifestRepository cursors: SourceCursorRepository external_skills: ExternalSkillRepository statistics: StatisticsRepository @@ -324,6 +344,7 @@ def __init__( Skill.family: SkillContent, }), connector_checkpoints=ConnectorCheckpointRepository(), + source_definitions=SourceDefinitionManifestRepository(), cursors=SourceCursorRepository(), external_skills=ExternalSkillRepository(), statistics=StatisticsRepository(), @@ -379,28 +400,69 @@ def statistics(self, scope_id: str, /) -> RelationalScopedStatistics: return self._services_for(scope_id).statistics() - async def run_connector( + async def register_source_definition( self, - connector: Connector, - binding: ConnectorBinding, + manifest: SourceDefinitionManifest, /, - ) -> ConnectorRunResult: - """Run one Connector binding with durable Sources and checkpoint ordering.""" + ) -> SourceDefinitionManifest: + """Register one immutable declarative Definition supplied by a worker.""" - services = self._services_for(binding.scope_id) + _validate_source_definition_manifest(manifest) + try: + async with self.database.transaction() as connection: + stored = await self.repositories.source_definitions.register(connection, manifest) + except StoredPayloadConflictError as error: + raise SourceConflictError("definition-manifest", error.identity) from None + return stored.manifest + + async def connector_checkpoint(self, binding: ConnectorBinding, /) -> ConnectorCheckpointState: + """Read the checkpoint owned by one remote Connector binding.""" + + checkpoint = await RelationalConnectorCheckpointStore( + self.database, + self.repositories.connector_checkpoints, + ).load(binding) + return ConnectorCheckpointState(binding=binding, checkpoint=checkpoint) + + async def submit_source_observation( + self, + request: SubmitSourceObservation, + /, + ) -> SourceReceipt: + """Validate and durably append one worker-materialized Source observation.""" + + source = request.source + try: + async with self.database.transaction() as connection: + stored_manifest = await self.repositories.source_definitions.get( + connection, + source.source_type, + source.definition_version, + ) + except RepositoryNotFoundError: + raise SourceDefinitionNotFoundError(source.source_type, source.definition_version) from None + _validate_projected_source(source, stored_manifest.manifest) + services = self._services_for(request.binding.scope_id) source_store, source_catalog = services.sources() - lifecycle = ConnectorLifecycle( - sink=CatalogConnectorSourceSink( - scope_id=services.scope_id, - catalog=source_catalog, - store=source_store, - ), - checkpoints=RelationalConnectorCheckpointStore( - self.database, - self.repositories.connector_checkpoints, - ), + stored = await source_store.add(source) + return SourceReceipt( + source_ref=source_catalog.as_ref(stored), + sequence=await source_store.position(stored), + ) + + async def commit_connector_checkpoint( + self, + request: CommitConnectorCheckpoint, + /, + ) -> ConnectorCheckpointState: + """Commit one worker checkpoint only when its starting value still matches.""" + + store = RelationalConnectorCheckpointStore( + self.database, + self.repositories.connector_checkpoints, ) - return await lifecycle.run(connector, binding) + await store.save(request.binding, request.checkpoint, expected=request.expected) + return ConnectorCheckpointState(binding=request.binding, checkpoint=request.checkpoint) async def estimate_recall_tokens( self, @@ -619,6 +681,8 @@ async def entries(self) -> tuple[SourceJournalEntry, ...]: ) def _as_ref(self, source: Source) -> SourceRef: + if isinstance(source, ProjectedSource): + return SourceRef(source_type=source.source_type, source_id=source.name) definition = self._registry.definition_for_source(source) return SourceRef(source_type=definition.name, source_id=source.name) @@ -885,6 +949,67 @@ def _validate_experience_plans( ) +def _validate_source_definition_manifest(manifest: SourceDefinitionManifest) -> None: + if len(manifest.model_dump_json(by_alias=True).encode()) > 64 * 1024: + raise InvalidSourceDefinitionError(type(manifest), "manifest", "must not exceed 64 KiB") + try: + BUILTIN_SOURCE_REGISTRY.definition_for_name(manifest.name) + except SourceDefinitionNotFoundError: + pass + else: + raise InvalidSourceDefinitionError(type(manifest), "name", "must not replace a built-in Source Definition") + _json_schema_validator(manifest.name, manifest.source_schema) + standard_text_schema = TextEvidence.model_json_schema() + for projection in manifest.projections: + _json_schema_validator(projection.key.name, projection.schema_) + if projection.key == TEXT_EVIDENCE_PROJECTION_KEY and projection.schema_ != standard_text_schema: + raise InvalidSourceDefinitionError( + type(manifest), + "projection", + f"{projection.key.name!r} must use the standard schema", + ) + + +def _validate_projected_source(source: ProjectedSource, manifest: SourceDefinitionManifest) -> None: + if source.source_type != manifest.name or source.definition_version != manifest.version: + raise InvalidSourceObservationError("definition", "does not match the registered manifest identity") + if source.definition_fingerprint != manifest.fingerprint: + raise InvalidSourceObservationError("fingerprint", "does not match the registered manifest") + if len(source.model_dump_json().encode()) > 4 * 1024 * 1024: + raise InvalidSourceObservationError("size", "must not exceed 4 MiB") + _validate_schema_value(manifest.name, manifest.source_schema, source.payload) + + declarations = {projection.key: projection for projection in manifest.projections} + supplied = {projection.key: projection.value for projection in source.projections} + if declarations.keys() != supplied.keys(): + raise InvalidSourceObservationError("projections", "must exactly match the registered manifest") + for key, declaration in declarations.items(): + value = supplied[key] + _validate_schema_value(key.name, declaration.schema_, value) + if key == TEXT_EVIDENCE_PROJECTION_KEY: + evidence = TextEvidence.model_validate(value) + if evidence.source_type != source.source_type or evidence.source_id != source.name: + raise InvalidSourceObservationError( + "text-evidence", + "source identity does not match the observation envelope", + ) + + +def _json_schema_validator(name: str, schema: Mapping[str, Any]) -> Validator: + try: + Draft202012Validator.check_schema(schema) + return Draft202012Validator(schema) + except SchemaError as error: + raise InvalidSourceDefinitionError(type(schema), "schema", f"{name!r} is not valid JSON Schema") from error + + +def _validate_schema_value(name: str, schema: Mapping[str, Any], value: object) -> None: + try: + _json_schema_validator(name, schema).validate(value) + except JsonSchemaValidationError as error: + raise InvalidSourceObservationError("schema", f"value does not match {name!r}") from error + + def _scoped_id_factory(memory_artifact_id: str, delegate: IdFactory | None) -> IdFactory: def new_id(kind: str) -> str: if kind == "memory": diff --git a/src/powercontext/builtin/sources/__init__.py b/src/powercontext/builtin/sources/__init__.py index 5ca040828..66a50d50e 100644 --- a/src/powercontext/builtin/sources/__init__.py +++ b/src/powercontext/builtin/sources/__init__.py @@ -38,22 +38,11 @@ SourceJournalEntry, validate_scope_id, ) -from powercontext.builtin.sources.projections import TEXT_EVIDENCE_PROJECTION_KEY, TextEvidence -from powercontext.builtin.sources.text_file import ( - TEXT_FILE_SNAPSHOT_SOURCE_ADAPTER, - TEXT_FILE_SNAPSHOT_SOURCE_DEFINITION, - TEXT_FILE_SNAPSHOT_SOURCE_NAME, - TextFileEvidenceProjection, - TextFileSnapshotCapture, - TextFileSnapshotSource, - TextFileSnapshotSourceAdapter, -) -from powercontext.sources import SourceDefinitionRegistry +from powercontext.sources import TEXT_EVIDENCE_PROJECTION_KEY, SourceDefinitionRegistry, TextEvidence BUILTIN_SOURCE_REGISTRY = SourceDefinitionRegistry(( CONTENT_SOURCE_DEFINITION, EXTERNAL_SKILL_SNAPSHOT_SOURCE_DEFINITION, - TEXT_FILE_SNAPSHOT_SOURCE_DEFINITION, )) __all__ = [ @@ -65,9 +54,6 @@ "EXTERNAL_SKILL_SNAPSHOT_SOURCE_DEFINITION", "EXTERNAL_SKILL_SNAPSHOT_SOURCE_NAME", "TEXT_EVIDENCE_PROJECTION_KEY", - "TEXT_FILE_SNAPSHOT_SOURCE_ADAPTER", - "TEXT_FILE_SNAPSHOT_SOURCE_DEFINITION", - "TEXT_FILE_SNAPSHOT_SOURCE_NAME", "ContentCapture", "ContentSource", "ContentSourceAdapter", @@ -80,9 +66,5 @@ "SourceJournal", "SourceJournalEntry", "TextEvidence", - "TextFileEvidenceProjection", - "TextFileSnapshotCapture", - "TextFileSnapshotSource", - "TextFileSnapshotSourceAdapter", "validate_scope_id", ] diff --git a/src/powercontext/builtin/sources/content.py b/src/powercontext/builtin/sources/content.py index f008777bb..34a7c0157 100644 --- a/src/powercontext/builtin/sources/content.py +++ b/src/powercontext/builtin/sources/content.py @@ -20,8 +20,7 @@ from pydantic import BaseModel, Field, JsonValue, field_validator -from powercontext.builtin.sources.projections import TEXT_EVIDENCE_PROJECTION_KEY, TextEvidence -from powercontext.sources import AdapterSourceDefinition +from powercontext.sources import TEXT_EVIDENCE_PROJECTION_KEY, AdapterSourceDefinition, TextEvidence from powercontext.sources.models import Source, SourceMaterialization CONTENT_SOURCE_NAME = "content" diff --git a/src/powercontext/client/__init__.py b/src/powercontext/client/__init__.py index 55ca5bd9e..673ffb321 100644 --- a/src/powercontext/client/__init__.py +++ b/src/powercontext/client/__init__.py @@ -16,11 +16,19 @@ from powercontext.client.client import PowerContextClient from powercontext.client.errors import ClientError, InvalidResponseError, ServerResponseError, TransportError +from powercontext.client.ingestion import ( + RemoteConnectorCheckpointStore, + RemoteConnectorSourceSink, + RemoteConnectorWorker, +) __all__ = [ "ClientError", "InvalidResponseError", "PowerContextClient", + "RemoteConnectorCheckpointStore", + "RemoteConnectorSourceSink", + "RemoteConnectorWorker", "ServerResponseError", "TransportError", ] diff --git a/src/powercontext/client/client.py b/src/powercontext/client/client.py index 2c1618f2f..47d60929e 100644 --- a/src/powercontext/client/client.py +++ b/src/powercontext/client/client.py @@ -35,8 +35,10 @@ Capabilities, CaptureContentSourceRequest, CaptureContentSourceResponse, + CommitConnectorCheckpointRequest, CommitHandoffRequest, CommittedHandoff, + ConnectorCheckpointState, ContinueHandoffRequest, CreateHandoffReportProjectRequest, CreateWorkContractRequest, @@ -51,6 +53,7 @@ GenerateExperienceRequest, GenerateSkillRequest, GetArtifactCandidateRequest, + GetConnectorCheckpointRequest, GetExperienceRequest, GetHandoffReportProjectRequest, GetHandoffReportRequest, @@ -97,6 +100,7 @@ RecordHandoffReportActivityRequest, RecordTaskOutcomeRequest, RegisterHandoffReportWorkstreamRequest, + RegisterSourceDefinitionRequest, RejectArtifactCandidateRequest, RememberMemoryRequest, ResolveExternalSkillRequest, @@ -109,7 +113,10 @@ SearchMemoryRequest, SearchMemoryResponse, SkillArtifact, + SourceDefinitionManifest, + SourceObservationReceipt, StoredHandoffReportActivity, + SubmitSourceObservationRequest, UpdateHandoffReportProjectRequest, UpdateHandoffReportWorkstreamRequest, WorkSourceReceipt, @@ -122,6 +129,7 @@ APPROVE_ARTIFACT_CANDIDATE, ATTACH_HANDOFF_REPORT_WORKSPACE, CAPTURE_CONTENT_SOURCE, + COMMIT_CONNECTOR_CHECKPOINT, COMMIT_HANDOFF, CONTINUE_HANDOFF, CREATE_HANDOFF_REPORT_PROJECT, @@ -133,6 +141,7 @@ GENERATE_SKILL, GET_ARTIFACT_CANDIDATE, GET_CAPABILITIES, + GET_CONNECTOR_CHECKPOINT, GET_EXPERIENCE, GET_HANDOFF_REPORT, GET_HANDOFF_REPORT_PROJECT, @@ -160,6 +169,7 @@ RECORD_HANDOFF_REPORT_ACTIVITY, RECORD_TASK_OUTCOME, REGISTER_HANDOFF_REPORT_WORKSTREAM, + REGISTER_SOURCE_DEFINITION, REJECT_ARTIFACT_CANDIDATE, REMEMBER_MEMORY, RESOLVE_EXTERNAL_SKILL, @@ -168,6 +178,7 @@ REVISE_MEMORY_ENTRY, SCAN_EXTERNAL_SKILLS, SEARCH_MEMORY, + SUBMIT_SOURCE_OBSERVATION, UPDATE_HANDOFF_REPORT_PROJECT, UPDATE_HANDOFF_REPORT_WORKSTREAM, Operation, @@ -423,6 +434,29 @@ async def capture_content_source(self, request: CaptureContentSourceRequest) -> return await self._request(CAPTURE_CONTENT_SOURCE, request) + async def register_source_definition(self, request: RegisterSourceDefinitionRequest) -> SourceDefinitionManifest: + """Register one immutable worker-owned Source Definition manifest.""" + + return await self._request(REGISTER_SOURCE_DEFINITION, request) + + async def get_connector_checkpoint(self, request: GetConnectorCheckpointRequest) -> ConnectorCheckpointState: + """Read the current opaque checkpoint for one Connector binding.""" + + return await self._request(GET_CONNECTOR_CHECKPOINT, request) + + async def submit_source_observation(self, request: SubmitSourceObservationRequest) -> SourceObservationReceipt: + """Submit one worker-materialized Source observation.""" + + return await self._request(SUBMIT_SOURCE_OBSERVATION, request) + + async def commit_connector_checkpoint( + self, + request: CommitConnectorCheckpointRequest, + ) -> ConnectorCheckpointState: + """Commit a binding checkpoint using optimistic comparison.""" + + return await self._request(COMMIT_CONNECTOR_CHECKPOINT, request) + async def create_work_contract(self, request: CreateWorkContractRequest) -> WorkSourceReceipt: """Create one grounded delegation baseline as durable Source evidence.""" diff --git a/src/powercontext/client/ingestion.py b/src/powercontext/client/ingestion.py new file mode 100644 index 000000000..eb049050f --- /dev/null +++ b/src/powercontext/client/ingestion.py @@ -0,0 +1,176 @@ +# Copyright (c) 2026 OceanBase. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Run worker-owned Connectors against the remote ingestion contract.""" + +from __future__ import annotations + +from pydantic import JsonValue +from typing_extensions import override + +from powercontext.client.client import PowerContextClient +from powercontext.errors import InvalidConnectorRunError +from powercontext.http import ( + CommitConnectorCheckpointRequest, + GetConnectorCheckpointRequest, + RegisterSourceDefinitionRequest, + SubmitSourceObservationRequest, +) +from powercontext.http import ( + ConnectorBinding as HttpConnectorBinding, +) +from powercontext.http import ( + ProjectedSource as HttpProjectedSource, +) +from powercontext.http import ( + SourceDefinitionManifest as HttpSourceDefinitionManifest, +) +from powercontext.sources import ( + Connector, + ConnectorBinding, + ConnectorCheckpointStore, + ConnectorLifecycle, + ConnectorRunResult, + ConnectorSourceSink, + ConnectorSubmissionResult, + ConnectorSubmissionStatus, + SourceDefinitionRegistry, + SourceRef, + manifest_for_definition, + project_source_for_transport, + validate_connector, +) + + +class RemoteConnectorSourceSink(ConnectorSourceSink): + """Resolve and project Definition-native values inside the worker.""" + + def __init__(self, *, client: PowerContextClient, registry: SourceDefinitionRegistry) -> None: + self._client = client + self._registry = registry + + @override + async def submit( + self, + binding: ConnectorBinding, + item_id: str, + definition_name: str, + value: object, + /, + ) -> ConnectorSubmissionResult: + del item_id + self._registry.definition_for_name(definition_name) + source = await self._registry.resolve(value) + projected = project_source_for_transport(self._registry, source) + if projected.source_type != definition_name: + raise InvalidConnectorRunError( + "definition-mismatch", + f"input resolved as {projected.source_type!r}, expected {definition_name!r}", + ) + receipt = await self._client.submit_source_observation( + SubmitSourceObservationRequest( + binding=_http_binding(binding), + source=HttpProjectedSource.model_validate(projected.model_dump(mode="json")), + ) + ) + source_ref = SourceRef(source_type=receipt.source.name, source_id=receipt.source.source_id) + expected_ref = SourceRef(source_type=projected.source_type, source_id=projected.name) + if source_ref != expected_ref: + raise InvalidConnectorRunError("identity-mismatch", "Server receipt changed the accepted Source identity") + return ConnectorSubmissionResult(status=ConnectorSubmissionStatus.ACCEPTED, source_ref=source_ref) + + +class RemoteConnectorCheckpointStore(ConnectorCheckpointStore): + """Load and compare-and-swap opaque checkpoints through the Server API.""" + + def __init__(self, client: PowerContextClient) -> None: + self._client = client + + @override + async def load(self, binding: ConnectorBinding, /) -> JsonValue | None: + state = await self._client.get_connector_checkpoint( + GetConnectorCheckpointRequest(binding=_http_binding(binding)) + ) + _validate_checkpoint_binding(binding, state.binding) + return state.checkpoint + + @override + async def save( + self, + binding: ConnectorBinding, + checkpoint: JsonValue | None, + /, + *, + expected: JsonValue | None, + ) -> None: + state = await self._client.commit_connector_checkpoint( + CommitConnectorCheckpointRequest( + binding=_http_binding(binding), + expected=expected, + checkpoint=checkpoint, + ) + ) + _validate_checkpoint_binding(binding, state.binding) + if state.checkpoint != checkpoint: + raise InvalidConnectorRunError("checkpoint-mismatch", "Server returned a different Connector checkpoint") + + +class RemoteConnectorWorker: + """Register worker-owned Definitions and execute one Connector binding.""" + + def __init__(self, *, client: PowerContextClient, registry: SourceDefinitionRegistry) -> None: + self._client = client + self._registry = registry + self._lifecycle = ConnectorLifecycle( + sink=RemoteConnectorSourceSink(client=client, registry=registry), + checkpoints=RemoteConnectorCheckpointStore(client), + ) + + async def run(self, connector: Connector, binding: ConnectorBinding, /) -> ConnectorRunResult: + source_definitions, _ = validate_connector(connector, binding) + for definition_name in sorted(source_definitions): + definition = self._registry.definition_for_name(definition_name) + manifest = manifest_for_definition(definition) + registered = await self._client.register_source_definition( + RegisterSourceDefinitionRequest( + manifest=HttpSourceDefinitionManifest.model_validate( + manifest.model_dump(mode="json", by_alias=True) + ) + ) + ) + if registered.model_dump(mode="json", by_alias=True) != manifest.model_dump(mode="json", by_alias=True): + raise InvalidConnectorRunError( + "manifest-mismatch", + f"Server returned a different manifest for {definition.name!r}", + ) + return await self._lifecycle.run(connector, binding) + + +def _http_binding(binding: ConnectorBinding) -> HttpConnectorBinding: + return HttpConnectorBinding.model_validate(binding.model_dump(mode="json")) + + +def _validate_checkpoint_binding( + expected_binding: ConnectorBinding, + actual_binding: HttpConnectorBinding, +) -> None: + if actual_binding.model_dump(mode="json") != expected_binding.model_dump(mode="json"): + raise InvalidConnectorRunError("binding-mismatch", "Server returned a different Connector binding") + + +__all__ = [ + "RemoteConnectorCheckpointStore", + "RemoteConnectorSourceSink", + "RemoteConnectorWorker", +] diff --git a/src/powercontext/errors.py b/src/powercontext/errors.py index 172032796..def1e0515 100644 --- a/src/powercontext/errors.py +++ b/src/powercontext/errors.py @@ -142,6 +142,15 @@ def __init__(self, projection_name: str, field: str, detail: str) -> None: super().__init__(f"invalid Source projection {projection_name!r} {field}: {detail}") +class InvalidSourceObservationError(SourceError, ValueError): + """Raised when a worker-projected observation violates its registered manifest.""" + + def __init__(self, issue: str, detail: str) -> None: + self.issue = issue + self.detail = detail + super().__init__(f"invalid Source observation {issue}: {detail}") + + class ConnectorError(PowerContextError): """Base exception for Connector contracts and run lifecycle failures.""" diff --git a/src/powercontext/http/__init__.py b/src/powercontext/http/__init__.py index 56c05bf89..8bce5c622 100644 --- a/src/powercontext/http/__init__.py +++ b/src/powercontext/http/__init__.py @@ -31,8 +31,11 @@ CaptureContentSourceRequest, CaptureContentSourceResponse, CaptureStatus, + CommitConnectorCheckpointRequest, CommitHandoffRequest, CommittedHandoff, + ConnectorBinding, + ConnectorCheckpointState, ContinueHandoffRequest, CreateHandoffReportProjectRequest, CreateWorkContractRequest, @@ -59,6 +62,7 @@ GenerateExperienceRequest, GenerateSkillRequest, GetArtifactCandidateRequest, + GetConnectorCheckpointRequest, GetExperienceRequest, GetHandoffReportProjectRequest, GetHandoffReportRequest, @@ -136,6 +140,7 @@ PreparedWorkHandoff, PrepareHandoffRequest, ProjectDescriptor, + ProjectedSource, ProjectPage, ProposeExperienceRequest, ProposeSkillRequest, @@ -149,6 +154,7 @@ RecordHandoffReportActivityRequest, RecordTaskOutcomeRequest, RegisterHandoffReportWorkstreamRequest, + RegisterSourceDefinitionRequest, RejectArtifactCandidateRequest, RememberMemoryRequest, ReportActivitySource, @@ -171,10 +177,16 @@ SkillGenerationOrigin, SkillProposal, SkillValidationItem, + SourceDefinitionManifest, SourceInventoryStatistics, + SourceObservationReceipt, + SourceProjectionKey, + SourceProjectionManifest, + SourceProjectionValue, SourceReference, StatsPeriod, StoredHandoffReportActivity, + SubmitSourceObservationRequest, TaskCheck, TaskCheckStatus, TaskOutcome, @@ -210,8 +222,11 @@ "CaptureContentSourceRequest", "CaptureContentSourceResponse", "CaptureStatus", + "CommitConnectorCheckpointRequest", "CommitHandoffRequest", "CommittedHandoff", + "ConnectorBinding", + "ConnectorCheckpointState", "ContinueHandoffRequest", "CreateHandoffReportProjectRequest", "CreateWorkContractRequest", @@ -238,6 +253,7 @@ "GeneratedCandidateResponse", "GeneratedCandidateStatus", "GetArtifactCandidateRequest", + "GetConnectorCheckpointRequest", "GetExperienceRequest", "GetHandoffReportProjectRequest", "GetHandoffReportRequest", @@ -316,6 +332,7 @@ "PreparedWorkHandoff", "ProjectDescriptor", "ProjectPage", + "ProjectedSource", "ProposeExperienceRequest", "ProposeSkillRequest", "PurgeHandoffReportActivitiesRequest", @@ -328,6 +345,7 @@ "RecordHandoffReportActivityRequest", "RecordTaskOutcomeRequest", "RegisterHandoffReportWorkstreamRequest", + "RegisterSourceDefinitionRequest", "RejectArtifactCandidateRequest", "RememberMemoryRequest", "ReportActivitySource", @@ -350,10 +368,16 @@ "SkillGenerationOrigin", "SkillProposal", "SkillValidationItem", + "SourceDefinitionManifest", "SourceInventoryStatistics", + "SourceObservationReceipt", + "SourceProjectionKey", + "SourceProjectionManifest", + "SourceProjectionValue", "SourceReference", "StatsPeriod", "StoredHandoffReportActivity", + "SubmitSourceObservationRequest", "TaskCheck", "TaskCheckStatus", "TaskOutcome", diff --git a/src/powercontext/http/_generated/models.py b/src/powercontext/http/_generated/models.py index ad2a598c1..f7a42405d 100644 --- a/src/powercontext/http/_generated/models.py +++ b/src/powercontext/http/_generated/models.py @@ -285,6 +285,109 @@ class CaptureContentSourceRequest(BaseModel): metadata: dict[str, Any] | None = None +class SourceProjectionKey(BaseModel): + model_config = ConfigDict( + extra="forbid", + ) + name: Annotated[StrictStr, Field(max_length=128, min_length=1, pattern=".*\\S.*")] + version: Annotated[StrictStr, Field(max_length=128, min_length=1, pattern=".*\\S.*")] + + +class SourceProjectionManifest(BaseModel): + model_config = ConfigDict( + extra="forbid", + ) + key: SourceProjectionKey + schema_: Annotated[dict[str, Any], Field(alias="schema")] + + +class SourceDefinitionManifest(BaseModel): + model_config = ConfigDict( + extra="forbid", + ) + name: Annotated[StrictStr, Field(max_length=128, min_length=1, pattern=".*\\S.*")] + version: Annotated[StrictStr, Field(max_length=128, min_length=1, pattern=".*\\S.*")] + fingerprint: Annotated[StrictStr, Field(pattern="^sha256:[0-9a-f]{64}$")] + source_schema: dict[str, Any] + projections: Annotated[list[SourceProjectionManifest], Field(max_length=16)] + + +class RegisterSourceDefinitionRequest(BaseModel): + model_config = ConfigDict( + extra="forbid", + ) + manifest: SourceDefinitionManifest + + +class ConnectorBinding(BaseModel): + model_config = ConfigDict( + extra="forbid", + ) + scope_id: Annotated[StrictStr, Field(max_length=256, min_length=1, pattern=".*\\S.*")] + binding_id: Annotated[StrictStr, Field(max_length=256, min_length=1, pattern=".*\\S.*")] + connector_name: Annotated[StrictStr, Field(max_length=128, min_length=1, pattern=".*\\S.*")] + connector_version: Annotated[StrictStr, Field(max_length=128, min_length=1, pattern=".*\\S.*")] + + +class GetConnectorCheckpointRequest(BaseModel): + model_config = ConfigDict( + extra="forbid", + ) + binding: ConnectorBinding + + +class ConnectorCheckpointState(BaseModel): + model_config = ConfigDict( + extra="forbid", + ) + binding: ConnectorBinding + checkpoint: Annotated[Any | None, Field(...)] + + +class SourceProjectionValue(BaseModel): + model_config = ConfigDict( + extra="forbid", + ) + key: SourceProjectionKey + value: Any + + +class Materialization(StrEnum): + CAPTURED = "captured" + REFERENCED = "referenced" + + +class ProjectedSource(BaseModel): + model_config = ConfigDict( + extra="forbid", + ) + name: Annotated[StrictStr, Field(max_length=256, min_length=1)] + definition_version: Annotated[StrictStr, Field(max_length=128, min_length=1)] + materialization: Materialization + description: StrictStr | None = None + source_type: Annotated[StrictStr, Field(max_length=128, min_length=1)] + definition_fingerprint: Annotated[StrictStr, Field(pattern="^sha256:[0-9a-f]{64}$")] + payload: dict[str, Any] + projections: Annotated[list[SourceProjectionValue], Field(max_length=16)] + + +class SubmitSourceObservationRequest(BaseModel): + model_config = ConfigDict( + extra="forbid", + ) + binding: ConnectorBinding + source: ProjectedSource + + +class CommitConnectorCheckpointRequest(BaseModel): + model_config = ConfigDict( + extra="forbid", + ) + binding: ConnectorBinding + expected: Annotated[Any | None, Field(...)] + checkpoint: Annotated[Any | None, Field(...)] + + class Kind(StrEnum): ARTIFACT = "artifact" @@ -1026,6 +1129,14 @@ class CaptureContentSourceResponse(BaseModel): position: Annotated[StrictInt, Field(ge=1)] +class SourceObservationReceipt(BaseModel): + model_config = ConfigDict( + extra="forbid", + ) + source: SourceReference + position: Annotated[StrictInt, Field(ge=1)] + + class HandoffMemoryCitation(BaseModel): model_config = ConfigDict( extra="forbid", diff --git a/src/powercontext/http/_generated/operations.py b/src/powercontext/http/_generated/operations.py index d344d87bd..e8671a1f0 100644 --- a/src/powercontext/http/_generated/operations.py +++ b/src/powercontext/http/_generated/operations.py @@ -16,8 +16,10 @@ Capabilities, CaptureContentSourceRequest, CaptureContentSourceResponse, + CommitConnectorCheckpointRequest, CommitHandoffRequest, CommittedHandoff, + ConnectorCheckpointState, ContinueHandoffRequest, CreateHandoffReportProjectRequest, CreateWorkContractRequest, @@ -31,6 +33,7 @@ GenerateExperienceRequest, GenerateSkillRequest, GetArtifactCandidateRequest, + GetConnectorCheckpointRequest, GetExperienceRequest, GetHandoffReportProjectRequest, GetHandoffReportRequest, @@ -77,6 +80,7 @@ RecordHandoffReportActivityRequest, RecordTaskOutcomeRequest, RegisterHandoffReportWorkstreamRequest, + RegisterSourceDefinitionRequest, RejectArtifactCandidateRequest, RememberMemoryRequest, ResolveExternalSkillRequest, @@ -89,7 +93,10 @@ SearchMemoryRequest, SearchMemoryResponse, SkillArtifact, + SourceDefinitionManifest, + SourceObservationReceipt, StoredHandoffReportActivity, + SubmitSourceObservationRequest, UpdateHandoffReportProjectRequest, UpdateHandoffReportWorkstreamRequest, WorkSourceReceipt, @@ -201,6 +208,79 @@ class Operation(BaseModel, Generic[RequestT, ResponseT]): }, ) +REGISTER_SOURCE_DEFINITION = Operation[RegisterSourceDefinitionRequest, SourceDefinitionManifest]( + method="POST", + path="/v1/source-definitions/register", + operation_id="register_source_definition", + request_type=RegisterSourceDefinitionRequest, + request_location="body", + response_type=SourceDefinitionManifest, + success_status=200, + summary="Register a worker-owned Source Definition manifest", + tags=("source-ingestion",), + responses={ + 200: {"description": "The exact manifest is registered or was already registered identically."}, + 409: {"$ref": "#/components/responses/Conflict"}, + 401: {"$ref": "#/components/responses/Unauthorized"}, + 422: {"$ref": "#/components/responses/InvalidRequest"}, + }, +) + +GET_CONNECTOR_CHECKPOINT = Operation[GetConnectorCheckpointRequest, ConnectorCheckpointState]( + method="POST", + path="/v1/connector-checkpoints/get", + operation_id="get_connector_checkpoint", + request_type=GetConnectorCheckpointRequest, + request_location="body", + response_type=ConnectorCheckpointState, + success_status=200, + summary="Read a Connector binding checkpoint", + tags=("source-ingestion",), + responses={ + 200: {"description": "The current opaque checkpoint, including a normal null initial value."}, + 401: {"$ref": "#/components/responses/Unauthorized"}, + 409: {"$ref": "#/components/responses/Conflict"}, + 422: {"$ref": "#/components/responses/InvalidRequest"}, + }, +) + +SUBMIT_SOURCE_OBSERVATION = Operation[SubmitSourceObservationRequest, SourceObservationReceipt]( + method="POST", + path="/v1/source-observations", + operation_id="submit_source_observation", + request_type=SubmitSourceObservationRequest, + request_location="body", + response_type=SourceObservationReceipt, + success_status=202, + summary="Submit a worker-materialized Source observation", + tags=("source-ingestion",), + responses={ + 202: {"description": "The observation is durably accepted and can be referenced exactly."}, + 401: {"$ref": "#/components/responses/Unauthorized"}, + 409: {"$ref": "#/components/responses/Conflict"}, + 404: {"$ref": "#/components/responses/NotFound"}, + 422: {"$ref": "#/components/responses/InvalidRequest"}, + }, +) + +COMMIT_CONNECTOR_CHECKPOINT = Operation[CommitConnectorCheckpointRequest, ConnectorCheckpointState]( + method="POST", + path="/v1/connector-checkpoints/commit", + operation_id="commit_connector_checkpoint", + request_type=CommitConnectorCheckpointRequest, + request_location="body", + response_type=ConnectorCheckpointState, + success_status=200, + summary="Commit a Connector binding checkpoint", + tags=("source-ingestion",), + responses={ + 200: {"description": "The new opaque checkpoint is durable."}, + 401: {"$ref": "#/components/responses/Unauthorized"}, + 409: {"$ref": "#/components/responses/Conflict"}, + 422: {"$ref": "#/components/responses/InvalidRequest"}, + }, +) + PREPARE_CONTEXT = Operation[PrepareContextRequest, PreparedContext]( method="POST", path="/v1/context/prepare", diff --git a/src/powercontext/http/_generated/schema.py b/src/powercontext/http/_generated/schema.py index 6be425400..7caa0a2a7 100644 --- a/src/powercontext/http/_generated/schema.py +++ b/src/powercontext/http/_generated/schema.py @@ -90,6 +90,110 @@ }, } }, + "/v1/source-definitions/register": { + "post": { + "tags": ["source-ingestion"], + "summary": "Register a worker-owned Source Definition manifest", + "description": "Registers an immutable declarative manifest without loading worker plugin code.", + "operationId": "register_source_definition", + "requestBody": { + "content": { + "application/json": {"schema": {"$ref": "#/components/schemas/RegisterSourceDefinitionRequest"}} + }, + "required": True, + }, + "responses": { + "200": { + "description": "The exact manifest is registered or was already registered identically.", + "content": { + "application/json": {"schema": {"$ref": "#/components/schemas/SourceDefinitionManifest"}} + }, + }, + "409": {"$ref": "#/components/responses/Conflict"}, + "401": {"$ref": "#/components/responses/Unauthorized"}, + "422": {"$ref": "#/components/responses/InvalidRequest"}, + }, + } + }, + "/v1/connector-checkpoints/get": { + "post": { + "tags": ["source-ingestion"], + "summary": "Read a Connector binding checkpoint", + "operationId": "get_connector_checkpoint", + "requestBody": { + "content": { + "application/json": {"schema": {"$ref": "#/components/schemas/GetConnectorCheckpointRequest"}} + }, + "required": True, + }, + "responses": { + "200": { + "description": "The current opaque checkpoint, including a normal null initial value.", + "content": { + "application/json": {"schema": {"$ref": "#/components/schemas/ConnectorCheckpointState"}} + }, + }, + "401": {"$ref": "#/components/responses/Unauthorized"}, + "409": {"$ref": "#/components/responses/Conflict"}, + "422": {"$ref": "#/components/responses/InvalidRequest"}, + }, + } + }, + "/v1/source-observations": { + "post": { + "tags": ["source-ingestion"], + "summary": "Submit a worker-materialized Source observation", + "description": "Validates the observation against " + "its registered manifest and " + "durably appends it before receipt.", + "operationId": "submit_source_observation", + "requestBody": { + "content": { + "application/json": {"schema": {"$ref": "#/components/schemas/SubmitSourceObservationRequest"}} + }, + "required": True, + }, + "responses": { + "202": { + "description": "The observation is durably accepted and can be referenced exactly.", + "content": { + "application/json": {"schema": {"$ref": "#/components/schemas/SourceObservationReceipt"}} + }, + }, + "401": {"$ref": "#/components/responses/Unauthorized"}, + "409": {"$ref": "#/components/responses/Conflict"}, + "404": {"$ref": "#/components/responses/NotFound"}, + "422": {"$ref": "#/components/responses/InvalidRequest"}, + }, + } + }, + "/v1/connector-checkpoints/commit": { + "post": { + "tags": ["source-ingestion"], + "summary": "Commit a Connector binding checkpoint", + "description": "Replaces the checkpoint only when its expected starting value still matches.", + "operationId": "commit_connector_checkpoint", + "requestBody": { + "content": { + "application/json": { + "schema": {"$ref": "#/components/schemas/CommitConnectorCheckpointRequest"} + } + }, + "required": True, + }, + "responses": { + "200": { + "description": "The new opaque checkpoint is durable.", + "content": { + "application/json": {"schema": {"$ref": "#/components/schemas/ConnectorCheckpointState"}} + }, + }, + "401": {"$ref": "#/components/responses/Unauthorized"}, + "409": {"$ref": "#/components/responses/Conflict"}, + "422": {"$ref": "#/components/responses/InvalidRequest"}, + }, + } + }, "/v1/context/prepare": { "post": { "tags": ["context"], @@ -2210,6 +2314,133 @@ "type": "object", "required": ["status", "source", "position"], }, + "SourceProjectionKey": { + "properties": { + "name": {"type": "string", "maxLength": 128, "minLength": 1, "pattern": ".*\\S.*"}, + "version": {"type": "string", "maxLength": 128, "minLength": 1, "pattern": ".*\\S.*"}, + }, + "additionalProperties": False, + "type": "object", + "required": ["name", "version"], + }, + "SourceProjectionManifest": { + "properties": { + "key": {"$ref": "#/components/schemas/SourceProjectionKey"}, + "schema": {"additionalProperties": True, "type": "object"}, + }, + "additionalProperties": False, + "type": "object", + "required": ["key", "schema"], + }, + "SourceDefinitionManifest": { + "properties": { + "name": {"type": "string", "maxLength": 128, "minLength": 1, "pattern": ".*\\S.*"}, + "version": {"type": "string", "maxLength": 128, "minLength": 1, "pattern": ".*\\S.*"}, + "fingerprint": {"type": "string", "pattern": "^sha256:[0-9a-f]{64}$"}, + "source_schema": {"additionalProperties": True, "type": "object"}, + "projections": { + "items": {"$ref": "#/components/schemas/SourceProjectionManifest"}, + "type": "array", + "maxItems": 16, + }, + }, + "additionalProperties": False, + "type": "object", + "required": ["name", "version", "fingerprint", "source_schema", "projections"], + }, + "RegisterSourceDefinitionRequest": { + "properties": {"manifest": {"$ref": "#/components/schemas/SourceDefinitionManifest"}}, + "additionalProperties": False, + "type": "object", + "required": ["manifest"], + }, + "ConnectorBinding": { + "properties": { + "scope_id": {"type": "string", "maxLength": 256, "minLength": 1, "pattern": ".*\\S.*"}, + "binding_id": {"type": "string", "maxLength": 256, "minLength": 1, "pattern": ".*\\S.*"}, + "connector_name": {"type": "string", "maxLength": 128, "minLength": 1, "pattern": ".*\\S.*"}, + "connector_version": {"type": "string", "maxLength": 128, "minLength": 1, "pattern": ".*\\S.*"}, + }, + "additionalProperties": False, + "type": "object", + "required": ["scope_id", "binding_id", "connector_name", "connector_version"], + }, + "GetConnectorCheckpointRequest": { + "properties": {"binding": {"$ref": "#/components/schemas/ConnectorBinding"}}, + "additionalProperties": False, + "type": "object", + "required": ["binding"], + }, + "ConnectorCheckpointState": { + "properties": { + "binding": {"$ref": "#/components/schemas/ConnectorBinding"}, + "checkpoint": {"nullable": True}, + }, + "additionalProperties": False, + "type": "object", + "required": ["binding", "checkpoint"], + }, + "SourceProjectionValue": { + "properties": {"key": {"$ref": "#/components/schemas/SourceProjectionKey"}, "value": {}}, + "additionalProperties": False, + "type": "object", + "required": ["key", "value"], + }, + "ProjectedSource": { + "properties": { + "name": {"type": "string", "maxLength": 256, "minLength": 1}, + "definition_version": {"type": "string", "maxLength": 128, "minLength": 1}, + "materialization": {"type": "string", "enum": ["captured", "referenced"]}, + "description": {"type": "string", "nullable": True}, + "source_type": {"type": "string", "maxLength": 128, "minLength": 1}, + "definition_fingerprint": {"type": "string", "pattern": "^sha256:[0-9a-f]{64}$"}, + "payload": {"additionalProperties": True, "type": "object"}, + "projections": { + "items": {"$ref": "#/components/schemas/SourceProjectionValue"}, + "type": "array", + "maxItems": 16, + }, + }, + "additionalProperties": False, + "type": "object", + "required": [ + "name", + "definition_version", + "materialization", + "source_type", + "definition_fingerprint", + "payload", + "projections", + ], + }, + "SubmitSourceObservationRequest": { + "properties": { + "binding": {"$ref": "#/components/schemas/ConnectorBinding"}, + "source": {"$ref": "#/components/schemas/ProjectedSource"}, + }, + "additionalProperties": False, + "type": "object", + "required": ["binding", "source"], + }, + "SourceObservationReceipt": { + "properties": { + "source": {"$ref": "#/components/schemas/SourceReference"}, + "position": {"type": "integer", "minimum": 1.0}, + }, + "additionalProperties": False, + "type": "object", + "required": ["source", "position"], + }, + "CommitConnectorCheckpointRequest": { + "properties": { + "binding": {"$ref": "#/components/schemas/ConnectorBinding"}, + "expected": {"nullable": True}, + "checkpoint": {"nullable": True}, + }, + "additionalProperties": False, + "type": "object", + "required": ["binding", "expected", "checkpoint"], + }, "CommitHandoffRequest": { "properties": { "scope_id": {"type": "string", "maxLength": 256, "minLength": 1, "pattern": ".*\\S.*"}, diff --git a/src/powercontext/server/app.py b/src/powercontext/server/app.py index 8cfd96edd..59990c673 100644 --- a/src/powercontext/server/app.py +++ b/src/powercontext/server/app.py @@ -134,6 +134,12 @@ from powercontext.builtin.runtime import ( ApproveArtifactCandidateRequest as RuntimeApproveArtifactCandidateRequest, ) +from powercontext.builtin.runtime import ( + CommitConnectorCheckpoint as RuntimeCommitConnectorCheckpoint, +) +from powercontext.builtin.runtime import ( + ConnectorCheckpointState as RuntimeConnectorCheckpointState, +) from powercontext.builtin.runtime import ( GenerateExperienceRequest as RuntimeGenerateExperienceRequest, ) @@ -192,6 +198,9 @@ from powercontext.builtin.runtime import ( StatisticsPeriod as RuntimeStatisticsPeriod, ) +from powercontext.builtin.runtime import ( + SubmitSourceObservation as RuntimeSubmitSourceObservation, +) from powercontext.builtin.work import ( AcknowledgeHandoff as RuntimeAcknowledgeHandoff, ) @@ -209,9 +218,13 @@ from powercontext.builtin.work import WorkSourceReceipt as RuntimeWorkSourceReceipt from powercontext.errors import ( ArtifactNotFoundError, + InvalidConnectorRunError, + InvalidSourceDefinitionError, + InvalidSourceObservationError, PowerContextError, RevisionConflictError, SourceConflictError, + SourceDefinitionNotFoundError, ) from powercontext.http import ( AcknowledgeHandoffRequest, @@ -223,8 +236,10 @@ Capabilities, CaptureContentSourceRequest, CaptureContentSourceResponse, + CommitConnectorCheckpointRequest, CommitHandoffRequest, CommittedHandoff, + ConnectorCheckpointState, ContinueHandoffRequest, CreateHandoffReportProjectRequest, CreateWorkContractRequest, @@ -240,6 +255,7 @@ GenerateExperienceRequest, GenerateSkillRequest, GetArtifactCandidateRequest, + GetConnectorCheckpointRequest, GetExperienceRequest, GetHandoffReportProjectRequest, GetHandoffReportRequest, @@ -286,6 +302,7 @@ RecordHandoffReportActivityRequest, RecordTaskOutcomeRequest, RegisterHandoffReportWorkstreamRequest, + RegisterSourceDefinitionRequest, RejectArtifactCandidateRequest, RememberMemoryRequest, ResolveExternalSkillRequest, @@ -298,7 +315,10 @@ SearchMemoryRequest, SearchMemoryResponse, SkillArtifact, + SourceDefinitionManifest, + SourceObservationReceipt, StoredHandoffReportActivity, + SubmitSourceObservationRequest, UpdateHandoffReportProjectRequest, UpdateHandoffReportWorkstreamRequest, WorkSourceReceipt, @@ -326,6 +346,7 @@ APPROVE_ARTIFACT_CANDIDATE, ATTACH_HANDOFF_REPORT_WORKSPACE, CAPTURE_CONTENT_SOURCE, + COMMIT_CONNECTOR_CHECKPOINT, COMMIT_HANDOFF, CONTINUE_HANDOFF, CREATE_HANDOFF_REPORT_PROJECT, @@ -337,6 +358,7 @@ GENERATE_SKILL, GET_ARTIFACT_CANDIDATE, GET_CAPABILITIES, + GET_CONNECTOR_CHECKPOINT, GET_EXPERIENCE, GET_HANDOFF_REPORT, GET_HANDOFF_REPORT_PROJECT, @@ -365,6 +387,7 @@ RECORD_HANDOFF_REPORT_ACTIVITY, RECORD_TASK_OUTCOME, REGISTER_HANDOFF_REPORT_WORKSTREAM, + REGISTER_SOURCE_DEFINITION, REJECT_ARTIFACT_CANDIDATE, REMEMBER_MEMORY, RESOLVE_EXTERNAL_SKILL, @@ -373,6 +396,7 @@ REVISE_MEMORY_ENTRY, SCAN_EXTERNAL_SKILLS, SEARCH_MEMORY, + SUBMIT_SOURCE_OBSERVATION, UPDATE_HANDOFF_REPORT_PROJECT, UPDATE_HANDOFF_REPORT_WORKSTREAM, Operation, @@ -385,6 +409,8 @@ reset_request_id, ) from powercontext.server.tracing import request_id_from_span +from powercontext.sources import ConnectorBinding as RuntimeConnectorBinding +from powercontext.sources import SourceDefinitionManifest as RuntimeSourceDefinitionManifest if TYPE_CHECKING: from powercontext.server.metrics import ServerMetrics @@ -410,6 +436,16 @@ class _SourceApplication(Protocol): def for_scope(self, scope_id: str, /) -> _ScopedSourceApplication: ... +class _RemoteIngestionApplication(Protocol): + async def register(self, manifest: RuntimeSourceDefinitionManifest, /) -> RuntimeSourceDefinitionManifest: ... + + async def checkpoint(self, binding: RuntimeConnectorBinding, /) -> RuntimeConnectorCheckpointState: ... + + async def submit(self, request: RuntimeSubmitSourceObservation, /) -> SourceReceipt: ... + + async def commit(self, request: RuntimeCommitConnectorCheckpoint, /) -> RuntimeConnectorCheckpointState: ... + + class _ScopedContextApplication(Protocol): async def prepare(self, request: RuntimePrepareContextRequest, /) -> RuntimePreparedContext: ... @@ -538,6 +574,7 @@ def for_scope(self, scope_id: str, /) -> _ScopedStatisticsApplication: ... class ServerApplication(Protocol): sources: _SourceApplication + ingestion: _RemoteIngestionApplication context: _ContextApplication experience: _ExperienceApplication external_skills: _ExternalSkillApplication @@ -653,6 +690,10 @@ async def unexpected_error(request: Request, error: Exception) -> JSONResponse: _add_route(app, DETACH_HANDOFF_REPORT_WORKSPACE, detach_handoff_report_workspace) _add_route(app, GET_HANDOFF_REPORT, get_handoff_report) _add_route(app, CAPTURE_CONTENT_SOURCE, capture_content_source) + _add_route(app, REGISTER_SOURCE_DEFINITION, register_source_definition) + _add_route(app, GET_CONNECTOR_CHECKPOINT, get_connector_checkpoint) + _add_route(app, SUBMIT_SOURCE_OBSERVATION, submit_source_observation) + _add_route(app, COMMIT_CONNECTOR_CHECKPOINT, commit_connector_checkpoint) _add_route(app, FLUSH_MEMORY, flush_memory) _add_route(app, REMEMBER_MEMORY, remember_memory) _add_route(app, SEARCH_MEMORY, search_memory) @@ -995,6 +1036,38 @@ async def capture_content_source( return mapping.capture_response(result) +async def register_source_definition( + request: RegisterSourceDefinitionRequest, + application: Annotated[ServerApplication, Depends(_require_application)], +) -> SourceDefinitionManifest: + result = await application.ingestion.register(mapping.runtime_source_definition_manifest(request.manifest)) + return mapping.source_definition_manifest_response(result) + + +async def get_connector_checkpoint( + request: GetConnectorCheckpointRequest, + application: Annotated[ServerApplication, Depends(_require_application)], +) -> ConnectorCheckpointState: + result = await application.ingestion.checkpoint(mapping.connector_checkpoint_request(request)) + return mapping.connector_checkpoint_response(result) + + +async def submit_source_observation( + request: SubmitSourceObservationRequest, + application: Annotated[ServerApplication, Depends(_require_application)], +) -> SourceObservationReceipt: + result = await application.ingestion.submit(mapping.submit_source_observation_request(request)) + return mapping.source_observation_receipt_response(result) + + +async def commit_connector_checkpoint( + request: CommitConnectorCheckpointRequest, + application: Annotated[ServerApplication, Depends(_require_application)], +) -> ConnectorCheckpointState: + result = await application.ingestion.commit(mapping.commit_connector_checkpoint_request(request)) + return mapping.connector_checkpoint_response(result) + + async def flush_memory( request: FlushMemoryRequest, application: Annotated[ServerApplication, Depends(_require_application)], @@ -1619,12 +1692,13 @@ def _map_report_error(error: Exception) -> tuple[int, str, str, dict[str, Any] | def _map_domain_error(error: Exception) -> tuple[int, str, str, dict[str, Any] | None]: + source_ingestion = _map_source_ingestion_error(error) + if source_ingestion is not None: + return source_ingestion if isinstance(error, ArtifactNotFoundError): return status.HTTP_404_NOT_FOUND, "artifact_not_found", "The requested Artifact was not found.", None if isinstance(error, MemoryEntryNotFoundError): return status.HTTP_404_NOT_FOUND, "memory_not_found", "The requested Memory value was not found.", None - if isinstance(error, SourceConflictError): - return status.HTTP_409_CONFLICT, "source_conflict", "The Source identity has different content.", None if isinstance(error, RevisionConflictError): return status.HTTP_409_CONFLICT, "revision_conflict", "The Memory Revision is stale.", None if isinstance(error, MemoryEntryInactiveError): @@ -1655,6 +1729,23 @@ def _map_domain_error(error: Exception) -> tuple[int, str, str, dict[str, Any] | return status.HTTP_500_INTERNAL_SERVER_ERROR, "internal_error", "The Server failed.", None +def _map_source_ingestion_error(error: Exception) -> tuple[int, str, str, dict[str, Any] | None] | None: + if isinstance(error, SourceConflictError): + return status.HTTP_409_CONFLICT, "source_conflict", "The Source identity has different content.", None + if isinstance(error, InvalidConnectorRunError): + return status.HTTP_409_CONFLICT, "connector_checkpoint_conflict", "The Connector checkpoint is stale.", None + if isinstance(error, SourceDefinitionNotFoundError): + return ( + status.HTTP_404_NOT_FOUND, + "source_definition_not_found", + "The Source Definition is not registered.", + None, + ) + if isinstance(error, (InvalidSourceDefinitionError, InvalidSourceObservationError)): + return status.HTTP_422_UNPROCESSABLE_CONTENT, "invalid_source_ingestion", "Source ingestion is invalid.", None + return None + + def _map_availability_error(error: Exception) -> tuple[int, str, str, dict[str, Any] | None] | None: if isinstance(error, _RuntimeNotReadyError): return status.HTTP_503_SERVICE_UNAVAILABLE, "runtime_not_ready", "The Runtime is not ready.", None diff --git a/src/powercontext/server/mapping.py b/src/powercontext/server/mapping.py index c26efc143..60ce5b1f4 100644 --- a/src/powercontext/server/mapping.py +++ b/src/powercontext/server/mapping.py @@ -75,6 +75,12 @@ from powercontext.builtin.runtime import ( ApproveArtifactCandidateRequest as RuntimeApproveArtifactCandidateRequest, ) +from powercontext.builtin.runtime import ( + CommitConnectorCheckpoint as RuntimeCommitConnectorCheckpoint, +) +from powercontext.builtin.runtime import ( + ConnectorCheckpointState as RuntimeConnectorCheckpointState, +) from powercontext.builtin.runtime import ( GenerateExperienceRequest as RuntimeGenerateExperienceRequest, ) @@ -127,6 +133,9 @@ from powercontext.builtin.runtime import ( Statistics as RuntimeStatistics, ) +from powercontext.builtin.runtime import ( + SubmitSourceObservation as RuntimeSubmitSourceObservation, +) from powercontext.builtin.sources import ExternalSkillImportMode as RuntimeExternalSkillImportMode from powercontext.builtin.work import ( AcknowledgeHandoff as RuntimeAcknowledgeHandoff, @@ -163,7 +172,9 @@ CaptureContentSourceRequest, CaptureContentSourceResponse, CaptureStatus, + CommitConnectorCheckpointRequest, CommittedHandoff, + ConnectorCheckpointState, CreateWorkContractRequest, EntryChange, EntryChangeOperation, @@ -179,6 +190,7 @@ GenerateExperienceRequest, GenerateSkillRequest, GetArtifactCandidateRequest, + GetConnectorCheckpointRequest, GetExperienceRequest, GetMemoryEntryRequest, GetSkillRequest, @@ -224,12 +236,16 @@ SkillArtifact, SkillProposal, SkillValidationItem, + SourceDefinitionManifest, + SourceObservationReceipt, SourceReference, + SubmitSourceObservationRequest, TaskCheck, WorkClaim, WorkSourceKind, WorkSourceReceipt, ) +from powercontext.http import ConnectorBinding as HttpConnectorBinding from powercontext.http import ( HandoffActivation as TransportHandoffActivation, ) @@ -278,6 +294,15 @@ from powercontext.http import ( RememberMemoryRequest as TransportRememberMemoryRequest, ) +from powercontext.sources import ( + ConnectorBinding as RuntimeConnectorBinding, +) +from powercontext.sources import ( + ProjectedSource as RuntimeProjectedSource, +) +from powercontext.sources import ( + SourceDefinitionManifest as RuntimeSourceDefinitionManifest, +) from powercontext.sources import SourceRef @@ -421,6 +446,62 @@ def capture_response(value: SourceReceipt) -> CaptureContentSourceResponse: ) +def runtime_source_definition_manifest(value: SourceDefinitionManifest) -> RuntimeSourceDefinitionManifest: + try: + return RuntimeSourceDefinitionManifest.model_validate(value.model_dump(mode="json", by_alias=True)) + except ValidationError as error: + raise InvalidRuntimeRequestError("source-definition-manifest") from error + + +def source_definition_manifest_response(value: RuntimeSourceDefinitionManifest) -> SourceDefinitionManifest: + return SourceDefinitionManifest.model_validate(value.model_dump(mode="json", by_alias=True)) + + +def runtime_connector_binding(value: HttpConnectorBinding) -> RuntimeConnectorBinding: + try: + return RuntimeConnectorBinding.model_validate(value.model_dump(mode="json")) + except ValidationError as error: + raise InvalidRuntimeRequestError("connector-binding") from error + + +def connector_checkpoint_request(value: GetConnectorCheckpointRequest) -> RuntimeConnectorBinding: + return runtime_connector_binding(value.binding) + + +def submit_source_observation_request(value: SubmitSourceObservationRequest) -> RuntimeSubmitSourceObservation: + try: + return RuntimeSubmitSourceObservation( + binding=runtime_connector_binding(value.binding), + source=RuntimeProjectedSource.model_validate(value.source.model_dump(mode="json")), + ) + except ValidationError as error: + raise InvalidRuntimeRequestError("source-observation") from error + + +def commit_connector_checkpoint_request( + value: CommitConnectorCheckpointRequest, +) -> RuntimeCommitConnectorCheckpoint: + try: + return RuntimeCommitConnectorCheckpoint( + binding=runtime_connector_binding(value.binding), + expected=value.expected, + checkpoint=value.checkpoint, + ) + except ValidationError as error: + raise InvalidRuntimeRequestError("connector-checkpoint") from error + + +def connector_checkpoint_response(value: RuntimeConnectorCheckpointState) -> ConnectorCheckpointState: + return ConnectorCheckpointState.model_validate(value.model_dump(mode="json")) + + +def source_observation_receipt_response(value: SourceReceipt) -> SourceObservationReceipt: + return SourceObservationReceipt( + source=source_reference(value.source_ref), + position=value.sequence, + ) + + def statistics_response(value: RuntimeStatistics) -> ScopedStats: return ScopedStats.model_validate(value.model_dump(mode="json")) diff --git a/src/powercontext/sources/__init__.py b/src/powercontext/sources/__init__.py index fed1522b2..fea55564a 100644 --- a/src/powercontext/sources/__init__.py +++ b/src/powercontext/sources/__init__.py @@ -29,6 +29,7 @@ ConnectorSourceSink, ConnectorSubmissionResult, ConnectorSubmissionStatus, + validate_connector, ) from powercontext.sources.definitions import ( AdapterSourceDefinition, @@ -37,9 +38,19 @@ SourceProjection, ) from powercontext.sources.models import Source, SourceMaterialization, SourceProjectionKey, SourceRef +from powercontext.sources.observations import ( + ProjectedSource, + SourceDefinitionManifest, + SourceProjectionManifest, + SourceProjectionValue, + manifest_for_definition, + project_source_for_transport, +) +from powercontext.sources.projections import TEXT_EVIDENCE_PROJECTION_KEY, TextEvidence from powercontext.sources.protocols import SourceCatalogBackend, SourceStore __all__ = [ + "TEXT_EVIDENCE_PROJECTION_KEY", "AdapterSourceDefinition", "CatalogConnectorSourceSink", "Connector", @@ -55,15 +66,23 @@ "ConnectorSourceSink", "ConnectorSubmissionResult", "ConnectorSubmissionStatus", + "ProjectedSource", "Source", "SourceAdapter", "SourceCatalog", "SourceCatalogBackend", "SourceDefinition", + "SourceDefinitionManifest", "SourceDefinitionRegistry", "SourceMaterialization", "SourceProjection", "SourceProjectionKey", + "SourceProjectionManifest", + "SourceProjectionValue", "SourceRef", "SourceStore", + "TextEvidence", + "manifest_for_definition", + "project_source_for_transport", + "validate_connector", ] diff --git a/src/powercontext/sources/catalog.py b/src/powercontext/sources/catalog.py index 182446f18..52b0717f6 100644 --- a/src/powercontext/sources/catalog.py +++ b/src/powercontext/sources/catalog.py @@ -25,6 +25,7 @@ from powercontext.sources.adapters import SourceAdapter from powercontext.sources.definitions import SourceDefinitionRegistry from powercontext.sources.models import Source, SourceProjectionKey, SourceRef +from powercontext.sources.observations import ProjectedSource from powercontext.sources.protocols import SourceCatalogBackend _AnySourceAdapter = SourceAdapter[Any, Any, Any] @@ -61,6 +62,8 @@ async def get(self, source: Source, /) -> Source: return stored def as_ref(self, source: Source, /) -> SourceRef: + if isinstance(source, ProjectedSource): + return SourceRef(source_type=source.source_type, source_id=source.name) definition = self._registry.definition_for_source(source) return SourceRef(source_type=definition.name, source_id=source.name) @@ -68,14 +71,20 @@ async def resolve(self, value: object, /) -> Source: return await self._registry.resolve(value) async def read(self, source: Source, /) -> object: + if isinstance(source, ProjectedSource): + return source.payload return await self._registry.read(source) def projection_keys(self, source: Source, /) -> tuple[SourceProjectionKey, ...]: """Return the exact named projection capabilities advertised for ``source``.""" + if isinstance(source, ProjectedSource): + return tuple(projection.key for projection in source.projections) return self._registry.projection_keys(source) def project(self, source: Source, key: SourceProjectionKey, /) -> JsonValue: """Evaluate one named projection against an exact Source value.""" + if isinstance(source, ProjectedSource): + return source.projection(key) return self._registry.project(source, key) diff --git a/src/powercontext/sources/connectors.py b/src/powercontext/sources/connectors.py index c3973fa96..b5ca098fa 100644 --- a/src/powercontext/sources/connectors.py +++ b/src/powercontext/sources/connectors.py @@ -280,7 +280,7 @@ def __init__(self, *, sink: ConnectorSourceSink, checkpoints: ConnectorCheckpoin self._checkpoints = checkpoints async def run(self, connector: Connector, binding: ConnectorBinding, /) -> ConnectorRunResult: - source_definitions, capabilities = _validate_connector(connector, binding) + source_definitions, capabilities = validate_connector(connector, binding) previous = await self._checkpoints.load(binding) if previous is not None and ConnectorCapability.CHECKPOINT_RESUME not in capabilities: raise InvalidConnectorRunError( @@ -353,10 +353,12 @@ async def submit( return ConnectorSubmissionResult(status=ConnectorSubmissionStatus.ACCEPTED, source_ref=stored_ref) -def _validate_connector( +def validate_connector( connector: Connector, binding: ConnectorBinding, ) -> tuple[frozenset[str], frozenset[ConnectorCapability]]: + """Validate one Connector declaration against the binding it will execute.""" + name = getattr(connector, "name", None) version = getattr(connector, "version", None) if name != binding.connector_name: diff --git a/src/powercontext/sources/observations.py b/src/powercontext/sources/observations.py new file mode 100644 index 000000000..60a54e427 --- /dev/null +++ b/src/powercontext/sources/observations.py @@ -0,0 +1,228 @@ +# Copyright (c) 2026 OceanBase. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Worker-owned Source Definition manifests and projected observations.""" + +from __future__ import annotations + +import hashlib +from typing import Any + +import rfc8785 +from pydantic import BaseModel, ConfigDict, Field, JsonValue, TypeAdapter, field_validator, model_validator + +from powercontext.errors import InvalidSourceDefinitionError, InvalidSourceProjectionError +from powercontext.limits import MAX_SOURCE_TYPE_LENGTH +from powercontext.sources.definitions import SourceDefinition, SourceDefinitionRegistry +from powercontext.sources.models import Source, SourceProjectionKey + +_JSON_VALUE = TypeAdapter(JsonValue) + + +class SourceProjectionManifest(BaseModel): + """Declarative schema for one worker-computed named projection.""" + + model_config = ConfigDict(extra="forbid", frozen=True) + + key: SourceProjectionKey + schema_: dict[str, JsonValue] = Field(alias="schema") + + +class SourceDefinitionManifest(BaseModel): + """Immutable declarative identity registered by a remote worker.""" + + model_config = ConfigDict(extra="forbid", frozen=True) + + name: str + version: str + fingerprint: str + source_schema: dict[str, JsonValue] + projections: tuple[SourceProjectionManifest, ...] = () + + @field_validator("name", "version") + @classmethod + def validate_identity(cls, value: str) -> str: + if not value or value.strip() != value or len(value) > MAX_SOURCE_TYPE_LENGTH: + raise ValueError("manifest identity must be a bounded non-empty trimmed string") # noqa: TRY003 + return value + + @field_validator("projections") + @classmethod + def validate_projection_limit( + cls, + value: tuple[SourceProjectionManifest, ...], + ) -> tuple[SourceProjectionManifest, ...]: + if len(value) > 16: + raise ValueError("manifest must not declare more than 16 projections") # noqa: TRY003 + return value + + @field_validator("fingerprint") + @classmethod + def validate_fingerprint_shape(cls, value: str) -> str: + if not value.startswith("sha256:") or len(value) != 71: + raise ValueError("manifest fingerprint must use sha256:") # noqa: TRY003 + try: + int(value.removeprefix("sha256:"), 16) + except ValueError as error: + raise ValueError("manifest fingerprint must contain lowercase hexadecimal") from error # noqa: TRY003 + if value != value.lower(): + raise ValueError("manifest fingerprint must contain lowercase hexadecimal") # noqa: TRY003 + return value + + @model_validator(mode="after") + def validate_manifest(self) -> SourceDefinitionManifest: + keys = tuple(projection.key for projection in self.projections) + if len(set(keys)) != len(keys): + raise ValueError("manifest projection keys must be unique") # noqa: TRY003 + expected = source_definition_fingerprint( + name=self.name, + version=self.version, + source_schema=self.source_schema, + projections=self.projections, + ) + if self.fingerprint != expected: + raise ValueError("manifest fingerprint does not match its declaration") # noqa: TRY003 + return self + + +class SourceProjectionValue(BaseModel): + """One named projection computed by the worker that owns the Definition.""" + + model_config = ConfigDict(extra="forbid", frozen=True) + + key: SourceProjectionKey + value: JsonValue + + +class ProjectedSource(Source): + """Canonical worker-materialized Source stored without loading plugin code.""" + + source_type: str + definition_fingerprint: str + payload: dict[str, JsonValue] + projections: tuple[SourceProjectionValue, ...] = () + + @field_validator("source_type") + @classmethod + def validate_source_type(cls, value: str) -> str: + if not value or value.strip() != value or len(value) > MAX_SOURCE_TYPE_LENGTH: + raise ValueError("source_type must be a bounded non-empty trimmed string") # noqa: TRY003 + return value + + @model_validator(mode="after") + def validate_envelope_identity(self) -> ProjectedSource: + expected = { + "name": self.name, + "definition_version": self.definition_version, + "materialization": self.materialization.value, + "description": self.description, + } + for field, value in expected.items(): + if self.payload.get(field) != value: + raise ValueError(f"projected Source payload {field} does not match its envelope") # noqa: TRY003 + keys = tuple(projection.key for projection in self.projections) + if len(set(keys)) != len(keys): + raise ValueError("projected Source projection keys must be unique") # noqa: TRY003 + return self + + def projection(self, key: SourceProjectionKey, /) -> JsonValue: + for projection in self.projections: + if projection.key == key: + return projection.value + raise InvalidSourceProjectionError(key.name, "key", "was not supplied by the worker") + + +def manifest_for_definition(definition: SourceDefinition[Any, Any, Any], /) -> SourceDefinitionManifest: + """Build the immutable declaration transported by a remote worker.""" + + source_schema = _json_object(definition.source_class.model_json_schema()) + projections = tuple( + SourceProjectionManifest( + key=SourceProjectionKey(name=projection.name, version=projection.version), + schema=projection.output_class.model_json_schema(), + ) + for projection in definition.projections + ) + return SourceDefinitionManifest( + name=definition.name, + version=definition.version, + fingerprint=source_definition_fingerprint( + name=definition.name, + version=definition.version, + source_schema=source_schema, + projections=projections, + ), + source_schema=source_schema, + projections=projections, + ) + + +def project_source_for_transport( + registry: SourceDefinitionRegistry, + source: Source, + /, +) -> ProjectedSource: + """Execute one worker-owned Definition and serialize its durable result.""" + + definition = registry.definition_for_source(source) + manifest = manifest_for_definition(definition) + payload = _json_object(source.model_dump(mode="json")) + projections = tuple( + SourceProjectionValue(key=key, value=registry.project(source, key)) for key in registry.projection_keys(source) + ) + return ProjectedSource( + name=source.name, + definition_version=source.definition_version, + materialization=source.materialization, + description=source.description, + source_type=definition.name, + definition_fingerprint=manifest.fingerprint, + payload=payload, + projections=projections, + ) + + +def source_definition_fingerprint( + *, + name: str, + version: str, + source_schema: dict[str, JsonValue], + projections: tuple[SourceProjectionManifest, ...], +) -> str: + declaration = { + "name": name, + "version": version, + "source_schema": source_schema, + "projections": [projection.model_dump(mode="json", by_alias=True) for projection in projections], + } + encoded = rfc8785.dumps(_JSON_VALUE.validate_python(declaration)) + return f"sha256:{hashlib.sha256(encoded).hexdigest()}" + + +def _json_object(value: object) -> dict[str, JsonValue]: + validated = _JSON_VALUE.validate_python(value) + if not isinstance(validated, dict): + raise InvalidSourceDefinitionError(type(value), "schema", "must be a JSON object") + return validated + + +__all__ = [ + "ProjectedSource", + "SourceDefinitionManifest", + "SourceProjectionManifest", + "SourceProjectionValue", + "manifest_for_definition", + "project_source_for_transport", + "source_definition_fingerprint", +] diff --git a/src/powercontext/builtin/sources/projections.py b/src/powercontext/sources/projections.py similarity index 84% rename from src/powercontext/builtin/sources/projections.py rename to src/powercontext/sources/projections.py index 2ee6657b6..3864a7534 100644 --- a/src/powercontext/builtin/sources/projections.py +++ b/src/powercontext/sources/projections.py @@ -12,13 +12,13 @@ # See the License for the specific language governing permissions and # limitations under the License. -"""Built-in named projection schemas shared by Source Definitions.""" +"""Standard named projection schemas understood without Source plugin code.""" from pydantic import BaseModel, Field, JsonValue -from powercontext.sources import SourceProjectionKey +from powercontext.sources.models import SourceProjectionKey -TEXT_EVIDENCE_PROJECTION_KEY = SourceProjectionKey(name="powercontext.builtin.text-evidence", version="1") +TEXT_EVIDENCE_PROJECTION_KEY = SourceProjectionKey(name="powercontext.text-evidence", version="1") class TextEvidence(BaseModel): diff --git a/tests/builtin/connectors/test_opendal.py b/tests/builtin/connectors/test_opendal.py deleted file mode 100644 index 033fa547f..000000000 --- a/tests/builtin/connectors/test_opendal.py +++ /dev/null @@ -1,234 +0,0 @@ -# Copyright (c) 2026 OceanBase. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -from __future__ import annotations - -import asyncio - -import pytest - -from powercontext.builtin.artifacts.memory import MemoryCandidateRequest, MemoryEntryInput -from powercontext.builtin.connectors import ( - OPENDAL_TEXT_FILE_CONNECTOR_NAME, - OpenDALTextFileConnector, -) -from powercontext.builtin.persistence.sqlite import SQLiteConfig -from powercontext.builtin.runtime import BuiltinConfig, open_builtin_contexts -from powercontext.builtin.sources import ( - TEXT_EVIDENCE_PROJECTION_KEY, - TextFileSnapshotSource, -) -from powercontext.sources import ( - ConnectorBinding, - ConnectorCapability, - ConnectorRunStatus, - ConnectorSubmissionStatus, -) - - -def _binding() -> ConnectorBinding: - return ConnectorBinding( - scope_id="project-a", - binding_id="documents-a", - connector_name=OPENDAL_TEXT_FILE_CONNECTOR_NAME, - connector_version="1", - ) - - -class MemoryFileSystem: - def __init__(self) -> None: - self.files: dict[str, bytes] = {} - - def pipe_file(self, path: str, content: bytes) -> None: - self.files[path] = content - - def find(self, path: str, *, detail: bool) -> dict[str, dict[str, object]]: - assert detail - prefix = f"{path.rstrip('/')}/" if path else "" - return { - name: {"name": name, "size": len(content), "type": "file"} - for name, content in self.files.items() - if not prefix or name.startswith(prefix) - } - - def cat_file(self, path: str) -> bytes: - return self.files[path] - - -def _filesystem() -> MemoryFileSystem: - return MemoryFileSystem() - - -class TextFileCandidatePipeline: - async def extract(self, request: MemoryCandidateRequest, /) -> tuple[MemoryEntryInput, ...]: - return tuple( - MemoryEntryInput( - kind="document", - text=source.content, - sources=(source,), - ) - for source in request.sources - if isinstance(source, TextFileSnapshotSource) - ) - - -def test_opendal_connector_persists_incremental_snapshots_across_runtime_restart(tmp_path) -> None: - async def scenario() -> None: - filesystem = _filesystem() - filesystem.pipe_file("docs/readme.md", b"First value") - filesystem.pipe_file("docs/nested/note.txt", b"Nested value") - filesystem.pipe_file("docs/image.bin", b"\x00\x01") - connector = OpenDALTextFileConnector( - filesystem, - source_namespace="workspace-a", - root="docs", - ) - config = BuiltinConfig(database=SQLiteConfig(url=f"sqlite+aiosqlite:///{tmp_path / 'powercontext.db'}")) - - async with open_builtin_contexts(config) as contexts: - first = await contexts.run_connector(connector, _binding()) - context = await contexts.get("project-a") - first_sources = await context.sources.list() - - assert first.status is ConnectorRunStatus.COMPLETE - assert [item.item_id for item in first.items] == ["nested/note.txt", "readme.md"] - assert all(item.status is ConnectorSubmissionStatus.ACCEPTED for item in first.items) - assert len(first_sources) == 2 - readme = next( - source - for source in first_sources - if isinstance(source, TextFileSnapshotSource) and source.path == "readme.md" - ) - assert context.sources.catalog.project(readme, TEXT_EVIDENCE_PROJECTION_KEY) == { - "source_type": "text-file-snapshot", - "source_id": readme.name, - "content": "First value", - "metadata": { - "namespace": "workspace-a", - "path": "readme.md", - "media_type": "text/markdown", - "encoding": "utf-8", - "content_digest": readme.content_digest, - "size": 11, - }, - } - - async with open_builtin_contexts(config) as contexts: - unchanged = await contexts.run_connector(connector, _binding()) - assert unchanged.previous_checkpoint == first.committed_checkpoint - assert unchanged.items == () - - filesystem.pipe_file("docs/readme.md", b"Second value") - changed = await contexts.run_connector(connector, _binding()) - context = await contexts.get("project-a") - sources = await context.sources.list() - - assert [item.item_id for item in changed.items] == ["readme.md"] - assert len(sources) == 3 - readme_snapshots = [ - source - for source in sources - if isinstance(source, TextFileSnapshotSource) and source.path == "readme.md" - ] - assert {source.content for source in readme_snapshots} == {"First value", "Second value"} - assert len({source.name for source in readme_snapshots}) == 2 - - asyncio.run(scenario()) - - -def test_opendal_connector_keeps_checkpoint_before_a_rejected_item() -> None: - async def scenario() -> None: - filesystem = _filesystem() - filesystem.pipe_file("good.md", b"Good value") - filesystem.pipe_file("invalid.txt", b"\xff") - connector = OpenDALTextFileConnector(filesystem, source_namespace="workspace-a") - - async with open_builtin_contexts(BuiltinConfig(database=SQLiteConfig())) as contexts: - rejected = await contexts.run_connector(connector, _binding()) - context = await contexts.get("project-a") - - assert rejected.status is ConnectorRunStatus.INCOMPLETE - assert rejected.committed_checkpoint is None - assert [(item.item_id, item.status) for item in rejected.items] == [ - ("good.md", ConnectorSubmissionStatus.ACCEPTED), - ("invalid.txt", ConnectorSubmissionStatus.REJECTED), - ] - assert len(await context.sources.list()) == 1 - - filesystem.pipe_file("invalid.txt", b"Recovered value") - recovered = await contexts.run_connector(connector, _binding()) - - assert recovered.status is ConnectorRunStatus.COMPLETE - assert recovered.committed_checkpoint is not None - assert len(await context.sources.list()) == 2 - - asyncio.run(scenario()) - - -def test_opendal_connector_sources_complete_the_memory_ingestion_loop() -> None: - async def scenario() -> None: - filesystem = _filesystem() - filesystem.pipe_file("decision.md", b"Use exact snapshot references.") - connector = OpenDALTextFileConnector(filesystem, source_namespace="workspace-a") - - async with open_builtin_contexts( - BuiltinConfig(database=SQLiteConfig()), - candidate_pipeline=TextFileCandidatePipeline(), - ) as contexts: - connector_result = await contexts.run_connector(connector, _binding()) - context = await contexts.get("project-a") - flush_result = await context.triggers.flush(limit=10) - memory = await context.artifacts.memory.head("memory") - entries = await context.artifacts.memory.entries(memory) - - assert flush_result.source_count == 1 - assert len(entries) == 1 - assert entries[0].text == "Use exact snapshot references." - assert entries[0].sources == (connector_result.items[0].source_ref,) - - asyncio.run(scenario()) - - -def test_opendal_connector_does_not_claim_authoritative_deletion() -> None: - connector = OpenDALTextFileConnector(_filesystem(), source_namespace="workspace-a") - - assert ConnectorCapability.CHECKPOINT_RESUME in connector.capabilities - assert ConnectorCapability.AUTHORITATIVE_DELETION not in connector.capabilities - assert ConnectorCapability.CHANGE_FEED not in connector.capabilities - - -def test_opendal_connector_reads_the_real_opendalfs_memory_backend() -> None: - opendalfs = pytest.importorskip("opendalfs") - - async def scenario() -> None: - filesystem = opendalfs.OpendalFileSystem( - scheme="memory", - asynchronous=False, - skip_instance_cache=True, - ) - filesystem.pipe_file("docs/readme.md", b"OpenDAL value") - connector = OpenDALTextFileConnector( - filesystem, - source_namespace="opendal-memory", - root="docs", - ) - - async with open_builtin_contexts(BuiltinConfig(database=SQLiteConfig())) as contexts: - result = await contexts.run_connector(connector, _binding()) - - assert result.status is ConnectorRunStatus.COMPLETE - assert result.items[0].item_id == "readme.md" - assert result.items[0].status is ConnectorSubmissionStatus.ACCEPTED - - asyncio.run(scenario()) diff --git a/tests/integrations/test_opendal_connector.py b/tests/integrations/test_opendal_connector.py new file mode 100644 index 000000000..57105ae8e --- /dev/null +++ b/tests/integrations/test_opendal_connector.py @@ -0,0 +1,296 @@ +# Copyright (c) 2026 OceanBase. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from __future__ import annotations + +import asyncio +from pathlib import Path + +import httpx +import pytest +from fastapi import FastAPI +from powercontext_connector_opendal import ( + OPENDAL_TEXT_FILE_CONNECTOR_NAME, + TEXT_FILE_SNAPSHOT_SOURCE_DEFINITION, + OpenDALTextFileConnector, + TextFileSnapshotCapture, +) + +from powercontext.builtin.artifacts.memory import MemoryCandidateRequest, MemoryEntryInput +from powercontext.builtin.persistence.sqlite import SQLiteConfig +from powercontext.client import PowerContextClient, RemoteConnectorWorker, ServerResponseError +from powercontext.http import ( + CommitConnectorCheckpointRequest, + FlushMemoryRequest, + ListMemoryEntriesRequest, + ListMemoryEntriesResponse, + RegisterSourceDefinitionRequest, + SubmitSourceObservationRequest, +) +from powercontext.http import ( + ConnectorBinding as HttpConnectorBinding, +) +from powercontext.http import ( + ProjectedSource as HttpProjectedSource, +) +from powercontext.http import ( + SourceDefinitionManifest as HttpSourceDefinitionManifest, +) +from powercontext.server.factory import create_server_app +from powercontext.server.settings import McpConfig, ServerSettings +from powercontext.sources import ( + TEXT_EVIDENCE_PROJECTION_KEY, + ConnectorBinding, + ConnectorCapability, + ConnectorRunResult, + ConnectorRunStatus, + ConnectorSubmissionStatus, + ProjectedSource, + SourceDefinitionRegistry, + TextEvidence, + manifest_for_definition, + project_source_for_transport, +) + + +class MemoryFileSystem: + def __init__(self) -> None: + self.files: dict[str, bytes] = {} + + def pipe_file(self, path: str, content: bytes) -> None: + self.files[path] = content + + def find(self, path: str, *, detail: bool) -> dict[str, dict[str, object]]: + assert detail + prefix = f"{path.rstrip('/')}/" if path else "" + return { + name: {"name": name, "size": len(content), "type": "file"} + for name, content in self.files.items() + if not prefix or name.startswith(prefix) + } + + def cat_file(self, path: str) -> bytes: + return self.files[path] + + +class TextEvidenceCandidatePipeline: + async def extract(self, request: MemoryCandidateRequest, /) -> tuple[MemoryEntryInput, ...]: + entries: list[MemoryEntryInput] = [] + for source in request.sources: + if not isinstance(source, ProjectedSource): + continue + evidence = TextEvidence.model_validate(source.projection(TEXT_EVIDENCE_PROJECTION_KEY)) + entries.append(MemoryEntryInput(kind="document", text=evidence.content, sources=(source,))) + return tuple(entries) + + +def _binding() -> ConnectorBinding: + return ConnectorBinding( + scope_id="project-a", + binding_id="documents-a", + connector_name=OPENDAL_TEXT_FILE_CONNECTOR_NAME, + connector_version="1", + ) + + +def _app(database: Path, *, memory: bool = False) -> FastAPI: + return create_server_app( + settings=ServerSettings( + database=SQLiteConfig(url=f"sqlite+aiosqlite:///{database}"), + mcp=McpConfig(enabled=False), + ), + candidate_pipeline=TextEvidenceCandidatePipeline() if memory else None, + ) + + +async def _run( + app: FastAPI, + connector: OpenDALTextFileConnector, + *, + flush_memory: bool = False, +) -> tuple[ConnectorRunResult, ListMemoryEntriesResponse | None]: + async with ( + app.router.lifespan_context(app), + httpx.AsyncClient( + transport=httpx.ASGITransport(app=app), + base_url="http://testserver", + ) as transport, + ): + client = PowerContextClient("http://testserver", http_client=transport, trust_transport_security=True) + worker = RemoteConnectorWorker( + client=client, + registry=SourceDefinitionRegistry((TEXT_FILE_SNAPSHOT_SOURCE_DEFINITION,)), + ) + result = await worker.run(connector, _binding()) + memory = None + if flush_memory: + await client.flush_memory(FlushMemoryRequest(scope_id="project-a")) + memory = await client.list_memory_entries(ListMemoryEntriesRequest(scope_id="project-a")) + return result, memory + + +def test_remote_opendal_worker_persists_incremental_checkpoint_across_server_restart(tmp_path: Path) -> None: + async def scenario() -> None: + filesystem = MemoryFileSystem() + filesystem.pipe_file("docs/readme.md", b"First value") + filesystem.pipe_file("docs/nested/note.txt", b"Nested value") + filesystem.pipe_file("docs/image.bin", b"\x00\x01") + connector = OpenDALTextFileConnector(filesystem, source_namespace="workspace-a", root="docs") + database = tmp_path / "powercontext.db" + + first, _ = await _run(_app(database), connector) + unchanged, _ = await _run(_app(database), connector) + filesystem.pipe_file("docs/readme.md", b"Second value") + changed, _ = await _run(_app(database), connector) + + assert first.status is ConnectorRunStatus.COMPLETE + assert [item.item_id for item in first.items] == ["nested/note.txt", "readme.md"] + assert all(item.status is ConnectorSubmissionStatus.ACCEPTED for item in first.items) + assert unchanged.previous_checkpoint == first.committed_checkpoint + assert unchanged.items == () + assert [item.item_id for item in changed.items] == ["readme.md"] + assert changed.previous_checkpoint == first.committed_checkpoint + assert changed.committed_checkpoint != first.committed_checkpoint + + asyncio.run(scenario()) + + +def test_remote_opendal_worker_keeps_checkpoint_before_a_rejected_item(tmp_path: Path) -> None: + async def scenario() -> None: + filesystem = MemoryFileSystem() + filesystem.pipe_file("good.md", b"Good value") + filesystem.pipe_file("invalid.txt", b"\xff") + connector = OpenDALTextFileConnector(filesystem, source_namespace="workspace-a") + database = tmp_path / "powercontext.db" + + rejected, _ = await _run(_app(database), connector) + filesystem.pipe_file("invalid.txt", b"Recovered value") + recovered, _ = await _run(_app(database), connector) + + assert rejected.status is ConnectorRunStatus.INCOMPLETE + assert rejected.committed_checkpoint is None + assert [(item.item_id, item.status) for item in rejected.items] == [ + ("good.md", ConnectorSubmissionStatus.ACCEPTED), + ("invalid.txt", ConnectorSubmissionStatus.REJECTED), + ] + assert recovered.status is ConnectorRunStatus.COMPLETE + assert recovered.previous_checkpoint is None + assert recovered.committed_checkpoint is not None + + asyncio.run(scenario()) + + +def test_remote_opendal_worker_completes_the_source_to_memory_loop(tmp_path: Path) -> None: + async def scenario() -> None: + filesystem = MemoryFileSystem() + filesystem.pipe_file("decision.md", b"Use exact snapshot references.") + connector = OpenDALTextFileConnector(filesystem, source_namespace="workspace-a") + + result, memory = await _run(_app(tmp_path / "powercontext.db", memory=True), connector, flush_memory=True) + + assert result.status is ConnectorRunStatus.COMPLETE + assert memory is not None + assert [entry.text for entry in memory.entries] == ["Use exact snapshot references."] + assert memory.entries[0].source_refs[0].name == "text-file-snapshot" + + asyncio.run(scenario()) + + +def test_opendal_connector_declares_only_enforced_capabilities() -> None: + connector = OpenDALTextFileConnector(MemoryFileSystem(), source_namespace="workspace-a") + + assert ConnectorCapability.CHECKPOINT_RESUME in connector.capabilities + assert ConnectorCapability.AUTHORITATIVE_DELETION not in connector.capabilities + assert ConnectorCapability.CHANGE_FEED not in connector.capabilities + + +def test_opendal_connector_reads_the_real_opendalfs_memory_backend(tmp_path: Path) -> None: + opendalfs = pytest.importorskip("opendalfs") + + async def scenario() -> None: + filesystem = opendalfs.OpendalFileSystem( + scheme="memory", + asynchronous=False, + skip_instance_cache=True, + ) + filesystem.pipe_file("docs/readme.md", b"OpenDAL value") + connector = OpenDALTextFileConnector( + filesystem, + source_namespace="opendal-memory", + root="docs", + ) + + result, _ = await _run(_app(tmp_path / "powercontext.db"), connector) + + assert result.status is ConnectorRunStatus.COMPLETE + assert result.items[0].item_id == "readme.md" + assert result.items[0].status is ConnectorSubmissionStatus.ACCEPTED + + asyncio.run(scenario()) + + +def test_remote_ingestion_rejects_invalid_projection_and_stale_checkpoint(tmp_path: Path) -> None: + async def scenario() -> None: + app = _app(tmp_path / "powercontext.db") + registry = SourceDefinitionRegistry((TEXT_FILE_SNAPSHOT_SOURCE_DEFINITION,)) + manifest = manifest_for_definition(TEXT_FILE_SNAPSHOT_SOURCE_DEFINITION) + source = await registry.resolve( + TextFileSnapshotCapture(namespace="workspace-a", path="decision.md", content="Keep worker authority.") + ) + projected = project_source_for_transport(registry, source) + malformed = projected.model_copy(update={"projections": ()}) + binding = HttpConnectorBinding.model_validate(_binding().model_dump(mode="json")) + + async with ( + app.router.lifespan_context(app), + httpx.AsyncClient( + transport=httpx.ASGITransport(app=app), + base_url="http://testserver", + ) as transport, + ): + client = PowerContextClient("http://testserver", http_client=transport, trust_transport_security=True) + with pytest.raises(ServerResponseError) as missing: + await client.submit_source_observation( + SubmitSourceObservationRequest( + binding=binding, + source=HttpProjectedSource.model_validate(projected.model_dump(mode="json")), + ) + ) + await client.register_source_definition( + RegisterSourceDefinitionRequest( + manifest=HttpSourceDefinitionManifest.model_validate( + manifest.model_dump(mode="json", by_alias=True) + ) + ) + ) + with pytest.raises(ServerResponseError) as invalid: + await client.submit_source_observation( + SubmitSourceObservationRequest( + binding=binding, + source=HttpProjectedSource.model_validate(malformed.model_dump(mode="json")), + ) + ) + await client.commit_connector_checkpoint( + CommitConnectorCheckpointRequest(binding=binding, expected=None, checkpoint={"cursor": 1}) + ) + with pytest.raises(ServerResponseError) as stale: + await client.commit_connector_checkpoint( + CommitConnectorCheckpointRequest(binding=binding, expected=None, checkpoint={"cursor": 2}) + ) + + assert (missing.value.status_code, missing.value.code) == (404, "source_definition_not_found") + assert (invalid.value.status_code, invalid.value.code) == (422, "invalid_source_ingestion") + assert (stale.value.status_code, stale.value.code) == (409, "connector_checkpoint_conflict") + + asyncio.run(scenario()) diff --git a/tests/test_source_observations.py b/tests/test_source_observations.py new file mode 100644 index 000000000..635bf58b7 --- /dev/null +++ b/tests/test_source_observations.py @@ -0,0 +1,71 @@ +# Copyright (c) 2026 OceanBase. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from __future__ import annotations + +import asyncio + +import pytest +from pydantic import ValidationError + +from powercontext.builtin.sources import CONTENT_SOURCE_DEFINITION, ContentCapture +from powercontext.sources import ( + TEXT_EVIDENCE_PROJECTION_KEY, + ProjectedSource, + Source, + SourceCatalog, + SourceDefinitionManifest, + SourceDefinitionRegistry, + TextEvidence, + manifest_for_definition, + project_source_for_transport, +) + + +class EmptySourceBackend: + async def list(self) -> tuple[Source, ...]: + return () + + async def get(self, source: Source, /) -> Source: + raise AssertionError(source) + + +def test_definition_manifest_has_a_stable_content_addressed_identity() -> None: + first = manifest_for_definition(CONTENT_SOURCE_DEFINITION) + second = manifest_for_definition(CONTENT_SOURCE_DEFINITION) + + assert first == second + assert first.fingerprint.startswith("sha256:") + with pytest.raises(ValidationError, match="fingerprint does not match"): + SourceDefinitionManifest.model_validate(first.model_dump(mode="json", by_alias=True) | {"version": "2"}) + + +def test_projected_source_remains_usable_without_worker_definition_code() -> None: + async def scenario() -> None: + registry = SourceDefinitionRegistry((CONTENT_SOURCE_DEFINITION,)) + source = await registry.resolve( + ContentCapture(source_id="turn-1", content="Keep the remote contract declarative.") + ) + projected = project_source_for_transport(registry, source) + catalog = SourceCatalog(backend=EmptySourceBackend()) + payload = await catalog.read(projected) + projection = catalog.project(projected, TEXT_EVIDENCE_PROJECTION_KEY) + + assert isinstance(projected, ProjectedSource) + assert catalog.as_ref(projected).model_dump() == {"source_type": "content", "source_id": "turn-1"} + assert payload == projected.payload + assert catalog.projection_keys(projected) == (TEXT_EVIDENCE_PROJECTION_KEY,) + assert TextEvidence.model_validate(projection).content == "Keep the remote contract declarative." + + asyncio.run(scenario()) diff --git a/uv.lock b/uv.lock index 8688c1996..dfa34748a 100644 --- a/uv.lock +++ b/uv.lock @@ -761,15 +761,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/4a/e3/f1fae3647d170919c2cf2a898e77e7d1a4e5c7cae0aed7bb4bd3f5ebff6f/filelock-3.29.5-py3-none-any.whl", hash = "sha256:8af830889ba3a0ffcefbd6c7d2af8a54012058103771f2e10848222f476a1693", size = 45073, upload-time = "2026-07-03T03:50:30.445Z" }, ] -[[package]] -name = "fsspec" -version = "2026.7.0" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/00/78/f34251dadb8f3921264a1d9b8946f5e542014ee2614b285261b4e40e6775/fsspec-2026.7.0.tar.gz", hash = "sha256:c803c40f4cf860b49dea58ee3e1c33cb9c790520e233537e1340049f89b82a88", size = 317040, upload-time = "2026-07-28T16:34:51.052Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/fd/3c/6a2bf344106328fd04963664a60b9bb6496fc25df8e962fcdc1367285fb9/fsspec-2026.7.0-py3-none-any.whl", hash = "sha256:b57ddbafedfaef7018c1ecab32aa200a9d7ca26b77965f64e48b70061249d279", size = 206583, upload-time = "2026-07-28T16:34:49.538Z" }, -] - [[package]] name = "genai-prices" version = "0.1.2" @@ -1826,45 +1817,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/12/cf/03675d8bd8ecbf4445504d8071adab19f5f993676795708e36402ab38263/openapi_pydantic-0.5.1-py3-none-any.whl", hash = "sha256:a3a09ef4586f5bd760a8df7f43028b60cafb6d9f61de2acba9574766255ab146", size = 96381, upload-time = "2025-01-08T19:29:25.275Z" }, ] -[[package]] -name = "opendal" -version = "0.47.6" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/97/a3/898b9097795c4015a3329dc3f99a96350570769367ddea7a9d087ffb1a05/opendal-0.47.6.tar.gz", hash = "sha256:297b876ab44162490d4e38041ecb712800ad14814ee35c4c3c196af4b26ffd61", size = 1801677, upload-time = "2026-08-20T17:32:58.452Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/f4/8e/51d2c1cfabb4052cf7bbf3aef80a49a699f3e5d55f7ebd67b9de3830ad15/opendal-0.47.6-cp311-abi3-macosx_10_12_x86_64.whl", hash = "sha256:ce28bee2ef19c04d16033b602604d4d26eedb295c7868ced2c55ef1255aa695b", size = 17705330, upload-time = "2026-08-20T17:32:19.061Z" }, - { url = "https://files.pythonhosted.org/packages/24/fa/84e3bb3e2ff046fd9b55c870badf929a51cf6e399843c125aa7da9f958f5/opendal-0.47.6-cp311-abi3-macosx_11_0_arm64.whl", hash = "sha256:94a481ba0931df63e361347efb02bb4e715c63b12ac5f3cb3bfb8f2f6bbed527", size = 16330575, upload-time = "2026-08-20T17:32:20.956Z" }, - { url = "https://files.pythonhosted.org/packages/28/d9/315ce33d51500222ad5e47769d423d9eff51b2b67275872b3afc03bf6a86/opendal-0.47.6-cp311-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:586dd31f6b0e337f5037d6f80035a5b8c615ed56fe0a13b72ece68c6675ed749", size = 16923201, upload-time = "2026-08-20T17:32:23.465Z" }, - { url = "https://files.pythonhosted.org/packages/1a/04/ddb50902fef4ac5f9a201f16cf1041ba7b4cf351bdce5fcf3dd26cb91012/opendal-0.47.6-cp311-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ee1443cbab2e95fd55ad35c890284693ced5132e74ab63e9953ecabb15efe4f5", size = 18146763, upload-time = "2026-08-20T17:32:25.583Z" }, - { url = "https://files.pythonhosted.org/packages/d4/0b/b0a993271eb0745e428c1358401af52d1293dddc7d758e5b3bde64ea2fac/opendal-0.47.6-cp311-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:a9c457824b161f8351801c03632c9b617f32dd597b2a6fd9c88083c8c9428a1a", size = 17220213, upload-time = "2026-08-20T17:32:27.878Z" }, - { url = "https://files.pythonhosted.org/packages/40/4b/cd2cd193a6cb250699cc4b816baebac9cf389d34bbaaac4dba9d21dedca2/opendal-0.47.6-cp311-abi3-musllinux_1_1_aarch64.whl", hash = "sha256:8be0535b4cbd28115458a9bc194030fde49ce9a626f0c225b6f900203a942cfc", size = 17561196, upload-time = "2026-08-20T17:32:30.343Z" }, - { url = "https://files.pythonhosted.org/packages/f9/e6/d2061fdef84f173e580dd42fedfc99e7c43b7bbe9bea4d0f271464cdbdb1/opendal-0.47.6-cp311-abi3-musllinux_1_1_armv7l.whl", hash = "sha256:4e815a0120f95d9ae37312f5c6b2e8da2010314c0c55b18cfb4c90384bcf2743", size = 17227288, upload-time = "2026-08-20T17:32:32.469Z" }, - { url = "https://files.pythonhosted.org/packages/57/09/51f24e3540f0777d0433f2559e8158e2297eb120265544b4be2d831f280a/opendal-0.47.6-cp311-abi3-musllinux_1_1_x86_64.whl", hash = "sha256:ac3519eb30648d5f8d159dd29e1c54dac1a5730c8fe656ccfd1664fd042e8b8a", size = 18373732, upload-time = "2026-08-20T17:32:34.66Z" }, - { url = "https://files.pythonhosted.org/packages/85/cd/45e8484b5b141e9605b49a8fe3b92c747d486d59a04da868dc5ae62ebb9a/opendal-0.47.6-cp311-abi3-win_amd64.whl", hash = "sha256:81d52f5919b0845f747eeef16bf41a968ad11a16ede3a686556c75b1d0903298", size = 19421219, upload-time = "2026-08-20T17:32:36.991Z" }, - { url = "https://files.pythonhosted.org/packages/75/c9/ce51d2f5cda6f961d77dd376c3a7db9c74fac8769a36a2d91f3a09c1f03e/opendal-0.47.6-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:8142f90419334550cc42ea655184e3a5744909d8cf8a53462cf22849b5448a06", size = 17713841, upload-time = "2026-08-20T17:32:39.279Z" }, - { url = "https://files.pythonhosted.org/packages/25/c2/8daa8ff57682104992fce97e3f776e3289516538e1267b8fcbc94eb79939/opendal-0.47.6-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:19ca7f2c9751fc1279d4803b65cf27c0ea55d035e29601e263cfcc4b31c6435f", size = 16314577, upload-time = "2026-08-20T17:32:41.397Z" }, - { url = "https://files.pythonhosted.org/packages/e7/0d/bc5a06ddca9d8d30bc5ca2fd1c5fd16c411ecf15e3523376760a93d9a972/opendal-0.47.6-cp314-cp314t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:dfa9b490c327745032ea865dd28cc5b2282b1a5e1ed6fe03e2e3320a74677479", size = 16917195, upload-time = "2026-08-20T17:32:43.532Z" }, - { url = "https://files.pythonhosted.org/packages/9c/61/c70cec20351311caf33c2a261e4e615798f0e733c68f84b81983b329bb46/opendal-0.47.6-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:43a0a12144066fff5246a8e34116724e8248bc5c3c34c6998fcea5d01586bfa1", size = 18136350, upload-time = "2026-08-20T17:32:45.701Z" }, - { url = "https://files.pythonhosted.org/packages/ad/57/15358c7e9455f34a5abe6dc156024d7681b78d64c4e2f2db9300e3a1ca18/opendal-0.47.6-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:71e99848b40787e9505f5423473488bc818499c92707dc168037a047d297be1b", size = 17208049, upload-time = "2026-08-20T17:32:47.801Z" }, - { url = "https://files.pythonhosted.org/packages/d4/f5/72cea960b30c643cd3b216d2ad4af3c14ce503c21045b75a7ea23905d1de/opendal-0.47.6-cp314-cp314t-musllinux_1_1_aarch64.whl", hash = "sha256:64ee2b40a4f457f60ddfb822a5263962ff1bc07399403ceff0e90a175defafef", size = 17558165, upload-time = "2026-08-20T17:32:49.903Z" }, - { url = "https://files.pythonhosted.org/packages/0c/6f/ae82bedcc1acc2d1047931c4aefd750afecc2f66b2e7967a3afdbffdbe48/opendal-0.47.6-cp314-cp314t-musllinux_1_1_armv7l.whl", hash = "sha256:0e3532a848e3473e1787ed29a745b85b699938a1e01eeea390ace833405fd545", size = 17221105, upload-time = "2026-08-20T17:32:52.026Z" }, - { url = "https://files.pythonhosted.org/packages/24/5b/de96032981f74e0e473c4e81efd4357657aacf4673c3e09b8c3cd665d5d5/opendal-0.47.6-cp314-cp314t-musllinux_1_1_x86_64.whl", hash = "sha256:0ab143680d79a189deae5cb9ed2dbd7adbe967bfd82947317f09f02709553a21", size = 18367917, upload-time = "2026-08-20T17:32:54.371Z" }, - { url = "https://files.pythonhosted.org/packages/57/c3/da0429e7da22bbddd2708259e134f24d228e862404f8c430fa4bdcd4f4b9/opendal-0.47.6-cp314-cp314t-win_amd64.whl", hash = "sha256:60673bbf72aad5f1b1be37be5c25014a852349b1e395671731cd49c6792aae04", size = 19402260, upload-time = "2026-08-20T17:32:56.707Z" }, -] - -[[package]] -name = "opendalfs" -version = "0.1.0" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "fsspec", marker = "python_full_version >= '3.12'" }, - { name = "opendal", marker = "python_full_version >= '3.12'" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/0d/23/fdb4d34f08ac68a27b2d553256e13b15ce139916d36bab33f9a0e71e8770/opendalfs-0.1.0.tar.gz", hash = "sha256:5ceaeccc0852ef10c8ef2ce20e5927dbc18083909c06550ad36f81bcacb5774d", size = 75112, upload-time = "2026-08-26T05:30:33.497Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/ec/7d/baac5ab7f0fcd4d4f06484531e30dbee2eaf924cc0bf007572bc54bbc14a/opendalfs-0.1.0-py3-none-any.whl", hash = "sha256:8b180c9dae767053023ea4923250ce66d40a9bcad2e6f947df5ec8152208365b", size = 16278, upload-time = "2026-08-26T05:30:32.22Z" }, -] - [[package]] name = "opentelemetry-api" version = "1.43.0" @@ -2115,6 +2067,7 @@ dependencies = [ builtin = [ { name = "aiosqlite" }, { name = "apscheduler" }, + { name = "jsonschema" }, { name = "pydantic-ai-slim", extra = ["anthropic", "openai"] }, { name = "pydantic-settings" }, { name = "pyobvector" }, @@ -2133,12 +2086,10 @@ client = [ { name = "opentelemetry-api" }, { name = "pydantic-settings" }, ] -opendal = [ - { name = "opendalfs", marker = "python_full_version >= '3.12'" }, -] seekdb = [ { name = "aiosqlite" }, { name = "apscheduler" }, + { name = "jsonschema" }, { name = "pydantic-ai-slim", extra = ["anthropic", "openai"] }, { name = "pydantic-settings" }, { name = "pylibseekdb", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, @@ -2152,6 +2103,7 @@ server = [ { name = "fastapi" }, { name = "fastmcp" }, { name = "jinja2" }, + { name = "jsonschema" }, { name = "opentelemetry-api" }, { name = "opentelemetry-sdk" }, { name = "platformdirs" }, @@ -2201,7 +2153,9 @@ requires-dist = [ { name = "httpx", extras = ["socks"], marker = "extra == 'cli'", specifier = ">=0.28,<1" }, { name = "httpx", extras = ["socks"], marker = "extra == 'client'", specifier = ">=0.28,<1" }, { name = "jinja2", marker = "extra == 'server'", specifier = ">=3.1,<4" }, - { name = "opendalfs", marker = "python_full_version >= '3.12' and extra == 'opendal'", specifier = ">=0.1,<0.2" }, + { name = "jsonschema", marker = "extra == 'builtin'", specifier = ">=4.23,<5" }, + { name = "jsonschema", marker = "extra == 'seekdb'", specifier = ">=4.23,<5" }, + { name = "jsonschema", marker = "extra == 'server'", specifier = ">=4.23,<5" }, { name = "opentelemetry-api", marker = "extra == 'cli'", specifier = ">=1.30,<2" }, { name = "opentelemetry-api", marker = "extra == 'client'", specifier = ">=1.30,<2" }, { name = "opentelemetry-api", marker = "extra == 'server'", specifier = ">=1.30,<2" }, @@ -2234,7 +2188,7 @@ requires-dist = [ { name = "typing-extensions", specifier = ">=4.12,<5" }, { name = "uvicorn", marker = "extra == 'server'", specifier = ">=0.34,<1" }, ] -provides-extras = ["builtin", "cli", "client", "opendal", "seekdb", "server", "tracing-otlp"] +provides-extras = ["builtin", "cli", "client", "seekdb", "server", "tracing-otlp"] [package.metadata.requires-dev] dev = [ From b8d35edfc6c030c52fc4d28becc51e8600e57fad Mon Sep 17 00:00:00 2001 From: Chojan Shang Date: Sun, 30 Aug 2026 20:25:27 +0800 Subject: [PATCH 6/9] docs(rfc): clarify source evidence guarantees Co-authored-by: Nene7ko_ <1604009816@qq.com> --- ...0000_source_definition_and_observation_model.md | 14 ++++++++++++++ ...0000_source_definition_and_observation_model.md | 11 +++++++++++ 2 files changed, 25 insertions(+) diff --git a/docs/en/rfcs/0000_source_definition_and_observation_model.md b/docs/en/rfcs/0000_source_definition_and_observation_model.md index 8600f3c3f..3f591c4a6 100644 --- a/docs/en/rfcs/0000_source_definition_and_observation_model.md +++ b/docs/en/rfcs/0000_source_definition_and_observation_model.md @@ -308,6 +308,10 @@ Reference resolution verifies all four identity components and the stored observ digest. Failure to resolve the exact observation is distinct from the logical Source being deleted, the head having advanced, or the Connector being unavailable. +An accepted compatibility reference without `observation_id` still denotes one immutable observation. Resolution +must not treat it as a SourceKey, a current head, or `latest`. A compatibility layer may restore the full SourceRef at +its boundary, but it cannot redirect the evidence. + ## Definition registration contract Executable Definitions belong to the worker that resolves definition-native inputs, canonicalizes Source values, @@ -418,6 +422,11 @@ An Artifact revision records exact SourceRefs used directly by its computation. change existing Artifact lineage. Recalculation against a newer observation produces a new Artifact revision rather than rewriting prior evidence. +An observation referenced by a durable Artifact revision is protected from ordinary retention and garbage +collection. Advancing or deleting a Source head does not authorize removing that observation. An explicit deletion +policy may make cited evidence unavailable, but it must preserve the SourceRef in lineage and report the +unavailability rather than resolve the reference to another observation. + Sources remain in their producing Scope. A Context Reference may expand a read selection according to the Scope organization contract, but it does not change Source ownership. Exact Artifact publication across Scopes retains the origin Scope and exact SourceRef in lineage. Publishing an Artifact does not publish every Source in its origin Scope. @@ -522,6 +531,11 @@ Connector replacement change Source identity. - OpenMetadata separates the Source that emits records from connection checks, workflow status, and the sink. - Nowledge Mem's TiddlyWiki importer uses stable logical IDs, canonical payload digests, source revalidation, and per-item outcomes. Those behaviors inform the separation between Source observations and Connector run state. +- [TencentDB-Agent-Memory](https://github.com/TencentCloud/TencentDB-Agent-Memory/tree/5299c00aaf65481703c180fd69df066d11254eb7) + uses a SourceFetcher registry for provider acquisition, provider revisions and content hashes for change detection, + and separate synchronization and audit state. Those patterns belong to Connector acquisition. They do not replace + immutable Source observations because an Artifact citation must retain the value it used after the provider's + current state changes. # Unresolved questions diff --git a/docs/zh/rfcs/0000_source_definition_and_observation_model.md b/docs/zh/rfcs/0000_source_definition_and_observation_model.md index 7336d610a..c78e26434 100644 --- a/docs/zh/rfcs/0000_source_definition_and_observation_model.md +++ b/docs/zh/rfcs/0000_source_definition_and_observation_model.md @@ -287,6 +287,9 @@ cross-Scope provenance 的 reference 必须显式携带 owner Scope。 Reference resolution 会验证全部四个 identity components，以及 stored observation 的 Definition version 与 digest。无法解析精确 observation，不等同于 logical Source 已删除、head 已推进或 Connector 不可用。 +不带 `observation_id` 的已接受兼容引用仍然标识一个不可变 observation。解析时不得将其视为 SourceKey、current +head 或 `latest`。compatibility layer 可以在边界恢复完整 SourceRef，但不得重定向该 evidence。 + ## Definition registration contract Executable Definition 属于 worker。Worker 用它解析 definition-native input、canonicalize Source value，并计算 @@ -389,6 +392,10 @@ package 在 PowerContext Server 之外执行，并使用 remote worker ingestion Artifact revision 记录其计算直接使用的精确 SourceRef。推进 Source head 不改变现有 Artifact lineage。针对较新 observation 的重新计算会产生新的 Artifact revision，而不是重写旧 evidence。 +被 durable Artifact revision 引用的 observation 受普通 retention 与 garbage collection 保护。推进或删除 Source +head 不会授权删除该 observation。显式 deletion policy 可以使被引用的 evidence 不再可用，但必须在 lineage 中 +保留 SourceRef 并报告不可用状态，不能把该引用解析到另一个 observation。 + Source 保留在 producing Scope。Context Reference 可以按照 Scope organization contract 扩展 read selection， 但不会改变 Source ownership。跨 Scope 的精确 Artifact publication 在 lineage 中保留 origin Scope 与精确 SourceRef。发布 Artifact 不会发布其 origin Scope 中的所有 Source。 @@ -486,6 +493,10 @@ replacement 改变 Source identity。 - OpenMetadata 把负责生成 record 的 Source 与 connection check、workflow status、sink 分离。 - Nowledge Mem 的 TiddlyWiki importer 使用 stable logical ID、canonical payload digest、source revalidation 与 per-item outcome。这些行为为 Source observation 与 Connector run state 的分离提供依据。 +- [TencentDB-Agent-Memory](https://github.com/TencentCloud/TencentDB-Agent-Memory/tree/5299c00aaf65481703c180fd69df066d11254eb7) + 使用 SourceFetcher registry 获取 provider value，以 provider revision 与 content hash 检测变化，并单独维护 + synchronization 和 audit state。这些模式属于 Connector acquisition，不能替代 immutable Source observation， + 因为 Artifact citation 必须在 provider current state 变化后仍然保留它使用过的值。 # Unresolved questions From ec9fc7b3fa3af9d2b7b20b0ef48e58f588c9704d Mon Sep 17 00:00:00 2001 From: Chojan Shang Date: Sun, 30 Aug 2026 20:39:33 +0800 Subject: [PATCH 7/9] docs(rfc): trim source prior art --- .../rfcs/0000_source_definition_and_observation_model.md | 7 ------- .../rfcs/0000_source_definition_and_observation_model.md | 7 ------- 2 files changed, 14 deletions(-) diff --git a/docs/en/rfcs/0000_source_definition_and_observation_model.md b/docs/en/rfcs/0000_source_definition_and_observation_model.md index 3f591c4a6..676a71b7a 100644 --- a/docs/en/rfcs/0000_source_definition_and_observation_model.md +++ b/docs/en/rfcs/0000_source_definition_and_observation_model.md @@ -519,13 +519,6 @@ Connector replacement change Source identity. - The [Scope organization and Agent integration design](https://github.com/oceanbase/powercontext/pull/1345) separates Scope ownership, read sharing, organization, delivery, and observation. This RFC applies the same separation to Source ownership, identity, exact evidence, and acquisition. -- [Apache OpenDAL OFS RFC-0016](https://github.com/apache/opendal-ofs/blob/main/rfcs/0016_filesystem_architecture.md) - separates namespace authority from access frontends and forbids a frontend from advertising guarantees that the - underlying layers cannot enforce. Source materialization follows the same authority rule. -- [opendalfs](https://github.com/fsspec/opendalfs) exposes OpenDAL services through the fsspec interface and - demonstrates backend-neutral filesystem acquisition. Its paths and file metadata do not define Source identity or - immutable revision semantics. A backend read can satisfy referenced materialization only when the complete stack - addresses and verifies an immutable revision. - DataHub stateful ingestion separates connector checkpoints and stale-entity detection from emitted metadata identity. Airbyte treats connector state as an opaque recovery boundary rather than record identity. - OpenMetadata separates the Source that emits records from connection checks, workflow status, and the sink. diff --git a/docs/zh/rfcs/0000_source_definition_and_observation_model.md b/docs/zh/rfcs/0000_source_definition_and_observation_model.md index c78e26434..dc589fe53 100644 --- a/docs/zh/rfcs/0000_source_definition_and_observation_model.md +++ b/docs/zh/rfcs/0000_source_definition_and_observation_model.md @@ -481,13 +481,6 @@ replacement 改变 Source identity。 - [Scope organization and Agent integration design](https://github.com/oceanbase/powercontext/pull/1345) 分离 Scope ownership、read sharing、organization、delivery 与 observation。本 RFC 对 Source ownership、identity、 exact evidence 与 acquisition 应用同样的分离原则。 -- [Apache OpenDAL OFS RFC-0016](https://github.com/apache/opendal-ofs/blob/main/rfcs/0016_filesystem_architecture.md) - 分离 namespace authority 与 access frontend，并禁止 frontend 宣称底层无法兑现的保证。Source materialization - 遵循同样的 authority rule。 -- [opendalfs](https://github.com/fsspec/opendalfs) 通过 fsspec interface 暴露 OpenDAL services，展示了 - backend-neutral filesystem acquisition。它的 path 与 file metadata 不定义 Source identity 或 immutable - revision semantics。只有完整调用链能够寻址并验证不可变 revision 时，backend read 才能满足 referenced - materialization。 - DataHub stateful ingestion 把 connector checkpoint 与 stale-entity detection 同 emitted metadata identity 分离。Airbyte 把 connector state 当作 opaque recovery boundary，而不是 record identity。 - OpenMetadata 把负责生成 record 的 Source 与 connection check、workflow status、sink 分离。 From 9c699db79f14e925bc7de5ff0a35acbeeaa12f0a Mon Sep 17 00:00:00 2001 From: Chojan Shang Date: Sun, 30 Aug 2026 20:51:11 +0800 Subject: [PATCH 8/9] fix(ci): synchronize source integration artifacts --- e2e/bub/uv.lock | 5 +- integrations/bub/pyproject.toml | 2 +- .../dsh/plugins/powercontext/lib/index.js | 24 ++ .../powercontext/openapi/powercontext.yaml | 273 ++++++++++++++++++ integrations/opendal/pyproject.toml | 6 + .../connector.py | 5 +- 6 files changed, 311 insertions(+), 4 deletions(-) diff --git a/e2e/bub/uv.lock b/e2e/bub/uv.lock index 3a361263c..6e3268e99 100644 --- a/e2e/bub/uv.lock +++ b/e2e/bub/uv.lock @@ -1627,6 +1627,9 @@ requires-dist = [ { name = "httpx", extras = ["socks"], marker = "extra == 'client'", specifier = ">=0.28,<1" }, { name = "inquirerpy", marker = "extra == 'cli'", specifier = ">=0.3,<1" }, { name = "jinja2", marker = "extra == 'server'", specifier = ">=3.1,<4" }, + { name = "jsonschema", marker = "extra == 'builtin'", specifier = ">=4.23,<5" }, + { name = "jsonschema", marker = "extra == 'seekdb'", specifier = ">=4.23,<5" }, + { name = "jsonschema", marker = "extra == 'server'", specifier = ">=4.23,<5" }, { name = "opentelemetry-api", marker = "extra == 'cli'", specifier = ">=1.30,<2" }, { name = "opentelemetry-api", marker = "extra == 'client'", specifier = ">=1.30,<2" }, { name = "opentelemetry-api", marker = "extra == 'server'", specifier = ">=1.30,<2" }, @@ -1697,7 +1700,7 @@ dependencies = [ requires-dist = [ { name = "bub", specifier = ">=0.4.0,<0.5.0" }, { name = "httpx", specifier = ">=0.28,<1" }, - { name = "powercontext", extras = ["client"], specifier = ">=0.0.3" }, + { name = "powercontext", extras = ["client"] }, { name = "pydantic-settings", specifier = ">=2.7,<3" }, ] diff --git a/integrations/bub/pyproject.toml b/integrations/bub/pyproject.toml index 4a3f36128..8063d100f 100644 --- a/integrations/bub/pyproject.toml +++ b/integrations/bub/pyproject.toml @@ -20,7 +20,7 @@ requires-python = ">=3.12,<4.0" dependencies = [ "bub>=0.4.0,<0.5.0", "httpx>=0.28,<1", - "powercontext[client]>=0.0.3", + "powercontext[client]", "pydantic-settings>=2.7,<3", ] diff --git a/integrations/dsh/plugins/powercontext/lib/index.js b/integrations/dsh/plugins/powercontext/lib/index.js index 9c977a528..e2a59576f 100644 --- a/integrations/dsh/plugins/powercontext/lib/index.js +++ b/integrations/dsh/plugins/powercontext/lib/index.js @@ -105,6 +105,30 @@ const OPERATIONS = { location: "body", scope: true }, + register_source_definition: { + method: "POST", + path: "/v1/source-definitions/register", + location: "body", + scope: false + }, + get_connector_checkpoint: { + method: "POST", + path: "/v1/connector-checkpoints/get", + location: "body", + scope: false + }, + submit_source_observation: { + method: "POST", + path: "/v1/source-observations", + location: "body", + scope: false + }, + commit_connector_checkpoint: { + method: "POST", + path: "/v1/connector-checkpoints/commit", + location: "body", + scope: false + }, prepare_context: { method: "POST", path: "/v1/context/prepare", diff --git a/integrations/dsh/plugins/powercontext/openapi/powercontext.yaml b/integrations/dsh/plugins/powercontext/openapi/powercontext.yaml index 2c8681f99..bed6472a9 100644 --- a/integrations/dsh/plugins/powercontext/openapi/powercontext.yaml +++ b/integrations/dsh/plugins/powercontext/openapi/powercontext.yaml @@ -111,6 +111,107 @@ paths: $ref: "#/components/responses/Unavailable" "500": $ref: "#/components/responses/InternalError" + /v1/source-definitions/register: + post: + tags: [source-ingestion] + summary: Register a worker-owned Source Definition manifest + description: Registers an immutable declarative manifest without loading worker plugin code. + operationId: register_source_definition + requestBody: + required: true + content: + application/json: + schema: + $ref: "#/components/schemas/RegisterSourceDefinitionRequest" + responses: + "200": + description: The exact manifest is registered or was already registered identically. + content: + application/json: + schema: + $ref: "#/components/schemas/SourceDefinitionManifest" + "409": + $ref: "#/components/responses/Conflict" + "401": + $ref: "#/components/responses/Unauthorized" + "422": + $ref: "#/components/responses/InvalidRequest" + /v1/connector-checkpoints/get: + post: + tags: [source-ingestion] + summary: Read a Connector binding checkpoint + operationId: get_connector_checkpoint + requestBody: + required: true + content: + application/json: + schema: + $ref: "#/components/schemas/GetConnectorCheckpointRequest" + responses: + "200": + description: The current opaque checkpoint, including a normal null initial value. + content: + application/json: + schema: + $ref: "#/components/schemas/ConnectorCheckpointState" + "401": + $ref: "#/components/responses/Unauthorized" + "409": + $ref: "#/components/responses/Conflict" + "422": + $ref: "#/components/responses/InvalidRequest" + /v1/source-observations: + post: + tags: [source-ingestion] + summary: Submit a worker-materialized Source observation + description: Validates the observation against its registered manifest and durably appends it before receipt. + operationId: submit_source_observation + requestBody: + required: true + content: + application/json: + schema: + $ref: "#/components/schemas/SubmitSourceObservationRequest" + responses: + "202": + description: The observation is durably accepted and can be referenced exactly. + content: + application/json: + schema: + $ref: "#/components/schemas/SourceObservationReceipt" + "401": + $ref: "#/components/responses/Unauthorized" + "409": + $ref: "#/components/responses/Conflict" + "404": + $ref: "#/components/responses/NotFound" + "422": + $ref: "#/components/responses/InvalidRequest" + /v1/connector-checkpoints/commit: + post: + tags: [source-ingestion] + summary: Commit a Connector binding checkpoint + description: Replaces the checkpoint only when its expected starting value still matches. + operationId: commit_connector_checkpoint + requestBody: + required: true + content: + application/json: + schema: + $ref: "#/components/schemas/CommitConnectorCheckpointRequest" + responses: + "200": + description: The new opaque checkpoint is durable. + content: + application/json: + schema: + $ref: "#/components/schemas/ConnectorCheckpointState" + "401": + $ref: "#/components/responses/Unauthorized" + "409": + $ref: "#/components/responses/Conflict" + "422": + $ref: "#/components/responses/InvalidRequest" /v1/context/prepare: post: tags: [context] @@ -2663,6 +2764,178 @@ components: position: type: integer minimum: 1 + SourceProjectionKey: + type: object + additionalProperties: false + required: [name, version] + properties: + name: + type: string + minLength: 1 + maxLength: 128 + pattern: '.*\S.*' + version: + type: string + minLength: 1 + maxLength: 128 + pattern: '.*\S.*' + SourceProjectionManifest: + type: object + additionalProperties: false + required: [key, schema] + properties: + key: + $ref: "#/components/schemas/SourceProjectionKey" + schema: + type: object + additionalProperties: true + SourceDefinitionManifest: + type: object + additionalProperties: false + required: [name, version, fingerprint, source_schema, projections] + properties: + name: + type: string + minLength: 1 + maxLength: 128 + pattern: '.*\S.*' + version: + type: string + minLength: 1 + maxLength: 128 + pattern: '.*\S.*' + fingerprint: + type: string + pattern: '^sha256:[0-9a-f]{64}$' + source_schema: + type: object + additionalProperties: true + projections: + type: array + maxItems: 16 + items: + $ref: "#/components/schemas/SourceProjectionManifest" + RegisterSourceDefinitionRequest: + type: object + additionalProperties: false + required: [manifest] + properties: + manifest: + $ref: "#/components/schemas/SourceDefinitionManifest" + ConnectorBinding: + type: object + additionalProperties: false + required: [scope_id, binding_id, connector_name, connector_version] + properties: + scope_id: + type: string + minLength: 1 + maxLength: 256 + pattern: '.*\S.*' + binding_id: + type: string + minLength: 1 + maxLength: 256 + pattern: '.*\S.*' + connector_name: + type: string + minLength: 1 + maxLength: 128 + pattern: '.*\S.*' + connector_version: + type: string + minLength: 1 + maxLength: 128 + pattern: '.*\S.*' + GetConnectorCheckpointRequest: + type: object + additionalProperties: false + required: [binding] + properties: + binding: + $ref: "#/components/schemas/ConnectorBinding" + ConnectorCheckpointState: + type: object + additionalProperties: false + required: [binding, checkpoint] + properties: + binding: + $ref: "#/components/schemas/ConnectorBinding" + checkpoint: + nullable: true + SourceProjectionValue: + type: object + additionalProperties: false + required: [key, value] + properties: + key: + $ref: "#/components/schemas/SourceProjectionKey" + value: {} + ProjectedSource: + type: object + additionalProperties: false + required: + [name, definition_version, materialization, source_type, definition_fingerprint, payload, projections] + properties: + name: + type: string + minLength: 1 + maxLength: 256 + definition_version: + type: string + minLength: 1 + maxLength: 128 + materialization: + type: string + enum: [captured, referenced] + description: + type: string + nullable: true + source_type: + type: string + minLength: 1 + maxLength: 128 + definition_fingerprint: + type: string + pattern: '^sha256:[0-9a-f]{64}$' + payload: + type: object + additionalProperties: true + projections: + type: array + maxItems: 16 + items: + $ref: "#/components/schemas/SourceProjectionValue" + SubmitSourceObservationRequest: + type: object + additionalProperties: false + required: [binding, source] + properties: + binding: + $ref: "#/components/schemas/ConnectorBinding" + source: + $ref: "#/components/schemas/ProjectedSource" + SourceObservationReceipt: + type: object + additionalProperties: false + required: [source, position] + properties: + source: + $ref: "#/components/schemas/SourceReference" + position: + type: integer + minimum: 1 + CommitConnectorCheckpointRequest: + type: object + additionalProperties: false + required: [binding, expected, checkpoint] + properties: + binding: + $ref: "#/components/schemas/ConnectorBinding" + expected: + nullable: true + checkpoint: + nullable: true CommitHandoffRequest: type: object additionalProperties: false diff --git a/integrations/opendal/pyproject.toml b/integrations/opendal/pyproject.toml index 5d08e8bb7..79f795c1b 100644 --- a/integrations/opendal/pyproject.toml +++ b/integrations/opendal/pyproject.toml @@ -5,6 +5,12 @@ # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. [project] name = "powercontext-connector-opendal" diff --git a/integrations/opendal/src/powercontext_connector_opendal/connector.py b/integrations/opendal/src/powercontext_connector_opendal/connector.py index 4a9438c80..6d8b95da3 100644 --- a/integrations/opendal/src/powercontext_connector_opendal/connector.py +++ b/integrations/opendal/src/powercontext_connector_opendal/connector.py @@ -23,6 +23,7 @@ import posixpath from collections.abc import Mapping, Sequence from datetime import UTC, datetime +from importlib import import_module from typing import Any, Literal, Protocol from pydantic import BaseModel, ConfigDict, Field, JsonValue, ValidationError, field_validator @@ -124,13 +125,13 @@ def from_service( """Create a Connector from one OpenDAL service and its runtime-only options.""" try: - from opendalfs import OpendalFileSystem + opendalfs = import_module("opendalfs") except ImportError as error: raise ImportError( # noqa: TRY003 "OpenDALTextFileConnector.from_service requires powercontext-connector-opendal on Python 3.12+" ) from error backend_options: dict[str, Any] = dict(storage_options or {}) - filesystem = OpendalFileSystem( + filesystem = opendalfs.OpendalFileSystem( scheme=service, asynchronous=False, skip_instance_cache=True, From 439f1bb9e74cfb24447b9d297ea9b3f9697736c2 Mon Sep 17 00:00:00 2001 From: PsiACE Date: Mon, 31 Aug 2026 10:21:46 +0800 Subject: [PATCH 9/9] docs(rfc): explain source scope flows --- ...source_definition_and_observation_model.md | 112 ++++++++++++++++++ ...source_definition_and_observation_model.md | 110 +++++++++++++++++ 2 files changed, 222 insertions(+) diff --git a/docs/en/rfcs/0000_source_definition_and_observation_model.md b/docs/en/rfcs/0000_source_definition_and_observation_model.md index 676a71b7a..f721ef57a 100644 --- a/docs/en/rfcs/0000_source_definition_and_observation_model.md +++ b/docs/en/rfcs/0000_source_definition_and_observation_model.md @@ -53,6 +53,22 @@ or become temporarily unreadable. Artifacts that used an earlier value must cont The two-part `(source_type, source_id)` Source reference cannot express both the stable logical object and its immutable observation without making every integration invent a composite `source_id`. +For example, using only the provider object ID makes the second value conflict with or replace the first. Using only +a value digest keeps both values but loses the fact that they describe the same continuing object: + +```text +provider object 42 + | + +-- value v1 ----> exact observation 1 + `-- value v2 ----> exact observation 2 + ^ + | + same logical Source +``` + +The model therefore keeps logical identity and exact evidence separate. Consumers can follow the continuing Source +when they need current state while Artifacts keep citing the observation they actually used. + The extension boundary is also incomplete. A Source adapter binds a native input class to a concrete Source class and a read result, while the built-in Runtime and relational persistence assemble a fixed adapter set. This does not state the durable rules an independently defined Source type must follow across identity, persistence, @@ -255,6 +271,78 @@ ContentSource is suitable for prompts, explicit text capture, import records, an already owns an immutable identity. Integrations that observe one logical object over time should define or reuse a multi-observation Source type instead. +The two ingestion paths differ at acquisition time but converge on Scope-owned Source history: + +| Concern | `ContentSource` capture | Source Definition and Connector ingestion | +| --- | --- | --- | +| Typical input | Text already held by the caller | Objects discovered in an external system | +| Identity | One caller-stable immutable identity | One logical identity with exact observations | +| Type contract | Built-in captured text and metadata | Definition-owned value, provenance, and projections | +| Synchronization | One request, with no checkpoint | Discovery, per-item outcomes, replay, and checkpoint comparison | +| Downstream use | Built-in text evidence | A named projection understood by the consumer | + +`ContentSource` is the shorter path when the caller already has final text and an immutable identity. The remote +ingestion APIs do not replace it. They add the lifecycle needed when a worker must discover, normalize, and +re-observe external objects without loading provider code or credentials into the Server. + +## Source participation in Scope flows + +Durable acceptance appends an exact observation to the Source journal of its owner Scope. Acceptance does not create +Memory or another Artifact by itself. A Scope-local processor later selects a bounded Source window, asks for a +projection it understands, and may produce a new Artifact revision that cites the exact SourceRef: + +```text +Connector or direct caller + | + | bind Scope A + v +Source observation + | + v +Scope A Source journal ----> Scope-local processor + | + named projection + | + v + Scope A Memory revision + cites exact SourceRef +``` + +A new observation can cause later processing, but it does not rewrite an earlier Artifact revision: + +```text +SourceKey(scope-a, record, provider-object-42) +|-- observation-1 ----> Memory revision 3 +`-- observation-2 ----> Memory revision 4 + +Memory revision 3 continues to cite observation-1. +``` + +A consumer uses a Source only through its native Definition or a compatible named projection. For example, a Memory +extractor that requires text evidence can consume any Source Definition that advertises the matching text +projection. It does not need to know whether the Source began as a file, page, issue, or `ContentSource`. A missing +capability remains explicit; the consumer does not infer text from metadata. + +Cross-Scope use depends on the intended ownership and delivery behavior: + +```text +Scope A Source history + | + +-- Context Reference from Scope B + | `-- later Prepare Context may read eligible Scope A material + | + +-- publish exact Artifact revision + | `-- Scope B receives one selected result with origin provenance + | + `-- deliberate capture into Scope B + `-- Scope B owns a new Source and runs its own downstream flow +``` + +Use a Context Reference for continuing read access and exact Artifact publication for a selected result. If Scope B +must own and independently process the external value, capture it into Scope B as a new Scope-owned observation and +retain the origin reference in provenance when applicable. None of these operations moves the original Source or +makes Parent imply read access. + # Reference-level explanation ## Source identity contract @@ -344,6 +432,30 @@ interaction consists of four generic operations: 3. submit a worker-materialized Source observation with all declared projections; and 4. compare-and-swap the binding checkpoint from the value read at run start. +The normal sequence is: + +```text +Connector worker PowerContext Server + | | + |-- register Definition manifest -------------->| + |<---------------- exact registered manifest ---| + | | + |-- get binding checkpoint -------------------->| + |<-------------------------- checkpoint C0 -----| + | | + |-- submit observation 1 ---------------------->| + |<---------------- durable SourceRef receipt ---| + |-- submit observation 2 ---------------------->| + |<---------------- durable SourceRef receipt ---| + | | + |-- commit checkpoint expected=C0, next=C1 ---->| + |<-------------------------- committed C1 ------| +``` + +If the worker stops after a durable receipt but before the checkpoint commit, the next run starts from the earlier +checkpoint and may submit the observation again. Identical submission is idempotent. Checkpoint comparison prevents +two runs of the same binding from silently replacing each other's progress. + The observation envelope carries the Definition name, version and fingerprint, canonical Source payload, and one value for every projection declared by the manifest. The Server validates envelope identity, payload schema, projection-key equality, projection schemas, and standard projection invariants before durable acceptance. Provider diff --git a/docs/zh/rfcs/0000_source_definition_and_observation_model.md b/docs/zh/rfcs/0000_source_definition_and_observation_model.md index dc589fe53..bfca3ab92 100644 --- a/docs/zh/rfcs/0000_source_definition_and_observation_model.md +++ b/docs/zh/rfcs/0000_source_definition_and_observation_model.md @@ -47,6 +47,22 @@ ETag 或 provider 当前值读取，并不能满足 referenced 契约。 精确证据。二元 `(source_type, source_id)` Source reference 无法同时表达稳定的逻辑对象和不可变观察，只能迫使每个集成自行发明复合 `source_id`。 +例如，只使用 provider object ID 时，第二个 value 会与第一个冲突或替换它。只使用 value digest 虽然能保留 +两个 value，却无法表达它们来自同一个持续存在的对象： + +```text +provider object 42 + | + +-- value v1 ----> exact observation 1 + `-- value v2 ----> exact observation 2 + ^ + | + same logical Source +``` + +因此，本模型分别保存 logical identity 与 exact evidence。需要 current state 的 consumer 可以沿着同一个 +logical Source 读取，而 Artifact 继续引用它实际使用的 observation。 + 扩展边界也不完整。Source adapter 将 native input class 绑定到具体 Source class 和读取结果，而内置 Runtime 与关系型持久化会组装固定 adapter 集合。它没有说明独立定义的 Source 类型在身份、持久化、传输与 Artifact evidence 上必须长期满足哪些规则。 @@ -237,6 +253,76 @@ logical Source lifecycle。 ContentSource 适合 prompt、显式文本捕获、import record，以及调用方已经拥有不可变身份的其他场景。持续观察同一 逻辑对象的集成应定义或复用 multi-observation Source type。 +两条 ingestion 路径的 acquisition 方式不同，但最终都进入 Scope-owned Source history： + +| Concern | `ContentSource` capture | Source Definition 与 Connector ingestion | +| --- | --- | --- | +| Typical input | 调用方已经持有的文本 | 从外部系统发现的对象 | +| Identity | 一个由调用方保持稳定的不可变身份 | 一个 logical identity 及其 exact observations | +| Type contract | 内置 captured text 与 metadata | Definition-owned value、provenance 与 projections | +| Synchronization | 单次请求，没有 checkpoint | Discovery、per-item outcomes、replay 与 checkpoint comparison | +| Downstream use | 内置 text evidence | Consumer 能理解的 named projection | + +当调用方已经持有最终文本和不可变身份时，`ContentSource` 是更短的路径。Remote ingestion API 不替代它； +这组 API 用于 worker 需要发现、规范化并重复观察外部对象的场景，同时避免把 provider code 或 credentials +加载进 Server。 + +## Source 如何参与 Scope 流程 + +Server 持久化接受 exact observation 后，会将它追加到 owner Scope 的 Source journal。接受 observation +本身不会创建 Memory 或其他 Artifact。Scope-local processor 随后选择一个有界 Source window，请求它能理解的 +projection，并可能产生一个引用 exact SourceRef 的新 Artifact revision： + +```text +Connector or direct caller + | + | bind Scope A + v +Source observation + | + v +Scope A Source journal ----> Scope-local processor + | + named projection + | + v + Scope A Memory revision + cites exact SourceRef +``` + +新的 observation 可以触发后续处理，但不会重写旧 Artifact revision： + +```text +SourceKey(scope-a, record, provider-object-42) +|-- observation-1 ----> Memory revision 3 +`-- observation-2 ----> Memory revision 4 + +Memory revision 3 continues to cite observation-1. +``` + +Consumer 只能通过 native Definition 或兼容的 named projection 使用 Source。例如，需要 text evidence 的 +Memory extractor 可以处理任何声明了对应 text projection 的 Source Definition，不需要知道 Source 最初来自 +file、page、issue 还是 `ContentSource`。缺失的 capability 必须保持显式，consumer 不会从 metadata 推断文本。 + +跨 Scope 使用 Source 时，需要先确定预期的 ownership 与 delivery 行为： + +```text +Scope A Source history + | + +-- Context Reference from Scope B + | `-- later Prepare Context may read eligible Scope A material + | + +-- publish exact Artifact revision + | `-- Scope B receives one selected result with origin provenance + | + `-- deliberate capture into Scope B + `-- Scope B owns a new Source and runs its own downstream flow +``` + +持续读取使用 Context Reference；交付一个选定结果时，发布 exact Artifact revision。如果 Scope B 必须拥有并 +独立处理这个外部值，应在 Scope B 中显式 capture，形成新的 Scope-owned observation，并在适用时把 origin +reference 保留到 provenance。以上操作都不会移动原始 Source，Parent 也不会因此获得 read access。 + # Reference-level explanation ## Source identity contract @@ -318,6 +404,30 @@ Connector 在独立 worker 进程中运行。Worker 拥有 provider access 与 3. 提交 worker 已物化的 Source observation 及其全部声明 projection； 4. 从 run 开始时读到的值 compare-and-swap binding checkpoint。 +正常时序如下： + +```text +Connector worker PowerContext Server + | | + |-- register Definition manifest -------------->| + |<---------------- exact registered manifest ---| + | | + |-- get binding checkpoint -------------------->| + |<-------------------------- checkpoint C0 -----| + | | + |-- submit observation 1 ---------------------->| + |<---------------- durable SourceRef receipt ---| + |-- submit observation 2 ---------------------->| + |<---------------- durable SourceRef receipt ---| + | | + |-- commit checkpoint expected=C0, next=C1 ---->| + |<-------------------------- committed C1 ------| +``` + +如果 worker 在收到 durable receipt 后、提交 checkpoint 前停止，下次 run 会从较早的 checkpoint 开始，并可能 +再次提交同一个 observation。相同提交具有幂等性。Checkpoint comparison 会阻止同一 binding 的两个 run +静默覆盖彼此的进度。 + Observation envelope 携带 Definition name、version 与 fingerprint、canonical Source payload，以及 manifest 声明的 每个 projection value。Server 在 durable acceptance 前验证 envelope identity、payload schema、projection key 集合相等、projection schema 与标准 projection invariant。Provider name、storage service、path、credential 或其他