Two open files designs disagree on where the bytes live, and neither references the other

[peteski22]

Problem

Two live pieces of work answer "where do a file's bytes live" in opposite ways, and neither names the other in its description, its comments or its reviews.

#1185 (open since 2026-09-15, "support provider-neutral Files API support for hybrid mode", fixes #984). Its description says, in as many words: "File contents stay with the provider." It adds Anthropic and OpenAI Files operations through any-llm, selected by a header, with shared services enforcing account binding, ownership, quotas, retention, revocation and cleanup.

#1470 (the files epic) and #1366, whose direction 1 reads: "Otari's store is the source of truth for files. There is one files API, and where the bytes live is hidden behind it. A copy at a provider is a short-lived cache, made only when a provider-native feature needs one, with an expiry. A file a provider's code produces is copied into Otari's store."

That second answer is not a proposal: it is merged and released. #1366 shipped the surface, and #1517 ("copy the files a provider's code produces into Otari's store", for #1475) is on main. Both are in v0.7.0, v0.8.0 and v0.9.0.

So the repository currently holds one released answer and one open PR built on the other, with no written reconciliation. Every downstream decision inherits the ambiguity: retention that reaches every copy (#1487), provider-native files (#1488), giving a provider's sandbox the files a request attaches (#1481), and how a hybrid data plane moves bytes at all.

What makes them genuinely hard to merge

They are not two spellings of one idea. Provider behavior differs by direction:

• An uploaded file on Anthropic is not downloadable. Only a file that code execution or skills created can be read back. So for uploads a provider copy is write-only: it cannot serve as a cache, and if Otari's store loses the bytes they are gone.
• A generated file can be read back, but on a clock. An OpenAI code-interpreter container expires about 20 minutes after its last use and its data goes with it, so copying out is a race.
• Anthropic files are scoped to the Anthropic workspace, not to an end user, and the Files API is not offered on Bedrock or Vertex, so the provider-native path has no equivalent on those routes.

Uploaded and generated files therefore have two different lifecycles, and the current design treats them as one.

That distinction has already shipped, which changes the cost of acting on it but not the answer. Otari is pre-1.0, so re-engineering a surface is a minor version bump rather than a deprecation window, and these releases are days old on a surface only standalone deployments serve. Shipped is not settled.

What this issue asks for

A written decision, referenced from both #1185 and #1470:

1. Where the bytes live, per direction (uploaded, generated), rather than one answer for both.
2. What a provider copy is: a cache with an expiry, or the record.
3. How a hybrid data plane, which has no database and no store, moves bytes. This is the same question as Decide how a data plane reaches the code-execution backend: control-plane proxy, or a scoped grant #1603 asks for the code-execution backend: a control-plane proxy, or a scoped grant such as a presigned URL. FileStoragePort's s3 and fsspec adapters make the grant shape natural, and answering the two separately risks two different answers to one question.
4. Whether feat(files): support provider-neutral Files API support for hybrid mode #1185 lands on top of the released answer, replaces it, or is withdrawn.

Not blocked on this

#1483 is a structural move of the code that already implements the released answer. It changes no behavior about where bytes live, and it leaves a thin route over a service rather than a 540-line route, which is the easier shape to reconcile into either way.

Sources for the provider facts above are Anthropic's and OpenAI's own API documentation, read 2026-09-22.

[peteski22]

Decision

Answering the four points this issue asks, in its order. The transport answer is #1603's and is not restated here.

The premise this issue starts from is not quite right

This issue says uploaded and generated files have two lifecycles, so one answer about where bytes live cannot fit both. The asymmetry is real, but it does not reach where the bytes live.

Once Otari's store is the source of truth, an uploaded file's provider copy is never read back, so Anthropic refusing to serve it costs nothing. A generated file is different, but the difference is a race in how its bytes arrive, not where they end up.

So: one rule about location, one caveat about timing. The timing caveat belongs to the Act leg of #1603, which is exactly where a container expiry belongs.

1. Where the bytes live

Otari's store is the source of truth in both directions.

An uploaded file goes from the caller into Otari's store. No provider is involved, and an upload never resolves a model credential. A provider copy is made later and only when a provider-native feature needs one.

A generated file is produced by the provider, so the data plane reads it back and writes it into Otari's store before forwarding the response or the event that mentions it. That read races the provider's container expiry, which for an OpenAI code interpreter container is about 20 minutes after last use, so the copy is best effort and bounded by count, bytes and a deadline.

One consequence, stated plainly rather than discovered: a generated file whose copy fails is announced with an ID that returns 404. This decision does not make that acceptable. It makes it a known Cancel outcome instead of an unexplained gap, and #1569 covers telling the caller rather than leaving them to find out.

2. What a provider copy is

A tracked derivative, never the record.

A provider copy has its own expiry, independent of the canonical file's. A cache hit or a refresh never extends canonical retention, and a provider copy's expiry never shortens it. A deleted, expired or revoked canonical file is never refreshed into a new copy. A provider file ID is never proof of ownership and is not assumed unique across tenants or provider accounts.

Copies are tracked so deletion can reach them. That is #1487.

3. How a hybrid data plane moves bytes

Try-Confirm-Cancel with a scoped grant, as proposed in #1603. This issue adds no transport shape of its own and defers to that one.

#1603 also carries the rule for when not to reach for TCC, which files needs on both sides of it: tenant deletion and credential retirement happen inside one transaction, and the bytes do not.

What follows from it for files:

• The record is created pending before the bytes exist, so there is never a blob with no row. That closes A refused upload leaves its partial bytes in the store, with no row and no sweep to reach them #1590 by construction and turns Reclaim file blobs that no row points at #1597 into a bounded walk of stale pending rows rather than a diff of a whole bucket.
• The control plane allocates the object key, so a grant is scoped to one object rather than a prefix.
• Quota is enforced at Try, before bytes are written, rather than discovered afterwards.
• A grant expires, so it cannot be the durable reference. The stored reference is the object key. A read grant is minted per read, scoped to one key, GET only, short deadline, never logged and never stored on the row.
• There are two kinds of reader. A completion referencing several files is one Try returning several grants, which is what A completion resolves each referenced file on its own, with its own query and its own transaction #1592 becomes. A user-facing download is one file by its nature, so it costs one authorization and one read.
• Confirm is retried to exhaustion with backoff and then parked for reconciliation. It is never swallowed. A usage report lost after its retries is dropped, not parked #1729 is the habit to avoid: a swallowed Confirm on a file means a caller is told an upload succeeded while no record exists, and then gets a 404 from the file it was just given.

Whose bucket

The platform supplies the storage, and the grant is a presigned write to it. The gateway operator supplying their own bucket is a later option, selected by the existing files_backend setting and served by adapters that already exist.

It is the same product option and a different transport, so it is deferred rather than covered here. A control plane cannot presign a bucket whose credentials it does not hold, so there is no grant to receive and the data plane would write with a credential of its own. That inverts #1603's rule that a data plane receives grants and never mints them. Bring your own bucket therefore needs its own authority model, and it is not this design with a setting flipped.

Three reasons. Hybrid keeps its promise of no database and no store, so a data plane gains no new service to run. "The data plane receives grants and never mints them" stays true rather than aspirational, because a control plane cannot presign a bucket whose credentials it does not hold. And "deleted means deleted" is a promise only the bucket's owner can make, which is a product statement that no interface can hide.

A download streams, and never redirects

Redirecting the caller to the presigned URL is cheaper, and it is wrong here.

The decisive reason is that files_backend: local is a directory on disk and has no URL to redirect to. A redirect would make one public endpoint behave differently depending on which backend a deployment configured, which is the divergence this design exists to remove. Streaming behaves identically on every backend and in either plane.

Three supporting reasons. The route keeps control of its own headers, which is what #1605 needs. GET /v1/files/{file_id}/content is a released surface that returns bytes, and a 302 is a visible change to it. And a redirect gives up the access log and the served-byte count for every download.

Streaming is also what lets #1603 hold its rule that a grant never leaves the data plane. A redirect would put a bearer credential with a deadline into a browser's history.

A redirect may later be an opt-in optimization for very large objects on object-store backends. It is never the default, and never something a caller meets by accident.

Streaming keeps #1604, where a download abandoned before its body starts does not close the store's handle promptly, and a grant-backed adapter gives that issue a fourth backend to hold for. A grant expires on its own, so an abandoned download's authority lapses even where its handle does not. That does not fix #1604, but it bounds it.

The storage port changes shape

FileStoragePort.put returns a reference the store itself mints. Try allocates the key before any write. So allocation moves out of the store, and the local, S3 and fsspec adapters change shape with it.

This is the mechanical form of the guarantee above rather than a cost of it, and it is what makes #1590 and #1597 close. It needs its own issue.

4. Whether #1185 lands on top, replaces, or is withdrawn

Replaced. The work continues on a fresh branch from main, in smaller reviewable pieces.

Its security and recovery work is salvaged: ownership and account binding, quotas, retention, revocation and cleanup, and its lifecycle, transaction, revocation and cancellation test scenarios. Those invariants hold against canonical storage even though their assertions change.

What is not carried over: the second public route, the provider-selection header, and the rule that made Anthropic the default flavor. Format selection answers in the caller's shape. It never selects a provider or a credential.

Some of those tests change meaning rather than passing unaltered. A provider copy expiring no longer makes the file a 404. A provider key being disabled no longer removes the bytes. An upload no longer calls a provider at all.

What this does not decide

• The hybrid retention ceiling. That is Decide the hybrid retention ceiling for canonical files #1731.
• Whether a workspace may bring its own credential for a capability. A workspace cannot bring or choose its own web search credential #1724 asks that for web search.
• The deprecation path for deployments configured to route through the control plane today. That is The control plane carries data plane traffic for web search and code execution #1688 question 3.

What this unblocks

#1488, #1487, #1481, #1443, #1377 and #984 were all waiting on this. #1569, #1590, #1592 and #1597 are named above because this decision changes how each is solved.