Store "size" for s3-ambient adjuncts

[donaldgray]

What does this change?

Fixes #1218

Resolves an issue where the "size" property wasn't stored for adjuncts that were ingested with S3-ambient strategy. This was a side-effect of the ingestion process being skipped, therefore the adjunct is never read. The solution is the HEAD the object to read the content-length.

An additional complexity was differentiating between the adjunct.size property indicating that we store actual bytes. The existence of "origin" was used for this purpose but that no longer fits so I added a new optimised prop to adjunct entity. This is not reflected in the hydra model as it's internal only.

The various properties are now

• Adjunct.Size - this stores the provided size of the adjunct, or calculated size.
• Adjunct.Optimised - stores whether a hosted adjunct (it one with "origin") is at an optimised origin. If optimised = true, DLCS stores no bytes.
• CustomerStorage.AdjunctSize - accumulative size of all adjuncts for customer. This was correctly counting values but was being cleared on asset reingest.
• ImageStorage.TotalSizeOfStoredAssets - total size of all adjuncts across space/customer. On adjunct re-ingest this was double counting adjuncts size. This would have been resolved by recalc run but resolved while I was making changes.

Database Migration

Note

Details of migration.

• Migration name: 20260709152931_Add Optimised to adjuncts
• What it does: adds column optimised to adjuncts table, defaulting to false
• Breaking Change? No

[donaldgray]

Note

As discussed internally with @tomcrane and @JackLewis-digirati , will merge this without review due to team annual leave.

Confirming I used Claude Opus 4.8 for review, which noted a few issues that I resolved with last couple of commits. Issues spotted where

Three things I'd want changed before merge.

1. Optimised non-annotation ingest now fails on any S3 error that isn't a 404. Engine/Ingest/File/FileChannelWorker.cs:213 calls bucketReader.GetObjectHeaders, which throws HttpException for every AmazonS3Exception other than NotFound (DLCS.AWS/S3/S3BucketReader.cs:78). Nothing catches it in GetOptimisedAdjunctSize, so it lands in the outer catch (Exception ex) at line 113, sets adjunct.Error, and returns Failed. Before this branch, that path returned Success unconditionally.

The scenario that worries me isn't exotic: when a caller lacks s3:ListBucket on a bucket, S3 returns 403 AccessDenied rather than 404 for a key that doesn't exist. Granting only s3:GetObject is the common shape for a customer origin bucket. Any such customer gets working optimised adjunct ingests turning into failures. The plan's own decision was "if size can't be determined: log a warning and succeed" — that holds for 404 but not for 403, and there's no test covering the throwing path (FileChannelWorkerTests only pins the not-found case). Wrapping the GetObjectHeaders call in a try/catch that logs and returns null restores the intended leniency, and also protects against Optimised being set on a non-S3 strategy where RegionalisedObjectInBucket.Parse could hand back a nonsense bucket from an arbitrary https origin.

2. if (newContentSize is > 0) treats zero bytes as "unknown". At FileChannelWorker.cs:150, a genuinely empty adjunct on the non-optimised path (adjunctInBucket.AssetSize == 0) leaves adjunct.Size at its stale previous value while StoredSizeDelta correctly moves the contribution to zero. On the next ingest, prevContribution is read from that stale Size and decrements again, so TotalSizeOfStoredAdjuncts drifts downward. Size also never reports 0 for an empty file, which is the thing the ticket asked for. The non-optimised call site always passes a non-null value, so the cleanest fix is if (newContentSize.HasValue) for the assignment and else if (isOptimised) for the warning — or tighten the signature so the two call sites can't be confused.

3. Dead code the change leaves behind in EngineAssetRepository. GetImageStorage (line 104, plus the interface member) now has no callers since AdjunctIngester stopped seeding. More importantly, IncreaseCustomerStorage's isAdjunct parameter is now permanently false — adjuncts pass a null ImageStorage and never reach it — and the TotalSizeOfStoredAdjuncts = cs.TotalSizeOfStoredAdjuncts + imageStorage.AdjunctSize term at line 196 survives only because asset ingest happens to build a fresh ImageStorage with AdjunctSize = 0. That's the exact over-count you just fixed, sitting one seeding change away from coming back. I'd drop the parameter and the adjunct term outright.

Two smaller notes. AdjunctIngestionContext.WithStoredSizeDelta assigns rather than accumulates; only FileChannelWorker implements IAdjunctIngesterWorker today, but IngestExecutor loops over workers, so += would be the safer default. And separately — pre-existing, not introduced here, but newly relevant — UpsertImageStorageRecord calls imageStorages.Update(imageStorage) on the whole row, so an asset re-ingest writes AdjunctSize = 0 and wipes the per-asset tally that AdjustAdjunctSize now solely maintains. CustomerStorage is unaffected, so it's a divergence between the two records rather than a billing error, but it's worth knowing about given the recalculators follow-up.

The test coverage is otherwise good — all four optimised↔normal transitions pinned at the unit level, and the 2048 + 4096 = 6144 integration regression test directly guards the over-count fix.