[WIP] Fix S3 append_log quadratic re-upload of growing buffer

[PawelPeczek-Roboflow]

🚧 Work in progress — not ready for review

Draft PR from a batch addressing findings from an in-depth engineering review. Fixes review finding 46 (Medium, Performance).

The bug

In append_log output mode, S3SinkBlockV1._append_to_buffer appended each entry to self._buffer and then, on every run(), rebuilt full_content = "".join(self._buffer) and put_object-ed the whole accumulated buffer (inference/core/workflows/core_steps/sinks/s3/v1.py:358-371). Within one rotation window of max_entries_per_file entries, entry k re-uploaded all k buffered entries, so total bytes uploaded ≈ S * N*(N+1)/2 — quadratic. It also contradicted the block's own LONG_DESCRIPTION (lines 91-97: "data is buffered in memory and only uploaded to S3 when the limit is reached or the block is destroyed"), and the documented teardown flush had no implementation (there was no __del__). On rotation the old code also reset the buffer without uploading the completed object, silently dropping data.

Root cause

The append path used the S3 put_object call as a per-run "validate credentials / surface errors immediately" mechanism, treating the accumulated buffer as the payload on every call. That makes cumulative upload volume grow as O(n²) per rotation window and diverges from the documented buffered semantics.

Fix

inference/core/workflows/core_steps/sinks/s3/v1.py only:

• Buffer entries in memory; do not upload on ordinary run() calls. A buffered run now returns error_status=False with a "N entries pending upload" message.
• Add _flush_buffer() that uploads the current buffer as one complete object and, on success, clears the buffer / entry count / current key.
• Upload only on rotation (when max_entries_per_file is reached, the completed object is flushed before a fresh key/buffer starts — fixing the previous silent data loss on rotation) and on teardown via a new __del__ that flushes any remaining buffered entries (matching the documented behaviour).
• Stash s3_client / bucket_name / content_type on the instance so the buffer can be flushed at teardown.

No LONG_DESCRIPTION change was needed — the docs already described this buffered behaviour; the code now matches them.

Verification

Standalone simulation of the buffer/rotation/flush accounting (S=10KB entries, N=1024):

• OLD (main): 1024 puts, 5373.95 MB uploaded, 512.5x amplification (matches closed-form S*N*(N+1)/2).
• NEW: 1 put, 10.49 MB, 1.00x amplification. At 1025 entries → 2 puts (one rotation flush + one teardown flush), confirming rotation and teardown both fire correctly.

Full runtime verification within the inference package deferred (WIP); the standalone repro reproduces the review's proven closed-form and the corrected behaviour.

Scope

Focused change; one of a coordinated batch (one PR per finding).

---

🤖 Generated with Claude Code