Repository navigation
Coordinate native stream metadata samples with owner appends while preserving responsiveness #4687
Description
Parent #4680; blocks #4659 and native acceptance under #4537/#4354/#4400.
The current live reader immediately refuses equal-size/different-mtime samples. A legitimate append can update mtime before extending the file: Linux v6.8's buffered ext4 path calls file_modified before generic_perform_write (source). Both latest retained native failures captured that metadata pattern. Their final prefix hashes match earlier admitted prefixes, but this does not prove the hosted cause or contemporaneous byte stability.
First establish a deterministic synthetic regression using the actual reader, real temporary file bytes and an explicitly scripted metadata schedule. Preserve both original failure diagnostics and exact append bytes; do not call this a hosted kernel reproduction. Add an independently reviewed coordination contract and small real advisory-lock controls, including separate processes.
Then implement the smallest explicit owner/reader coordination successor that passes the contract. The candidate mechanism is a shared lock for each metadata sample paired with exclusive locking around every owner write plus flush, using the actual stream inode and separately opened descriptors. Protect the initial compared lstat/fstat pair as well as both passes and final samples. Retain all existing mtime, identity, bounded extent, two-pass, previous-prefix and future-tail checks. Do not replace these with a growth waiver.
Production integration must preserve the responsive pump: bounded contention under existing absolute deadlines, cancellation handling, exact pending-byte order/accounting, no callback grant before journal persistence, and bounded stopped-tail/worker shutdown. Do not add immediate failures on ordinary lock collisions or indefinite blocking. Prove the actual lifecycle behavior before adopting a production protocol; introduce only the state necessary to satisfy it.
All writers and all live readers, including helper subprocesses, must participate. Preserve historical frozen controller sources/manifests and retained baselines; use explicit successor source/profile pins if owner code changes. Saved/final readers do not gain a live-lock fallback.
Adversarial controls must retain refusal of same-size rewrites/touches, rewrite-plus-append, late second-pass mutation, changed prior prefixes, truncation, replacement, links, overflow and short reads. Preserve first-stop/cursor behavior and current time/queue/frame limits. No provider/authenticated/paid generation is involved.
The first local commit may contain only the regression and contract; this issue remains open until production coordination, validation and the required acceptance path are completed or explicitly split into linked follow-ups. Do not push an intermediate change merely to rerun the same known-failing hosted acceptance.
Detailed reviewed proposal: /private/tmp/d4d-goal-resume-5sVO4ntI/native-owner-snapshot-proposal.md, SHA256 5f0f75ff4450407bc3b55c1a838b04a284d566ad3a8967f6a49616c90f9f990d. Latest diagnostic findings: #4680 (comment).
Activity
realmarcin commentedon Oct 8, 2026
The private owner-control layer now passes 228 focused tests: 55 new owner/adversarial controls, 131 prior reader/stream/diagnostic controls, and 42 frozen history-controller controls. There were no failures, errors or skips. Tested local commit:
f1825bce5c1e943789c347298dc519571977d02a; notes-only successor:0c163ef965df12f49a5db6900bf2af91e1fc53f7. These commits are retained locally and have not been pushed.The controls cover actual independent advisory-lock contention, responsive intake/cancellation, journal-before-response ordering, original deadline propagation, short writes, immutable failed-operation evidence, loader lifecycle restoration and bounded shutdown. Review findings #4695, #4697, #4698, #4699 and #4700 have regression coverage. All 12 frozen controller modules and their manifest are unchanged.
JUnit: 228 cases, 70.611 seconds, 37,917 bytes, SHA256
2e68c695994ab4bb168fd3ec2047d3adbbb115e194aea8d11e98880d7275564d. Complete before/after inventories agree for all 9,232 actual files, including hidden/ignored files and modes, excluding.git. The evidence notes distinguish coordinator invocation records from JUnit outcomes.This is an unrouted private layer. Production source/profile binding, runner integration, every live reader including helpers, captured-only evidence reconstruction, and full acceptance remain pending. Its close operation currently retains a separate failed suffix; the normal EX-protected tail append is still an explicit integration obligation. The next bounded step is the fixed source-profile and verified-loader boundary, without activating a selected execution route.
These are ordinary local software controls. They establish neither the hosted failures' cause nor 900-second native acceptance. This issue and the broader integration/acceptance issues remain open.
realmarcin commentedon Oct 8, 2026
The next private source-profile slice passed 201 focused tests: 65 new profile controls and 136 existing owner/controller/registration controls, with zero failures, errors or skips. Tested local commit:
c1607c5cba2cba6993eb58c256e80d40a2f1f7d8; independently verified notes-only successor:2ed2a2243b9745952b2afdef6e733c2c990d9c69. These commits remain local and unpushed.The slice adds fixed coordination/control source pins and a selected loader lifetime. Controls expire at context exit even when Python caches preserve the same modules. Startup refusal stays terminal after the graph is restored, and the profile-defining source cannot disappear from its own selected declaration. Findings #4701–#4703 have regression coverage, including actual copied Git repositories and ordinary pipes. Existing execution registrations reject injected profile identities, so no partial runtime route is activated.
JUnit records 393.885 seconds and 32,117 bytes, SHA256
bcbbead348ef6bd9c79f7eba4448ebe57e1318b9d5296e6206c8c3a9d5f90fee. Before/after inventories are identical across all 9,237 actual files and their modes, including hidden/ignored entries outside.git. Earlier 131-case and 228-case evidence remains intact; these overlapping software runs are not independent native executions.The remaining native work is runner/deadline integration, normal EX-protected shutdown writes, all live readers/helpers, captured evidence and activation, then actual acceptance. Separately, #4704/#4705 address pre-existing completeness and Git-replacement gaps in the common committed-source proof; the current private result does not claim those broader repairs. No hosted-cause, scientific-acceptance or 900-second-acceptance conclusion follows from these passes.
realmarcin commentedon Oct 8, 2026
The private runner/append-cleanup slice is implemented and reviewed locally. It binds the consumed instruction/policy and dispatch inputs to the supplied C/E/S evidence, preserves the original attempt deadline and open descriptors through cleanup, and uses one internally bounded 0.25-second tail/append/join allowance. Review findings are tracked in #4707 and #4708.
Validation: the first 298-case run at
20d24b0epassed all 243 existing regressions and 49 new cases; six new cases correctly hit the frozen file-policy guard because the synthetic peer had invalid initialization metadata. The fixture-only repair (#4709) atefc92dc5then passed all 55 new cases. Production and frozen dependency bytes are identical between those runs. Both full inventories were unchanged across all 9,242 checkout entries.Evidence notes are retained in the local, unpushed notes-only successor
71373f70undernotes/native_owner_snapshot_2026-10-08/RUNNER.mdandrunner_validation.json. Independent checks verified the two JUnit results, 28 selected source pins, frozen manifest/dependencies, 15 external artifacts and the fixture-only/notes-only differences. This is not a single 298-case all-pass run.The scope remains private software controls with ordinary Python peers. Source-roster integration, independent-helper clock-domain verification, live-reader/capture/replay integration, public C/E/permission routing and full native acceptance remain pending. No native/provider/auth execution occurred, and this issue and the review issues remain open.
realmarcin commentedon Oct 8, 2026
Two further prerequisites have been validated locally. The integration and native acceptance obligations remain open.
The complete original-Git-object source proof from merged PR #4706 is integrated with the private selected profile and runner. At
a7084dafb1e9128a6f32d882afc2a8e776c58ea2, 159 of 160 selected tests passed, including both complete offline helper workflows. The sole failure was a stale test assertion: the untracked fixture file was correctly rejected earlier by the complete roster check. Review #4711 records the exact test-only correction at9e62296522bbfacce3365661bbae2a1e6d121903. All 65 affected profile cases then passed; the other 95 earlier cases passed at identical production bytes. These are two runs, not one 160-pass run. Both source-preservation checks passed.The private clock-domain module at
79ac69aa64db2dd311bc4f297b50a9e3ef2244eepassed 79 synthetic schema, syscall and lease-lifetime controls. Its real macOS smoke was initially skipped because the sandbox denied the boot-session query. A focused retry with the required local access passed: the parent and an isolated ordinary Python child independently acquired equal domains and closed their leases. Both runs preserved all 9,250 source files. Real Linux namespace and container behavior has not been validated by these runs. Review #4710's thread-ownership contract is implemented; it remains part of the pending integration. Notes-only successor6fa966ba71f832771c360f81ca60a47a4c1bfdd3records the separate results and source/artifact hashes.Next is the reviewed clock-bound owner-record version and fixed source role, then all live readers and captured owner/stop/failure evidence. Public composition, permission and execution routing must be integrated before fresh hosted offline acceptance. The original deadline, first-stop behavior, frozen controller sources and historical evidence remain intact. No native executable, provider, authenticated or paid scientific run occurred.
realmarcin commentedon Oct 8, 2026
The private clock-bound owner integration is locally implemented and independently reviewed at
68984e4d9fb7e9f10a3ccb73d6faff7e897d430a; evidence-only successor785a5794d0bae5d558a5aed9fbfb04e995abd0f9.One serialized run passed all 380 tests (zero failures, errors or skips). The complete before/after inventory matches for 9,255 entries; 45 committed source/evidence pins and eight external artifacts were verified. This covers ordinary fixture-owned Python peers and deterministic clock-domain substitutions. The separate real macOS clock-domain smoke result remains separately reported.
Review issues #4712, #4713 and #4714 were addressed locally: exact budget/lease lifetime, bounded cleanup timing and original worker failure handling. They remain open until integration. These results do not establish full native acceptance.
Next is the private live-reader primitive: metadata sampled under SH on the same stream descriptor, both finite read passes outside the lock, exact owner/selection associations, and stop/deadline checks during lock contention. The reviewed plan incorporates #4715. Public execution, worker/helper propagation and terminal artifact publication remain subsequent integration steps.
realmarcin commentedon Oct 8, 2026
The private live-reader primitive is locally implemented at
89a1e9aa11d332dbb908ff27a490443a3bfe92a6; notes-only successor4a4f5ffb9e578020ab0986103cb61d6ee98095bdrecords the evidence.One serialized run passed 535 tests across 20 modules, with zero failures, errors or skips: 76 new reader cases and 459 existing regressions. Before/after inventories match all 9,258 entries. Independent review verified 54 committed source/evidence pins, nine external artifact pins and the full current inventory. The existing absent/default reader body is unchanged apart from optional dispatch.
The selected branch checks captured selection/E/C/started/owner associations, samples compared metadata under SH, reads both finite byte passes outside the lock, and preserves the original deadline/stop checks through result construction. Review issues #4715, #4716 and #4717 were addressed locally, including selection joins, final hashing admission, exact cleanup ownership and primary-error preservation.
A separate known gap remains: #4718 records that a copied live parent Context can currently be used from another thread with a fresh bound budget. The suite rejects moving an already-created reader across threads, but does not establish protection against that different path. Work is now implementing original-loader thread affinity and a fresh reader-only scope inside each actual worker operation, preserving the existing worker deadline and cancellation events.
These are synthetic private software fixtures using real file descriptors/locks and an invented clock backend. Public adapter/helper/capture activation, saved reconstruction, terminal artifact publication, actual receipt/provenance write checks and full native acceptance remain pending. No real native/provider or scientific campaign is credited by this result.
realmarcin commentedon Oct 8, 2026
The next local integration layer is implemented and independently reviewed: actual native reader workers receive fresh, thread-bound selected-owner scopes, and scope checks retain the fixed origins of inherited bare modules. This addresses the local defects recorded in #4718 and #4719.
Validation at implementation commit
2461a1b311f6804660070d512ae86efcf2208ab3: 583 passed, 0 failed, 0 errors, 0 skipped, including 48 new worker-scope cases and the prior 535-case regression set. The positive cases use ordinary isolated Python protocol peers and the actual runner/observer/classifier machinery with synthetic clock-domain declarations. This does not establish real native/provider acceptance. Existing deadlines are unchanged, and retained-evidence inventories matched before and after the run.The notes-only successor is
19f3ecefbc80142f548bba2079c857ea2d28f5e1, with evidence innotes/native_owner_snapshot_2026-10-08/WORKER_SCOPE.mdandworker_scope_validation.json. Root and independent review checked the notes against the tested source, artifact hashes, all 583 unique JUnit cases, and the retained inventory. These commits remain local while public integration is unfinished; #4718 and #4719 remain open for that integration.Next: validate selected-owner initialized observations and reconstruct detached saved initialization evidence. Review found that returning the existing mutable captured-run object would expose direct publication paths before live publication checks are integrated, so this step will return an immutable evidence-only carrier. Live adapter/helper activation, terminal evidence retention, complete publication gating, and fresh hosted offline acceptance remain required. The existing hosted same-size/different-mtime failures remain unresolved; this result is not a new acceptance or performance claim.
realmarcin commentedon Oct 8, 2026
The private captured-initialization slice (#4720) is implemented and reviewed locally, including the runtime-join correction in #4721 and fixture correction in #4722.
At local commit
5efca813a2b159f8fa7cbdf51dfd6fa55224f785, all 101 focused cases passed: 11 basic initialization, 78 independent adversarial, and 12 default saved-results cases, with zero failures/errors/skips. The saved carrier contains detached captured evidence only; it cannot supply run/effect/receipt/publication authority.The evidence retains the earlier wider run separately: 370 passes and one fixture failure at
4a6d1cfe, including all 288 existing regression cases passing. It also retains the pre-fix witness whose passing result demonstrated two admitted runtime contradictions. Those outcomes have not been relabeled as corrected passes.The corrected preservation subprocess returned 120 after writing a complete matching after-inventory; its precise exit cause was not captured. A fresh metadata-only audit exited zero and matched both inventories, and independent verification rehashed all 9,266 source entries. The unresolved process status and successful independent preservation checks are both recorded. The notes-only successor is local commit
6ce623c06dfb095f7f914b53f204581ec93530da, undernotes/native_owner_snapshot_2026-10-08/INITIALIZED_CAPTURE.mdandinitialized_capture_validation.json.Next, #4723 implements private terminal retention. Its first concrete adversarial finding is #4724, an impossible saved owner/control lifecycle that needs an explicit implication guard and regression. These issues and this parent remain open through the remaining public composition/execution/adapter integration and acceptance obligations. This local validation does not establish full native completion or scientific eligibility.
realmarcin commentedon Oct 8, 2026
Terminal-retention engineering (#4723) is implemented locally and adversarially reviewed. It preserves immutable post-run evidence with one-use publication and a saved-only validator; its evidence does not authorize execution or certify full native acceptance.
Three review defects were fixed before validation: owner/control lifecycle consistency (#4724), failure/normal-close consistency (#4725), and stopping publication after its first failure (#4726). The initial 286-case run passed 285 cases, including all 247 existing regressions. Its one failure was a no-op adversarial test mutation (#4727): the actual captured exception summary was already
capture_refused. The test now asserts that positive refusal shape and rejects both real contradictory variants, with an explicit guard against no-op mutations.The corrected test-only commit
11fb4d617c5acf4a24c5439728d3bda85b8fee6epasses both complete terminal modules: 39 passed, zero failures/errors/skips (32.16 seconds). Production bytes are identical between runs. Both runs preserved the entire source inventory, including ignored/hidden files other than Git internals. Original failure and corrected evidence are retained separately; counts are not presented as one combined run.The next coherent public integration is tracked in #4728: actual C/E reconstruction and dispatch, callback and helper lifetimes through final writes, and full saved-history/terminal validation. Those remain pending, as do hosted and real native acceptance. These changes are local; #4687 and the review issues remain open until public integration.