engine: XMILE conveyor and queue stock types#869
Open
bpowers wants to merge 33 commits into
Open
Conversation
First of five build-sequence steps for XMILE conveyor support (docs/design/conveyors.md §12). This step adds the datamodel/proto/XMILE representation and fixes the import-then-export corruption where the <conveyor> block, leak flows, and isee:spreadflow inputs were silently dropped. Runtime simulation lands in later steps. datamodel: add Conveyor, Leakage, and SpreadFlow types. These ride on Compat rather than as first-class Stock/Flow fields: non_negative -- the other mutually-exclusive XMILE stock option -- already lives there, and it avoids churning ~314 struct-literal construction sites. proto: new Conveyor/Leakage/SpreadFlow messages and fresh Compat fields 6/7/8 (backward compatible -- absent fields decode to None). xmile: read/write the <conveyor> block, both leak encodings, leak_integers, leak zones, and isee:spreadflow + <isee:distrib_eq>. Round-trip verified idempotent on all three vendored fixtures. Known follow-ups: conveyor data is not yet plumbed through the JSON/SDAI format, and ignore_earlier_zone_losses is not yet persisted to XMILE.
Second build-sequence step (docs/design/conveyors.md §12): the pure functional-core runtime engine, a faithful Rust port of the executable reference test/conveyors/reference_prototype.py. Not yet wired into the VM -- that is the next step -- so this lands as a self-contained, directly-testable module. conveyor.rs implements the §4.3 two-phase per-DT pass (arrest, latch, leak, exit / admit, shift, insert), linear and exponential leakage with zones and the staggered-zone retained-profile r_k (§5), integer leakage (§5.4), capacity and per-time-unit inflow limits (§6.3), variable transit with cohort merging (§6.2), discrete quantized admission with per-inflow attribution (§6.4), steady-state and explicit-list initialization (§7), and time-unit-block lumping for discrete conveyors. It holds only belt state and does no expression evaluation: every dynamic quantity is passed in per step, so chains and cycles need no topological ordering. conveyor_tests.rs reproduces scenarios S1-S15 from the spec's §15 as oracle tests (asserting the conservation identity every step) plus integer-leak, arrested-destination, and explicit/discrete-init property tests. An adversarial review confirmed the engine is bit-identical to the reference prototype on full per-step trajectories (S2/S9/S13).
Wire the standalone conveyor engine (docs/design/conveyors.md) into the simulator so real XMILE conveyor models simulate. A conveyor is VM-native, not bytecode (spec 9.3): conveyor_compile::expand_conveyors rewrites each belt into hidden parameter auxes ($conv$stock$len/$cap/$inlim/$sample/ $arrest and per-leak $conv$leak$flow$frac) plus placeholder-0 driven flows, so the parameters flow through normal layout/compilation into ordinary data slots; resolve_plans maps the synthesized names to offsets; and a native two-phase pass (run_pass) runs between the Flows and Stocks phases each Euler step. Because the pass writes the driven-flow rates before stock integration, the conveyor stock integrates to the sum of belt contents through the ordinary Stocks phase -- no special-casing (the spec 4.3 conservation identity guarantees dstock = admitted - out - leak). The belts are initialized in run_initials after one extra flows evaluation, since the synthesized parameter auxes are not in the initials runlist. Make the production path safe and reachable. The ordinary compile path integrated a conveyor stock as a plain INTEG, which silently mis-simulates real Stella files (their primary outflow carries a placeholder 0, so no empty-equation error fires). expand_conveyors now clears the conveyor/leakage markers so the expanded model is plain, and compile_project_incremental rejects any surviving conveyor marker with ConveyorNotExpanded -- so every ordinary path (including wasmgen) fails loudly instead of producing a valid-but-wrong result. libsimlin's simlin_sim_new routes conveyor models through the conveyor build path (build_compiled) and caches the compiled sim plus the resolved plans so reset can recreate the VM and re-attach the pass. An equation that reads a conveyor-driven flow by name would see the pre-pass placeholder 0, so expansion rejects it with ConveyorDrivenFlowRead rather than mis-compute. Conveyors are Euler-only (spec 9.4), enforced at build time and guarded in run_to.
Implement the isee:spreadflow inflow placement methods (conveyors spec 8) in the conveyor runtime. At insert time each admitted component -- the lumped conveyor-driven volume and every equation inflow -- is distributed across belt slats per its placement into per-slat shares, which are then inserted once per slat. Because a cohort's linear-leak schedule is linear in its volume, shares that land on the same slat (same insertion depth d_c = i+1) merge exactly into one cohort_schedule call, so mid-belt shares get their leak budget prorated to the zone slats they will traverse (spec 8 / 5.1). The default beginning placement reduces to the prior single-entry insert, so non-spread models are byte-identical. Placement threads from the flow's isee:spreadflow through InflowMeta/InflowPlan to the phase-B pass. beginning/even/dest are supported; dist (needs the distribution graphical function evaluated per slat) and source (needs upstream-leak coupling) are recognized and rejected loudly with ConveyorSpreadflowUnsupported rather than silently placed at the entry.
Complete the isee:spreadflow methods (conveyors spec 8). dist distributes an inflow as A_i = A * w_i / sum(w) over the entry-path slats, with the weights w_i = max(0, g(x_i)) evaluated per step (the entry depth d, and thus the slat positions x_i = 1 - (i+0.5)/d, vary with a time-varying transit). The <isee:distrib_eq> is stored as an opaque string; resolve_dist_profile accepts two representable forms -- a graphical-function variable name (evaluated with the engine's own continuous LOOKUP interpolation, reusing crate::vm::lookup so dist matches how a LOOKUP call site would read the same profile) or a comma-separated numeric array (indexed floor(x_i * m)) -- and rejects an empty distribution, an inline expression, or a name without a graphical function loudly rather than silently placing at the entry. The GF is sampled at the raw x_i in [0,1] against the table's own x-values, which is correct for the conventional [0,1]-scaled density profile. source mirrors an upstream leak: for a conveyor-driven inflow that is itself an upstream leak flow, each upstream slat's leaked volume is placed on the nearest downstream slat by fractional position (ties toward the exit), reproduced exactly as a Dist weight vector. This needs Phase A to expose a per-leak, per-slat volume breakdown (PhaseAResult.leak_slat_vols); integer leakage keeps it in sync with the requantized totals so it still sums to leak_vols. The lumped conveyor-driven inflow input (conv_vol) becomes a per-inflow (volume, placement) list so a source-coupled leak carries its own placement.
An arrayed conveyor stock (conveyors spec 10) is N independent conveyors, one per array element, each with its own transit time, leak flows, capacity, and inflow limit. Detection: the conveyor stock's equation carries array dimensions. expand_conveyors then synthesizes the hidden parameter auxes ($conv$stock$len/$cap/$inlim/$sample/$arrest and per-leak fractions) as apply-to-all over the stock's dimensions instead of scalar, and the driven-flow placeholders as apply-to-all "0", so every element gets its own len/cap/fraction/flow slots. A shared per-attribute expression is the only arrayed form the datamodel can represent (Conveyor holds one string per attribute); per-element values come from array references in that expression. resolve_plans flattens each arrayed meta into one scalar ConveyorPlan per element, so run_pass, init_belts, and ConveyorState are unchanged and a scalar conveyor stays the 1-element degenerate case (byte-identical). Per-element offsets are resolved by reconstructing the compiler's canonical name[elem1,elem2] keys via the same SubscriptIterator the flattened-offset builder uses (row-major, honoring declared element order). A held-exit destination links element e to element e of the downstream conveyor. An arrayed conveyor over an undefined dimension errors with ConveyorArrayedDimensionUnresolved. Container access (conv[j], SUM/SIZE over belt contents) and cross-shape conveyor chaining remain follow-ups.
Reading a conveyor's belt slats in an equation -- conv[j], or a single-argument reducer/SIZE over a conveyor (conveyors spec 10 container access) -- cannot be compiled today: expand_conveyors rewrites a conveyor into an ordinary INTEG stock and clears the marker before compilation, so the compiler and bytecode VM never see a conveyor and the stock slot holds only the sum of slat contents. The physical belt length is also runtime-dynamic and unbounded (a variable <len> can grow, and a transit shrink leaves a stale tail), so there is no compile-time-fixed array to reduce over. Left as-is these forms silently return the belt total (SIZE -> 1, MEAN -> total, etc.), exactly the silent-wrong class the spec forbids. Detect container-position use during expansion and reject it loudly with ConveyorContainerAccessUnsupported. The boundary rejects: any subscript of a scalar conveyor; over-subscripting an arrayed conveyor past its array dims; any reducer/SIZE whose argument subtree references a scalar conveyor (bare or wrapped, e.g. MEAN(belt/x)) or selects a single belt of an arrayed conveyor (SUM(conv[a])); and a bare arrayed conveyor under any reducer except SUM. It allows the operations that resolve correctly over per-element belt totals: a bare read, a single-element read conv[a], SUM over an arrayed conveyor or an array expression of it, and two-argument MIN/MAX. Full native container-access reads are the next step.
Support container access into a conveyor's belt from equations: the reducers SUM/MEAN/SIZE/MIN/MAX/STDDEV over a single belt (a bare scalar conveyor, or a single-belt subscript of an arrayed one), and conv[j] / conv[elem, j] for a compile-time-constant j (1-based from the exit; out-of-range or a stale post- shrink tail yields NaN). This replaces the interim loud rejection for exactly these forms. The compiler never sees a conveyor -- it is expanded into an ordinary stock whose slot holds only the sum of slat contents -- and the physical belt length is runtime-dynamic and unbounded, so there is no fixed array for the bytecode to reduce over. Instead the conveyor pass computes each container result natively from the ConveyorState and publishes it into a synthesized hidden stock's slot at step-start: at the top of the Euler loop before the flows eval, and in run_initials after the belts are initialized. A stock is the one slot kind the flows phase reads but never recomputes, and a no-inflow stock is carried unchanged by the stocks phase, so the published start-of-step value survives the flows eval and is visible to any equation that reads it -- an ordinary aux would be recomputed to a placeholder and clobber it. The reader-to-container dependency is a stock read, which is treated as start-of-step, so the spec's visibility rule (the belt as it was before this step's leak/exit/insert) holds for free. Each supported container subexpression is rewritten in place (AST substitution plus reprint) to reference its synthesized stock, arrayed over the conveyor's dimensions for an arrayed conveyor. The same rewrite is applied to the conveyor parameter and leak-fraction expressions, so container access there is likewise supported (as a lagged start-of-step dependency) or loudly rejected, never silently mis-bound to the scalar stock. Residual forms that genuinely need the per-slat vector -- a reducer over an expression of the belt, a dynamic index, or ranges/wildcards over slats -- keep the loud ConveyorContainerAccessUnsupported rejection.
A conveyor is a stock with non-INTEG dynamics, but the LTM flow-to-stock link-score formula assumes plain INTEG under Euler, so a conveyor's internal slat dynamics are not scored correctly. Rather than emit a silently-wrong score, LTM analysis over a model containing a conveyor now degrades loudly: one Warning per conveyor stock, carrying ErrorCode::ConveyorLtmDegraded and naming the conveyor, on the same diagnostic path as the auto-flip advisories. Full LTM-through-conveyor attribution is a separate enhancement. The warning is emitted from model_all_diagnostics, inside its existing ltm_enabled branch, iterating that model's own conveyor stocks. That query is drained exactly once per model by collect_all_diagnostics and -- unlike model_ltm_variables -- is never invoked transitively across a module edge, so the warning fires exactly once per conveyor even when the conveyor lives in a sub-model referenced as a module (the transitively-reachable path would otherwise double-report; the same latent duplication for the other model_ltm_variables warnings is tracked as #866). It is a Warning carrying the real error code (DiagnosticError::Model, not Assembly, which would surface as NotSimulatable), so the model still reports simulatable and still runs through the conveyor build path.
Check each conveyor-block expression against its expected units: <len> against the model time unit t, <capacity> against the conveyor stock's units S, <in_limit> against S/t, a linear leak fraction against dimensionless, and an exponential leak rate against 1/t. <sample> and <arrest> are deliberately not checked -- they are nonzero conditions (a predicate like TIME > 10 type-checks dimensionless), so requiring t would reject valid expressions. A mismatch is a best-effort Warning, never a hard error; the model still simulates. The conveyor parameters live as datamodel expression strings on Compat, not as ModelStage1 variables, so the ordinary per-variable unit check never sees them. The check runs in check_model_units, which has datamodel access: for each conveyor stock it synthesizes one hidden aux per parameter (the parameter string as its equation) into a clone of the stage-0 model, lowers that clone, and evaluates each expression through the existing UnitEvaluator. The clone is used only for these checks -- inference and the ordinary unit check run on the un-augmented model, so nothing about existing unit checking changes. A conveyor stock with no declared units, or a parameter referencing a variable with unknown units, is skipped, consistent with the rest of the unit subsystem.
Author the queue design doc that conveyors.md 11 references as the "companion queue document." It specifies the XMILE queue stock type (FIFO batch tracking, driven outflows, priority/overflow outflows, container access, arrayed queues) and the queue side of the queue-conveyor coupling, mirroring the conveyor spec's structure and its Stella-wins precedence rule. Key design decisions documented (with Stella-probe flags where OASIS prose is silent): a batch is a scalar volume (no age, unlike a conveyor slat); queue outflows are engine-driven, not user equations; an unconstrained outflow (to a cloud or regular stock) empties the queue while a conveyor-bound outflow is throttled by the conveyor's capacity/inflow-limit; append-then-serve ordering; overflow drains exactly what a capacity/inflow-limit/arrest block rejected; container access reuses the conveyor mechanism verbatim; and the runtime mirrors conveyor_compile (pre-pass expansion, QueueNotExpanded guard, Euler-only, per-instance side table).
…phase 1) Add the datamodel and format-surface representation for XMILE queue stocks, mirroring how conveyors are represented one feature over. A queue is a bare marker (no options, XMILE 4.2): Compat gains queue: Option<Queue> on a stock and overflow: bool on a flow (the <overflow/> outflow property), with proto fields appended (Queue message; Compat queue = 9, overflow = 10 -- nothing renumbered, since a DB holds serialized instances). The XMILE reader recognizes <queue/>, <overflow/>, and the <uses_queue> header; the writer re-emits them (<uses_queue overflow="true"/> when any queue outflow is an overflow), which fixes the import-then-export corruption of a Stella queue model. serde carries both fields datamodel<->proto symmetrically. This is representation and round-trip only; no runtime or simulation yet (the <queue/> marker will be rejected loudly by the ordinary compile path until the runtime lands, the same staging conveyors used). See docs/design/queues.md 11.
Add queue.rs: QueueState, a FIFO of batch volumes (front = oldest) that is the pure functional core of queue simulation, mirroring conveyor.rs. It exposes init_empty/init_from_value (a positive initial value seeds one front batch), admit (append one back batch of max(inflow,0)*dt, skipping a non-positive volume so the count never accrues empty batches), take_from_front (the serving primitive: pop whole front batches that fit and split the boundary batch on a partial take, leaving a strictly-positive remainder), serve_unconstrained (the cloud/regular-stock drain that empties the queue), and the container-access accessors batch_contents/batch_count/batch/total. take_from_front is the one primitive every outflow shape composes from: unconstrained drain is the whole queue, the future conveyor coupling is a take of the conveyor's admission budget with the batch rules layered on top, and an overflow is a take of the redirectable front volume. serve_unconstrained sums then clears rather than requesting total() or INFINITY through take_from_front: the former drifts when a tiny batch sits behind a huge one, the latter strands batches behind a non-finite one (INFINITY - INFINITY = NaN), while sum-then- clear empties unconditionally and returns exactly the batch sum. No VM or datamodel coupling yet; the constrained, overflow, and coupling cases build on take_from_front in later phases. See docs/design/queues.md 4, 7, 8.
Wire the queue runtime core into the VM and compile path, mirroring the conveyor integration. queue_compile expands each queue stock into an ordinary INTEG stock with placeholder-0 driven outflows, clearing the <queue/> marker, so the stock integrates through the ordinary Stocks phase from the driven rates (conservation identity, no special stock handling). A native run_queue_pass runs between the Flows and Stocks phases: it admits the summed inflow as a batch and serves the unconstrained (cloud / regular-stock) outflow by draining the whole queue, writing the driven outflow rate. Queues and conveyors coexist: the VM carries both plan sets and runs both passes, and a conveyor-only model stays byte-identical (the queue pass is a no-op on empty plans). Scoped to a scalar queue with unconstrained outflows; container access, arrayed queues, overflow, and the conveyor coupling build on this in later phases. A surviving <queue/> marker on the ordinary or wasmgen compile path is rejected loudly (QueueNotExpanded), queues are Euler-only (QueueNonEulerMethod), and an equation reading a queue driven outflow -- which would see the pre-pass placeholder 0 -- is rejected loudly (QueueDrivenFlowRead), each mirroring the conveyor guard. libsimlin routes a project with a queue or a conveyor through the expanding build path and re-attaches both plan sets on reset. See docs/design/queues.md 4, 10.3, 10.7.
…spec phase 4) Two capabilities, both mirroring the conveyor work. Arrayed queues: an arrayed queue is N independent FIFOs, one QueueState per element. resolve_plans flattens each arrayed queue into N scalar plans keyed by the compiler's canonical name[elem...] row-major offsets, reusing the conveyor's element_subscripts_for_dims helper; run_queue_pass/init_queues/QueueState are unchanged (scalar is the one-element case). Container access: queue[k] (k-th front batch, 1-based, out-of-range -> NaN), SUM/MEAN/MIN/MAX/STDDEV over the batch-volume vector, and SIZE(queue) = batch count. This shares the conveyor container-access machinery: the subexpression rewrite, the ContainerKind enum, the synthesized-hidden-stock publish pattern, and the residual loud-rejection are parameterized by a ContainerNaming so both conveyors and queues drive them, differing only in the source vector (a conveyor supplies its slats exit-first, a queue its batches front-to-back). Conveyor numerics stay byte-identical: the conveyor keeps its own state-reading container_value, and only the queue routes through the shared container_value_from_slice. The queue pass publishes each container result into its synthesized stock at step-start (top of the Euler loop and in run_initials), so an equation reading SUM(queue) sees start-of-step batch state, exactly like conveyors. Residual forms (a reducer over an expression of the batches, a dynamic index, ranges/wildcards) keep the shared loud rejection. See docs/design/queues.md 6, 8.
Implement the queue-conveyor coupling, flipping conveyors.md 11 from loud-rejected to supported. When a queue's outflow is a discrete conveyor's single equation-driven inflow, the two share one flow whose rate is demand- pulled by the conveyor rather than set by an equation: each DT the conveyor computes its admission budget req = min(cap_room, limit_vol) after its own Phase A, the queue supplies from its front under the batch rules, and the taken volume is admitted to the belt and written once as the shared flow rate, so the queue integrates -taken and the conveyor +taken from the same slot. Batches the conveyor cannot take wait in the queue. QueueState::take_for_conveyor implements the four one_at_a_time/batch_integrity rules (queues.md 9): one_at_a_time takes at most the front batch per DT; batch_integrity never splits a batch, taking whole batches that fully fit or nothing. ConveyorState::admission_budget mirrors phase_b's capacity/inflow-limit math exactly, excluding the queue volume it is sizing. A non-discrete conveyor with a queue directly upstream is rejected loudly (ConveyorQueueUpstreamNotDiscrete). The conveyor run_pass is split into run_phase_a plus a per-conveyor conveyor_phase_b_one, and run_coupled_passes orchestrates the interleave (all Phase A, then per coupled pair: budget, queue supply, admit, then that belt's Phase B; then uncoupled queues). The uncoupled path stays byte-identical (it runs all Phase A before any Phase B, preserving conveyor-to-conveyor chains and held exits). Coupling rides entirely inside the two plan sets, so the VM and libsimlin thread it for free. Arrayed coupling works per element (element e's queue feeds element e's belt). Overflow of the blocked volume is the next phase. See docs/design/queues.md 4.4, 9.
Serve a queue's outflows in declaration (priority) order and support the <overflow/> outflow property. After the primary outflow is served, each subsequent outflow is served against the remaining front: an ordinary secondary outflow drains the remainder (the competing-outflows case, queues.md 5.4), and an <overflow/> outflow drains only the redirectable volume -- the front material the primary could not take because it was blocked by capacity, inflow limit, or arrest (the only three redirect conditions, 4.5). The redirectable volume is desire - taken, where taken is the primary's actual take at the finite admission budget and desire is what the batch policy alone would take with unbounded room: the front batch under one_at_a_time (batches behind wait and never redirect), or the whole queue otherwise. desire is measured by a non-mutating QueueState::conveyor_desire before the take, so arrest (budget 0) redirects the whole desire and an unconstrained primary (never blocked) redirects nothing. redirectable never exceeds the post-primary front, so overflow can only drain what is present. An <overflow/> on a flow that is not a queue outflow, or on a queue's first (highest-priority) outflow, is rejected loudly (QueueOverflowNotOnQueue), per XMILE (overflow may appear only after the first outflow). Single-outflow queues are byte-identical to before. See docs/design/queues.md 4.5, 5.
…e 7) A queue, like a conveyor, is a stock with non-INTEG dynamics (a FIFO of batches whose outflow is demand-driven), so LTM's flow-to-stock link-score numerator -- which assumes plain INTEG under Euler -- makes any score touching a queue stock silently wrong. Emit one QueueLtmDegraded Warning per queue stock, mirroring ConveyorLtmDegraded: from model_all_diagnostics inside its existing ltm_enabled branch, so it is drained exactly once per model and fires once per queue even when the queue lives in a module-referenced sub-model (the cross-module double-drain #866 tracks for the model_ltm_variables warnings). It is a Warning carrying the real error code, so the model still reports simulatable.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #869 +/- ##
==========================================
+ Coverage 91.00% 91.20% +0.19%
==========================================
Files 226 230 +4
Lines 145803 152925 +7122
==========================================
+ Hits 132684 139468 +6784
- Misses 13119 13457 +338 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
ReviewFocused review across the conveyor/queue runtime + compile passes plus the integration/serde/xmile files. Three actionable findings; queue.rs, queue_compile.rs, and the vm/proto/serde integration paths look correct. [P1]
|
json::Compat mirrored only five of datamodel::Compat's ten fields, so every datamodel -> json -> datamodel round-trip (simlin-serve project hydration and the libsimlin UpsertStock patch path) silently demoted a conveyor or queue stock to a plain INTEG stock and stripped the leak/spreadflow/overflow markers from flows -- before any user edit. Add serde-optional json::Conveyor/Leakage/SpreadFlow/Queue mirrors (and the matching JsonCompat types in @simlin/engine's hand-maintained json-types.ts), plumb all five fields in both directions, and drop the ..Default::default() escape hatches from the reverse conversions so a future datamodel::Compat field is compiler-forced through json.rs. The json_proptest generators now produce the whole compat surface, so the round-trip and schema properties pin it. SpreadFlow is adjacently tagged (the only serde form that carries Dist's payload); the JSON absent -> false convention for the conveyor bools deliberately matches proto3 (the XMILE reader is where the spec's one_at_a_time=true default is resolved) and is documented at both declarations.
The ConveyorNotExpanded/QueueNotExpanded guard in compile_project_incremental correctly rejects a special stock that reaches ordinary compilation un-expanded, but only libsimlin's simlin_sim_new knew about the special build path -- so simlin-cli simulate, simlin-serve's MCP simulate tool, analyze_model, and libsimlin's is_simulatable / get_errors / apply_patch all hard-failed or mis-reported valid conveyor and queue models (apply_patch rejected and rolled back every edit to such a project). Add one engine entry point, build_sim: special-stock models route to the expansion build path (plans attached), ordinary models keep the incremental salsa path (caching and the caller's ltm_enabled flag preserved), and all six callers now use it. simlin_sim_new keeps its bespoke dispatch because it additionally threads LTM snapshots and the SimState plan cache for reset. The guard itself is unchanged and still covered by a test: it now fires only on a genuine internal bug. Returning a ready Vm (rather than hanging plans off the salsa-tracked CompiledSimulation) avoids forcing salsa::Update onto the whole plan type tree for fields the tracked query never populates.
A conveyor or queue stock inside a module-referenced (or dead) sub-model was permanently un-simulatable with the internal-sounding ConveyorNotExpanded/QueueNotExpanded invariant error: the special-stock predicates and the expanders are main-model-only (submodule conveyors are a deliberately deferred build-sequence step per the module doc), while the compile_project_incremental guard scans every synced model. Teach the guard -- the one chokepoint every compile surface funnels through, including simlin_sim_new's ordinary branch that never touches build_compiled -- to compare each offending stock's owning model against the (canonicalized) main model name: a main-model marker keeps the internal invariant codes, a non-main marker now yields the clear user-facing ConveyorInSubmodelUnsupported/QueueInSubmodelUnsupported naming the stock, its model, and the main-model-only limitation. Dead (never-instantiated) models are rejected the same way, since a special stock anywhere outside the main model can never simulate correctly as a plain INTEG.
A negative queue inflow was clamped to zero only inside the FIFO side table (QueueState::admit), while the ordinary Stocks phase integrated the raw negative rate from the untouched flow slot -- so the flat queue stock drifted negative while the batch table stayed empty, silently breaking the queues.md 4.1 invariant (sum of batches equals the stock) and 3.4's rule that a negative inflow contributes nothing. Both admit sites (the uncoupled queue pass and the coupled queue-conveyor interleave) now route through one shared admit_inflows helper that clamps each inflow slot in place to max(0, rate), sums the clamped rates, and admits that sum -- the sum-of-clamps form 4.2 step 1 specifies, which also corrects the mixed-sign multi-inflow case (each inflow is clamped independently before summing). Writing the clamped rate back into the modeler-visible flow slot follows the conveyor precedent (conveyor_phase_b_one's admitted-rate write-back) and keeps the Stocks-phase integral exactly equal to the admitted volume; a NaN inflow rate degrades to zero instead of poisoning the stock.
run_coupled_passes kept a single CoupledServe slot per conveyor, so when two queues' primary outflows both fed the same discrete belt the second coupling overwrote the first while both queues were flagged coupled: the first queue was served by neither the interleaved loop nor the uncoupled tail -- never admitted, never served, its outflow frozen at the flows-phase placeholder 0 and its stock accumulating inflow forever, silently. The spec is not silent on the shape: a discrete belt admits its equation-driven inflows in <inflow> declaration order (conveyors.md 4.3 step 4 / 11), and a coupled queue take is the same construct with the request sized from the queue's front. So support N coupled queues per conveyor: the per-conveyor slot is now a list sorted by each shared flow's position in the belt's inflow list (deterministic, compile-time order), each queue served in turn between phase A and phase B. Each successive budget charges its predecessors' takes on the capacity arm via a new prior_coupled_vol argument to coupled_admission_budget, while the inflow-limit arm is already charged through consume_inflow_budget's in_carry advance -- charged once each, never double-counted. A single coupled queue passes prior_coupled_vol = 0 and is byte-identical to before.
resolve_plans hard-coded LeakPlan.dest_conveyor to None, so run_phase_a's leak_dest_arrested vector was always all-false and the runtime's arrested-destination skip (conveyors.md 4.3 step 2: a leak whose destination conveyor is arrested is skipped this step -- rate 0, content stays) was dead code. A leak feeding an arrested belt kept flowing: the Stocks phase integrated the rate into the arrested conveyor's stock while its frozen belt admitted nothing, leaving the stock permanently above the belt's contents. Wire the leak's destination exactly the way the primary outflow's held-exit rule already is: LeakMeta records the owning downstream conveyor (same self-loop filter -- provably equivalent for a self-leak since an arrested conveyor never runs leak_step at all) and resolve_plans links element e to element e of the destination's flattened plan range. leak_step's skip semantics were already implemented and untouched; this only feeds them real data.
The slat count transit/dt had no upper bound on the production path: init_belts validated only finite-and-positive, slat_count's saturating cast turned a <len> like 1e300 into usize::MAX, and init_steady's belt allocation panicked with capacity overflow -- a process abort under libsimlin's panic=abort release builds (wasm, pysimlin, serve). A merely typo'd <len> requested terabytes instead. Enforce MAX_SLATS_PER_BELT (1,000,000; far beyond any physical transit/dt ratio) with the new ConveyorTransitTooLong error naming the belt, the computed count, and the bound. The check runs before the belt-init allocation and, for a time-varying transit, at the mid-run re-latch under exactly phase_a's latch condition (not arrested, sampling, finite) -- so after any successful latch every downstream allocation is bounded. Threading the mid-run error out of the conveyor pass is the first simulation error surfaced from inside the Euler step; the VM restores its data buffer before returning, mirroring the belt-init error path. A silent clamp was rejected: it would keep simulating a belt geometry the model does not specify. Tests exercise the gate through a tiny thread-local override (SlatBoundGuard), never a production-sized belt; the bound is documented in conveyors.md 4.1.
run_initials snapshotted initial_values before the conveyor belts and queue FIFOs were initialized and their container-access slots published, so any initials-phase reader of a hidden container stock captured its '0' placeholder: INIT(SUM(belt)) was 0 (ratios inf), and a plain stock initialized from SUM(belt) started at 0, silently. A surgical copy of the published container slots into initial_values would fix only the direct LoadInitial shape. The arrayed case routes INIT(SUM(belt[a])) through a synthesized per-element helper aux that sits in the initials runlist itself and had already copied the placeholder during the first pass -- unreachable by a slot patch. So after both publishes run_initials re-runs the initials runlist over curr, skipping exactly the container stocks (a variable whose entire CompiledInitial write-set lies in the container-slot set -- only a container stock qualifies, and re-running its '0' init would clobber the published value), then re-snapshots initial_values. The re-run is value-idempotent for every non-container initial and gated on container presence, so container-free models are byte-identical. PREVIOUS(container) needs no analog: the first step uses the fallback and later steps read the end-of-step prev snapshot, which already carries the published no-flow slot.
The ConveyorDrivenFlowRead scan iterated only model.variables, but a conveyor's parameter expressions (len, capacity, in_limit, sample, arrest, and each leak fraction) are lifted into synthesized auxes during Pass 1 and appended after the scan -- so a parameter like capacity = attriting * 20, where attriting is the belt's own leak outflow, silently evaluated from the flow's placeholder-0 slot every step (capacity 0, belt admits nothing) while the identical reference in an ordinary equation was loudly rejected. Scan the synthesized auxes too, after the driven set is complete and before the container rewrite mutates their equations. Errors name the conveyor and the offending parameter (never the internal aux name) via an origin map recorded at synthesis time. The bare-leak fraction aux derived from the leak flow's own eqn is deliberately scanned uniformly -- a self-reference still reads a placeholder slot, so rejection is correct there too. Both scans now probe driven flows in sorted order, so the flow named on a multi-reference equation is deterministic.
detect_coupling_specs coupled and discipline-guarded only a queue's FIRST outflow. A secondary outflow (ordinary or <overflow/>) whose destination is a conveyor escaped the ConveyorQueueUpstreamNotDiscrete guard entirely -- even a continuous destination raised no error -- and at runtime serve_secondary_outflows and the destination belt's phase_b independently wrote the same flow slot: depending on plan order the belt either admitted nothing while its stock integrated the served volume, or re-clamped and overwrote the slot so the queue's FIFO lost more than its stock was debited. Silent conservation breakage either way, while the module doc mislabeled it a benign documented limitation. queues.md sketches a constrained overflow-to-conveyor but does not define how a secondary's redirectable budget interleaves with a second belt's admission budget inside the combined pass, so faithful support is deferred: any queue outflow after the first whose destination is a conveyor is now a loud QueueSecondaryOutflowToConveyor error naming the queue, the outflow, and the conveyor (discrete or continuous, overflow or ordinary). The module and serve_secondary_outflows docs now describe the rejection, and the queues.md 10.7 diagnostics table gains the new row.
A conveyor's second (or later) non-leak outflow was silently left as an ordinary equation-driven outflow of the expanded INTEG stock: the Stocks phase drained the stock by its rate while the belt side table never removed the material, so the reported stock fell below the belt contents permanently -- silent conservation breakage acknowledged only by a code comment calling it a documented limitation. The slat model exits material through exactly one primary outflow (conveyors.md 3.3; Stella marks every leak explicitly), and the spec assigns no meaning to a second plain outflow, so expansion now rejects it loudly: ConveyorMultipleNonLeakOutflows names the conveyor, its primary, and every extra outflow in declaration order, with the hint to mark the flow <leak/> if leakage was intended. XMILE 4.2.1's implicit-trailing-leakage convention is deliberately not followed -- silently leaking an unmarked flow would mis-simulate a typo -- and the spec's 3.3 text now says so explicitly.
Nothing validated the combination: the XMILE reader, serde, and proto all faithfully carry a <conveyor> block and a <queue/> marker on one stock, expand_conveyors cleared only the conveyor marker, and expand_queues then re-expanded the same stock -- one stock and its shared outflow slot got BOTH a ConveyorPlan and a QueuePlan, each pass overwrote the other's driven rates every step, and with both markers cleared even the compile-path guard could not fire. Silent garbage. A stock has one type. reject_conveyor_queue_conflict now fires with the new StockBothConveyorAndQueue error at the top of the unified special-stock build path -- before either expansion, the one chokepoint both build_sim and simlin_sim_new funnel through. The scan is deliberately main-model-only: a both-marked stock in a sub-model never double-expands and already gets the clear submodel rejection. The reader/serde layers keep preserving both markers faithfully -- rejection is a compile-time concern, and dropping a marker at parse time would mask the conflict instead of surfacing it.
For a conveyor or queue model, simlin_sim_new's special-stock branch short-circuited the whole incremental-compile block -- which held the only ltm_requested latch -- so enable_ltm=true silently produced a sim with no LTM data and, worse, simlin_project_get_errors never took the LtmEnabledGuard: the ConveyorLtmDegraded/QueueLtmDegraded warnings the branch added for exactly these models were unreachable through libsimlin and pysimlin. Latch ltm_requested under the db lock in the special branch. The sim itself stays LTM-uninstrumented (get_ltm_mode remains disabled -- LTM over a non-INTEG stock is the documented degradation, not a mode to fabricate), but get_errors now transiently re-enables LTM for the diagnostic harvest and the degraded warnings surface, explaining why scores are absent. set_project_ltm_enabled is deliberately NOT called here: the special build never consults the salsa flag, the ordinary branch resets it to false at rest, and the get_errors guard keys purely on the latch -- reasoning recorded at the latch site. simlin.h regenerated (doc-only) with the enable_ltm degradation contract.
The discrete conveyor's per-time-unit in_limit budget reset compared floor(TIME) against the last unit, but the VM accumulates TIME additively (next = curr + dt), so for any dt not exactly representable in binary the clock sits a few ULPs below each integer boundary and the reset fired one DT late: the step modeling t=k.0 still saw the previous unit's exhausted budget and admitted nothing, shifting every unit's admission pulse. Derive the boundary from the ideal grid instead of the drifted clock: conveyor_time_unit recovers the exact step index k by rounding (time - start) / dt -- accumulated drift is orders of magnitude below dt/2 for any memory-bounded run -- and floors start + k*dt, formed with a single correctly-rounded multiply (10 * 0.1 is exactly 1.0 in f64, as are the 1/3 and 1/7 families). This fixes the drift at its source rather than tolerating it with an epsilon that no fixed value makes robust across run lengths, and mirrors the existing index-derived floor(i * dt) idiom in the belt's block arithmetic. run_pass/run_phase_a/run_coupled_passes thread the spec start through; the conveyor_last_unit init sites equal the helper at k = 0 by construction. Dyadic dt behavior is unchanged.
The salsa diagnostics path compiles variable fragments over the UN-expanded datamodel, so a conveyor's primary and leak outflows and a queue's outflows -- which carry no <eqn> by XMILE design, because the native pass writes their slots -- were each reported as an Error-severity empty_equation on a perfectly valid model. The phantom Errors flowed into simlin_project_get_errors, MCP read_model, and patch first_error_code, which rejected benign edits to conveyor models with allow_errors=false. lower_var_fragment now suppresses exactly the EmptyEquation code for a flow whose canonical name is an outflow of a stock carrying the conveyor or queue marker -- determined through salsa-tracked reads of the owning stock, so editing the marker away invalidates the fragment and the diagnostic returns (pinned by a test). An empty equation on any non-driven variable, and any other error on a driven flow (e.g. a malformed non-empty equation), still surface. The F2-era get_errors tolerance test tightens to zero errors for a valid conveyor model, and apply_patch now accepts benign edits to conveyor projects.
Code review — 2 findings (medium/low)[P2]
[P3] Arrayed conveyor runtime error names the bare stock, not the element —
Overall correctnessThe patch looks correct. The changes are well-tested (hand-computed oracles + conservation asserts + arrayed independence + loud-rejection coverage), the compat-based side-table design keeps the compiler/VM unaware of both constructs, and the loud-error guards are consistent with the spec. The two items above are quality-of-diagnostic gaps rather than blockers. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Implements the full XMILE conveyor and queue stock types in
simlin-engine, ending the round-trip corruption where importing a Stella model with either construct silently dropped it and re-exported a plain INTEG stock. Both are now first-class: they parse, round-trip, simulate, participate in arrays and container access, unit-check, degrade LTM loudly, and — the whole point of queues — a queue can feed a discrete conveyor with the XMILE batch-taking semantics.Two specifications drive the work and live with it:
docs/design/conveyors.md(the conveyor spec + a verified reference prototype)docs/design/queues.md(the companion queue spec authored on this branch — conveyors.md §11 referenced it before it existed)Design decisions (I owned these; flagged the ones to probe against Stella)
datamodel::Compat, alongsidenon_negative, rather than as first-classStock/Flowfields — adding them does not churn hundreds of construction sites. Proto fields append only (a DB holds serialized instances).0driven flows, and a native pass runs between the VM's Flows and Stocks phases; the stock integrates through the ordinary Stocks phase from the pass-written rates (the conservation identityΔstock = admitted − out − leak/Δqueue = inflow − outflow). The belt/batch side table tracks the same total at slat/batch granularity. This keeps the compiler and bytecode VM unaware of either construct.ConveyorNotExpanded/QueueNotExpanded); reading a driven flow in an equation (which would see the pre-pass0) is rejected (ConveyorDrivenFlowRead/QueueDrivenFlowRead); container access forms that need a per-slat/batch vector that cannot be lowered are rejected (ConveyorContainerAccessUnsupported); both are Euler-only; LTM over either degrades to aWarning(ConveyorLtmDegraded/QueueLtmDegraded) because the flow-to-stock link score assumes plain INTEG. wasmgen stays loud-unsupported until a lowering exists.conv[j]/queue[k],SUM/MEAN/SIZE/MIN/MAX/STDDEV, published by the pass into a synthesized hidden stock at step-start (a stock is read-but-not-recomputed by the Flows phase), which gives the spec's start-of-step visibility for free and sidesteps the belt/queue length being runtime-dynamic. The conveyor and queue paths share this machinery (parameterized by the source vector).desire − takenvolume a capacity/inflow-limit/arrest block rejected. The append-then-serve ordering and a few §4/§5/§6 corners are flagged in the specs as Stella-probe points to confirm against a Stella run.admission_budget(mirroring itsphase_bcapacity/inflow-limit math) sizes the demand-pull, andtake_for_conveyorimplements the fourone_at_a_time/batch_integrityrules. The conveyorrun_passwas split intorun_phase_a+conveyor_phase_b_oneso a coupled pair interleaves (belt Phase A → budget → queue supply → belt Phase B); the uncoupled path stays byte-identical (all Phase A before any Phase B, preserving conveyor→conveyor chains and held exits).How it was built
Each chunk was implemented, then adversarially reviewed by a fresh reviewer, iterated until no material findings, and committed only on a green pre-commit (Rust + WASM + TypeScript + Python). The 18 commits are individually reviewable: conveyor representation → runtime → VM integration → spread inputs → arrays → container access → §9.6/§9.8; then the queue spec → representation → runtime → VM integration → arrays+container → conveyor coupling → overflow → LTM.
Tested
Hand-computed oracles for the runtime cores (conveyor S1–S15 trajectories against the reference prototype; queue batch arithmetic and the four batch rules), end-to-end simulation with per-step conservation asserts on both stocks for the coupling, arrayed independence, container access with start-of-step visibility and out-of-range NaN, and loud-rejection coverage for every guard. New fixtures under
test/conveyors/andtest/queues/.Deferred, tracked follow-ups (not closed by this PR)
PREVIOUS(driven_flow), JSON serialization of conveyor data)model_ltm_variableswarnings (the conveyor/queue LTM warnings dodge it by emitting frommodel_all_diagnostics)wasmgen lowering of conveyors/queues (both currently loud-
Unsupported) and the diagram-editor authoring surface are separate future enhancements noted in the specs.🤖 Generated with Claude Code