Skip to content

pysimlin JSON converter silently drops Compat fields (data_source today; conveyor/queue/leakage/spreadflow/overflow) + ArrayedEquation.has_except_default #882

Description

@bpowers

Problem

pysimlin hand-maintains a mirror of the Rust json::Compat schema (src/simlin-engine/src/json.rs), and it is out of sync -- both the dataclass and the cattrs structure hook silently drop compat fields on every patch/edit round-trip.

The canonical Rust Compat (src/simlin-engine/src/json.rs:78) now has 10 fields:
active_initial, non_negative, can_be_module_input, is_public, data_source, and (new on the conveyor-engine branch) conveyor, leakage, spreadflow, queue, overflow.

pysimlin declares only 4:

  • src/pysimlin/simlin/json_types.py:35 -- the Compat dataclass declares only active_initial / non_negative / can_be_module_input / is_public. It is missing data_source (a pre-existing lag) and the five new conveyor / leakage / spreadflow / queue / overflow fields.
  • src/pysimlin/simlin/json_converter.py:490 -- structure_compat hand-reads only those same four fields, so it actively drops data_source today and would drop conveyor / leakage / spreadflow / queue / overflow.

The module header at src/pysimlin/simlin/json_types.py:2-3 claims "These types match the Rust JSON types in src/simlin-engine/src/json.rs" -- but they do not.

Separately, ArrayedEquation (src/pysimlin/simlin/json_types.py:55) is missing has_except_default, which exists in the Rust struct (json.rs:212) and in the TypeScript mirror (src/engine/src/json-types.ts:145 hasExceptDefault).

Why it matters (correctness -- active data loss)

Any pysimlin edit/patch that round-trips a variable through this converter demotes it:

  • A conveyor or queue stock loses its conveyor / queue marker and is silently demoted to a plain stock.
  • A conveyor leakage / spread / queue-overflow flow loses its leakage / spreadflow / overflow marker.
  • data_source (external-data reference) is dropped today, before the conveyor work even landed.

This is silent and irreversible on a routine edit. Same bug class as PR #810 / #811 (fidelity fields dropped on edit round-trips), but for pysimlin's Compat mirror rather than core's view/variable path.

Components affected

  • src/pysimlin/simlin/json_types.py -- the Compat dataclass (line 35) and ArrayedEquation (line 55).
  • src/pysimlin/simlin/json_converter.py -- structure_compat (line 490) plus the unstructure hooks and per-variable structure paths that build Compat (there are several Compat(...) / conv.structure(..., Compat) call sites around lines 544-704).

Suggested fix

  • Extend the Compat dataclass with data_source, conveyor, leakage, spreadflow, queue, overflow.
  • Add Conveyor / Leakage / SpreadFlow / Queue dataclasses mirroring json.rs (note SpreadFlow is adjacently tagged {type, distribution} in JSON; check the exact tagging against the Rust serde attributes).
  • Update the cattrs structure/unstructure hooks (structure_compat and the Compat-omit-when-default unstructure logic near line 132) to cover the full field set.
  • Add has_except_default to ArrayedEquation.
  • Add tests (the pysimlin gate is mypy strict + ruff + pytest); a JSON round-trip test asserting a conveyor/queue stock and leak/spread/overflow flow survive a structure/unstructure cycle would pin the regression.

Context

Discovered while fixing a json::Compat serialization drop on the conveyor-engine branch. The data_source drop is pre-existing (predates the conveyor work), and the five new fields are the conveyor/queue additions; both are follow-ups, not part of the conveyor-engine branch's regression fix. Distinct from #811 (core view-element fidelity) and tech-debt item #58 (TS Dimension type).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions