Repository navigation
Feature: source dt4acc integration lattice/config from pyaml-test-lattice, pin companion deps #394
Description
Description, motivation and use case
The dt4acc integration pipeline (.github/workflows/dt4acc-integration.yml +
tests/integration/test_dt4acc_twin_smoke.py) currently vendors its own hand-maintained copy of
the FODO_1GeV_6D test lattice and its pyaml/dt4acc/catalog configuration fragments directly under
tests/integration/data/. This is the same synthetic FODO lattice that has since been extracted
into its own dedicated package, pyaml-test-lattice,
intended as the shared source of truth for pyAML integration-test/tutorial fixtures across the
ecosystem (pyaml, tango-pyaml, pyaml-cs-oa, dt4acc).
Keeping a hand-maintained duplicate here means:
- lattice/config fixes made in
pyaml-test-latticedon't propagate to this repo automatically, so
the FODO fixtures can silently drift from what companion packages test against; - the pipeline installs its control-system companions (
tango-pyaml,pyaml-cs-oa) unpinned from
gitmain, so a breaking change there (e.g. to thereadback()/get()return contract) can land
in this CI without warning — this is no longer hypothetical, see the PR Fix typing for _fill_device() methods #393 incident under
Additional context below.
Proposed solution
- Add
pyaml-test-latticeas a test/integration dependency, pinned via a git VCS URL to a
specific tag or commit (it has no PyPI release yet — only alpha tags0.1.0a1/0.1.0a2— and a
git pin is wanted regardless, so the same setup works for local dev/integration runs, not just CI). - This depends on
pyaml-test-latticefirst shipping the accelerator/catalog configuration
fragments this pipeline actually needs (lattice, dt4acc accelerator-setup,tango-pyaml
pyaml+catalog config,pyaml-cs-oapyaml+catalog config) — that authoring work, including the
choice of static vs. dynamic Tango catalog, is happening in parallel in that repo and isn't
pyaml's call to make. Track/coordinate with that work rather than migrating unilaterally. - Once available, replace
tests/integration/data/fodo_1gev_6d/*with references resolved
throughpyaml_test_lattice.lattices/pyaml_test_lattice.configurations, updating
test_dt4acc_twin_smoke.pyand the--lattice-file/--accelerator-setup-filearguments in
dt4acc-integration.ymlaccordingly. - Pin
tango-pyamlandpyaml-cs-oaindt4acc-integration.ymlto a specific tag/commit instead
of trackingmainunpinned, so a breaking companion-package change can't silently break this
pipeline; bump the pin deliberately when a companion release is meant to be picked up.
Describe alternatives you've considered
- Keep vendoring the lattice/config here and only fix the immediate pinning gap: rejected, it
doesn't address the duplication/drift withpyaml-test-latticeand companion repos that the new
shared package exists to solve. - Pin
tango-pyaml/pyaml-cs-oato amaincommit SHA updated manually on every merge: a possible
short-term stop-gap, but doesn't scale; prefer tagged releases on those repos with the pin bumped
deliberately.
Example
from pyaml_test_lattice import lattices, configurations
lattice_path = lattices["fodo_1gev_6d.json"]
accel_config_path = configurations["pyaml/tango/tango-pyaml/fodo_1gev_6d_pyaml.yaml"]used in place of the current hardcoded Path(__file__).parent / "data" / "fodo_1gev_6d" / ... in
tests/integration/test_dt4acc_twin_smoke.py, and equivalently to build the --lattice-file /
--accelerator-setup-file arguments consumed by tests/integration/start_dt4acc_twin.sh in
dt4acc-integration.yml. (Exact registry keys to use depend on what pyaml-test-lattice ends up
shipping — see point 2 above.)
Additional context
pyaml-test-latticeis still Alpha with only pre-release tags (0.1.0a1,0.1.0a2); a stable
0.1.0release + PyPI publish would be a nice follow-up but isn't a blocker since we pin via git
regardless.- The registry key path in
pyaml-test-lattice's own README example
(pyaml/tango/fodo_1gev_6d_pyaml.yaml) doesn't match the actual on-disk path
(pyaml/tango/tango-pyaml/fodo_1gev_6d_pyaml.yaml) — worth a heads-up/PR on that repo too. - Two smaller issues surfaced in
tango-pyamlwhile investigating this (not blockers here, but
worth filing separately in that repo): a dormantAttributeErrorin
TangoControlSystem.get_device_access()'s error-handling path (it callscatalog.get_name(),
which no catalog class defines), and a staleagent-notes.mdthere still describing the
pre-renameget_device()/get_devices()API. - Concrete incident confirming the need for point 4 (2026-09-10): on
PR #393 ("Fix typing for
_fill_device()methods" — apyaml-only typing change, unrelated to control-system backends),
Jean-Luc Pons reported the dt4acc smoke test failing only for thetango-pyamlparametrization:and confirmed it reproduces locally and worked with his previoustest_dt4acc_twin_accelerator_instantiates_and_reads_live_values[tango-pyaml] FAILED test_dt4acc_twin_accelerator_instantiates_and_reads_live_values[pyaml-cs-oa] PASSED ... for attribute in self._attributes: E TypeError: 'NoneType' object is not iterable .../tango/pyaml/multi_attribute.py:60: TypeErrortango-pyamlrelease. Since the
PR itself touches no control-system code, this is a breaking change that landed ontango-pyaml
mainand was picked up silently bydt4acc-integration.yml's unpinned install — exactly the
failure mode point 4 is meant to prevent. - Related constraints already noted in this repo's
agent-notes.md: dt4acc tests must run from
WSL/Linux on Windows dev machines (not Windows Conda Python), and the twin can take ~8 min to
start (CI timeout kept above 7 min).
Checklist
- I've assigned this issue to a project
- I've @-mentioned relevant people
Activity
TeresiaOlsson commentedon Sep 10, 2026
I think starting to pin versions for tests is a good way to go. Until now we could maybe manage by testing directly against main but it is starting to become increasingly difficult to manage the dependencies (and the ecosystem is still quite small so I'm guessing it will become worse as it grows...).
Testing against stable versions (even if they are just pre-releases) at least makes it possible to control when you are making a version jump so you are prepared for potential bug fixes.
gupichon commentedon Sep 10, 2026
Yes, we should rethink our branching strategy for feature development.
dev/mainbranch: Acts as our staging branch, merged intomainon a regular basis.Feature branching hierarchy:
dev/feature.dev/feature_part_1,dev/feature_part_2, etc.Integration workflow:
dev/feature.dev/featureintodev/main.dev/mainintomainto trigger the production release.TeresiaOlsson commentedon Sep 10, 2026
@gubaidulinvadim Can you perhaps add this to a future maintainer meeting?