Add VLBI VDIF recording and metadata post-processing by richarms · Pull Request #843 · ska-sa/katsdpcontroller

richarms · 2026-06-19T14:44:05Z

This adds product configuration and controller support for VLBI VDIF recording and postprocessing metadata generation from a gpucbf V-engine.

This branch introduces the sdp.vdif output stream type in product-config schema, wires it through product configuration, and generates the corresponding capture-time katsdpvlbi recorder task. The post-processing task is scheduled with the capture block, receives telstate context, can run in antab, pass_through, or disabled mode, and is colocated with the corresponding VLBI recorder. Implements version 4.8 of the product-configure schema. Tested end-to-end in the lab.

Introduce the sdp.vdif output type to the product configuration model and schema so a VLBI recorder path can be expressed in product configs. This commit also adds focused product-config tests for the new stream type.

Teach the logical graph builder to create a dedicated vlbi.<stream> task for sdp.vdif outputs backed by gpucbf tied-array resampled voltage. The recorder task uses the katsdpvlbi image, computes the jive5ab mode from the upstream stream geometry, and extends the shared VLBI test config so the path is covered by focused graph tests.

Expose vlbimeta.mode through the product configuration model and schema so controllers can choose between disabled, pass-through, and antab post-processing. Keep the coverage narrow by extending only the product-config tests that validate option parsing.

Create a vlbimeta.<stream> post-process task for sdp.vdif outputs and pass the configured mode through to the entrypoint. This also reuses the already-resolved telstate endpoint for batch graphs so vlbimeta can consume capture-block telstate during post-processing.

Add a configurable VLBI recorder protocol to develop options, and use it when generating sdp.vdif recorder tasks. The recorder now requests a CBF interface with multicast and ibverbs metadata so scheduling matches the VLBI data path, and its command exports the assigned interface name for the recorder-side route guard. Focused tests cover option parsing and the generated logical task.

Give each vlbimeta task the vis stream it should read metadata from by selecting the calibrated vis stream when one sdp.cal output is present, and otherwise falling back to a single vis stream. When post-processing is instantiated, pin vlbimeta to the same host that ran the matching VLBI recorder so it can read the capture output locally. Focused tests cover the generated vlbimeta command and dataset-stream selection.

Use the SubarrayProduct node map when pinning vlbimeta to the VLBI recorder host during postprocessing. The previous code mistakenly looked for recorder nodes on CaptureBlock, which caused postprocessing to crash before vlbimeta could launch. Add a focused regression test that enables vlbimeta for a VDIF stream and checks that postprocessing completes and accounts for the extra batch task.

Replace hardcoded batch task counts in the controller postprocess tests with counts derived from the capture block's generated postprocess graph. This keeps the assertions aligned with the configured postprocess tasks when vlbimeta or future batch tasks (thinking ahead to `katsdprfi` here) are added.

bmerry · 2026-06-22T07:31:02Z

              less_resources:
                type: boolean
                default: false
+              vlbi_recorder_protocol:


I don't see this in the schema document. And once you've added it, it needs to be gated on if version >= "4.8".

It's still not in the schema document, nor in the signed M1000-0001-009 v4. So you'll probably need an ECP for this.

Still not addressed as far as I'm aware.

This should NOT be merged as is, as it doesn't implement version 4.8 and does not support the 4.8 features in 4.9 documents. That will come from #843. See NGC-1992.

richarms · 2026-07-09T07:34:12Z

I've pushed follow-up changes to adress review comments:

Added 4.8 gating for the new VLBI config options in product config schema.
Removed duplicate Python-side enum validation for vlbi_recorder_protocol and vlbimeta.mode; these are now left to jsonschema validation.
Added cbf.tied_array_resampled_voltage as the CBF-facing stream type for vlbi. Changed sdp.vdif to require cbf.tied_array_resampled_voltage as its source.
Removed the SDP stream from CONFIG_CBF_ONLY.
Moved the generator-specific VLBI tests into test_generator.py.

bmerry · 2026-07-09T09:57:04Z

              less_resources:
                type: boolean
                default: false
+              vlbi_recorder_protocol:


It's still not in the schema document, nor in the signed M1000-0001-009 v4. So you'll probably need an ECP for this.

bmerry · 2026-07-09T10:08:11Z

+    """VDIF-ready tied-array-resampled-voltage stream from CBF."""
+
+    stream_type: ClassVar[str] = "cbf.tied_array_resampled_voltage"
+    _valid_src_types: ClassVar[_ValidTypes] = {"gpucbf.tied_array_resampled_voltage"}


Suggested change

_valid_src_types: ClassVar[_ValidTypes] = {"gpucbf.tied_array_resampled_voltage"}

_valid_src_types: ClassVar[_ValidTypes] = {"cbf.tied_array_channelised_voltage"}

I see M1000-0000-009 incorrectly lists it at cbf.tied_array_resampled_voltage. That'll need fixing.

Addresses the core review comments: cbf.tied_array_resampled_voltage is now an input type, it follows the other CBF input schema pattern, it requires a pair of cbf.tied_array_channelised_voltage sources, and the stream now gets bandwidth/channels/bits/pols from CBF sensors instead of the gpucbf stream model. The redundant VdifStream.__init__ validation was also removed.

richarms · 2026-08-04T12:10:01Z

Thanks for the detailed reviews. I have reworked the branch to implement your stream-model and schema suggestions.

The VLBI additions are gated on product-config schema 4.8.
cbf.tied_array_resampled_voltage is now an external input type following the cbf.* pattern.
It requires exactly two cbf.tied_array_channelised_voltage source streams.
Its bandwidth, channel count, sample depth and polarisation ordering come from CBF sensors obtained through CAM.
sdp.vdif accepts only this external CBF stream as its source; the direct gpucbf.* paths have been removed.
Redundant schema validation was removed.
The shared product-config fixture was updated to 4.8, and generator-specific tests were moved to test_generator.py.
Mean-power history is now handled by katsdpcam2telstate, rather than adding things to controller.

I also (successfully) ran the branch through in the lab.

bmerry

Looking much better architecturally.

bmerry · 2026-08-05T07:46:49Z

              less_resources:
                type: boolean
                default: false
+              vlbi_recorder_protocol:


Still not addressed as far as I'm aware.

Move the VLBI task alongside its friends, clarify the CBF VDIF stream docstring, and correct the new generator test copyright.

`vgpu` labels tied-array resampled-voltage multicast outputs with the vdif port name, while its channelised inputs use spead. The plumbing of `katsdpvlbi` wasn't following this convention, using spead as the stream name. Apply the correct(/vgpu) convention to external cbf.tied_array_resampled_voltage inputs: accept a vdif URL, construct the multicast node with a vdif port, and connect the recorder to the corresponding _vdif endpoint. Also update the fixtures and generator assertions.

bmerry

Code changes look okay, but you've still got this vlbi_recorder_protocol in the implementation that's not defined in the ICD.

richarms · 2026-08-06T11:45:57Z

Great thanks. I suggest that we not make that block this PR, and I'll follow this up as an ICD amendment

richarms added 13 commits April 10, 2026 09:15

Add sdp.vdif product configuration support

9f2a38e

Introduce the sdp.vdif output type to the product configuration model and schema so a VLBI recorder path can be expressed in product configs. This commit also adds focused product-config tests for the new stream type.

Add vlbimeta mode configuration

0b30dac

Expose vlbimeta.mode through the product configuration model and schema so controllers can choose between disabled, pass-through, and antab post-processing. Keep the coverage narrow by extending only the product-config tests that validate option parsing.

flake8

51a66ac

Persist VLBI mean-power sensor history to telstate

90ff193

Merge remote-tracking branch 'origin/master' into vlbi-incorporation

05f3303

Merge remote-tracking branch 'origin/master' into vlbi-incorporation

c9bc54d

Add product config schema 4.8 for VLBI

4ac3b0b

richarms requested review from bmerry and ludwigschwardt June 19, 2026 14:44

bmerry requested changes Jun 22, 2026

View reviewed changes

bmerry added a commit that referenced this pull request Jun 23, 2026

Implement product-configure schema version 4.9

ca8d519

This should NOT be merged as is, as it doesn't implement version 4.8 and does not support the 4.8 features in 4.9 documents. That will come from #843. See NGC-1992.

bmerry added a commit that referenced this pull request Jun 23, 2026

Implement product-configure schema version 4.9

6d2485b

This should NOT be merged as is, as it doesn't implement version 4.8 and does not support the 4.8 features in 4.9 documents. That will come from #843. See NGC-1992.

bmerry added a commit that referenced this pull request Jun 24, 2026

Implement product-configure schema version 4.9

7b70d56

This should NOT be merged as is, as it doesn't implement version 4.8 and does not support the 4.8 features in 4.9 documents. That will come from #843. See NGC-1992.

bmerry added a commit that referenced this pull request Jun 29, 2026

Implement product-configure schema version 4.9

5f5bd5c

This should NOT be merged as is, as it doesn't implement version 4.8 and does not support the 4.8 features in 4.9 documents. That will come from #843. See NGC-1992.

bmerry mentioned this pull request Jun 29, 2026

Implement product-configure schema version 4.9 #847

Draft

richarms added 4 commits July 8, 2026 11:59

Gate VLBI config options on schema 4.8

f858604

Rely on schema for VLBI option validation

3aaae89

Model VDIF input as a CBF resampled stream

6522f0b

Move VLBI generator tests to generator test module

dbd224d

richarms force-pushed the vlbi-incorporation branch from dc1b393 to dbd224d Compare July 8, 2026 13:04

richarms added 2 commits July 9, 2026 07:41

Update product config test fixture to schema 4.8

4c1ad53

Remove redundant VLBI protocol schema type

0293628

richarms requested a review from bmerry July 9, 2026 07:46

bmerry requested changes Jul 9, 2026

View reviewed changes

richarms force-pushed the vlbi-incorporation branch from 1430103 to 7d9d035 Compare July 22, 2026 11:32

Remove duplicate telstate sensor observer

912b51d

richarms requested a review from bmerry August 4, 2026 12:10

bmerry requested changes Aug 5, 2026

View reviewed changes

richarms added 3 commits August 6, 2026 09:26

Address review comments

7a94b65

Move the VLBI task alongside its friends, clarify the CBF VDIF stream docstring, and correct the new generator test copyright.

Fix sensor override order

6491ca2

bmerry reviewed Aug 6, 2026

View reviewed changes

	_valid_src_types: ClassVar[_ValidTypes] = {"gpucbf.tied_array_resampled_voltage"}
	_valid_src_types: ClassVar[_ValidTypes] = {"cbf.tied_array_channelised_voltage"}

Uh oh!

Conversation

richarms commented Jun 19, 2026

Uh oh!

bmerry Jun 22, 2026

Choose a reason for hiding this comment

Uh oh!

bmerry Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

bmerry Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

richarms commented Jul 9, 2026

Uh oh!

Uh oh!

Uh oh!

bmerry Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bmerry Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

richarms commented Aug 4, 2026

Uh oh!

bmerry left a comment

Choose a reason for hiding this comment

Uh oh!

bmerry Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bmerry left a comment

Choose a reason for hiding this comment

Uh oh!

richarms commented Aug 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants