AUTOSAR container-PDU extraction for CAN bus logging (supersedes #912)

[kipp-ing]

AUTOSAR container-PDU extraction for CAN bus logging

Clean-room reimplementation of dynamic (and static) AUTOSAR container-PDU
extraction, superseding the approach in #912.

Why a rewrite of #912

#912's extract_pdu decoded contained-PDU signals directly from the whole
frame with no per-PDU header walk, so it only produced correct values when
a PDU sat at a fixed frame offset. Dynamic containers — a variable sequence of
contained PDUs whose byte offsets shift per frame — were mis-decoded. It also
returned a list with a non-standard entry tuple instead of the
dict[entry, dict[name, ExtractedSignal]] contract the channel-group machinery
consumes, and touched utils.py / mdf.py / .coveragerc.

What this does

• Dynamic containers: real per-frame header walk — reads Header_ID /
  Header_DLC, advances a per-frame byte offset, vectorized across frames.
  Reference algorithm is canmatrix.Frame.unpack. Header geometry is derived
  from the header signals (short 24+8, long 32+32), not hardcoded. Both header
  fields are read unsigned: canmatrix's ARXML parser marks them signed, so a
  0xFF padding byte decodes as a DLC of −1 and walks the offset backwards over
  a padded tail, rescanning the frame misaligned and inventing contained PDUs
  out of padding.
• PDU-payload-relative signals: each contained PDU's byte-aligned payload
  slice is isolated, then the existing extract_signal applies unchanged.
• Short transmissions: a sender may transmit a contained PDU shorter than
  its declared size — the header DLC is the authority. Signals reaching past the
  transmitted length are flagged through invalidation_bits rather than
  reported as measured data.
• Static (header-less) containers: decoded straight from the full frame
  (canmatrix rebases their signal start bits to frame-relative via each PDU's
  OFFSET; Frame.unpack itself refuses static containers).
• One channel group per contained PDU, via the existing untouched
  channel-group machinery — PDU identity is carried in the entry's muxer
  slot; same return contract as extract_mux.
• CAN-FD: extraction is purely CAN_DataFrame.DataBytes-width driven, so
  the FD flag has no functional effect on decoding.
• Two fixes in extract_signal (shared with the extract_mux path):
  • a signed signal of standard bit width (8/16/32/64) at a non-byte-aligned
    offset was viewed as i{std_size} on the padded width instead of being
    sign-extended from its real bit width (e.g. an 8-bit signed field at bit
    offset 1 returned 225 instead of −31);
  • a signal too wide for any integer dtype is kept as a raw byte matrix, so
    two's complement cannot be applied to it. Real containers carry opaque
    216/288/400-bit blobs declared signed, and those raised OverflowError
    on the 1 << bit_count mask.

Footprint

Only src/asammdf/blocks/bus_logging_utils.py (new extract_pdus,
_emit_pdu_signals, _contained_pdu_muxer, _signal_byte_extent) plus routing
in mdf_v4.py (is_pdu_container -> extract_pdus, else extract_mux). No
changes to utils.py, mdf.py, or .coveragerc.

Testing

Fully offline in test/test_CAN_pdu_extraction.py:

• dynamic containers vs the canmatrix.Frame.unpack oracle (BE/LE headers,
  0x00/0xFF padding, unique multi-PDU frames, bit-packed non-byte-aligned
  signed signals);
• static containers vs an equivalent flat frame carrying the same
  frame-relative signals;
• full MDF.extract_bus_logging end-to-end on a 32-byte container and on a
  genuine 64-byte CAN-FD container (EDL flag + DataLength set);
• a contained-PDU signal wider than 64 bits declared signed;
• a header DLC shorter than the declared PDU size.

The existing test/test_CAN_bus_logging.py (real OBD/J1939 data) still passes,
confirming no regression to the extract_mux path.

Validation on real measurements

Validated against two real OEM CAN-FD bus logs (5.1 M and 2.3 M CAN frames) and
three production ARXML databases covering three CAN channels — 15 real container
messages in total.

Against canmatrix.Frame.unpack as the oracle, sample by sample:
1 237 247 decoded values, 0 mismatches.

Message-based vs signal-based. The same drive was recorded twice by the
logger: once as raw CAN frames, once as signals decoded on the fly by the logger
toolchain. Cross-checking the container decode against that second recording is
an oracle that shares no code with asammdf or canmatrix. Of 303 comparable
signals, 298 agree on every one of 114 251 samples. The five that do not are
free-running sequence counters and CRCs whose sample instants differ between the
two recordings by more than the comparison window — same value range, same
cardinality.

No regression to the existing path. Running the full
MDF.extract_bus_logging on the first measurement, on master and on this
branch: all 541 non-container signals are bit-identical; on master the
container messages yield only Header_ID/Header_DLC, this branch adds 789
real contained-PDU signals in 58 additional channel groups. A further 676 008
non-container values were separately confirmed against the oracle.

Decoded values are physically plausible: HV DC-link 400 V mean / 794 V peak on an
800 V platform, inverter and coolant temperatures 25–33 °C, 14.5 V rail.

Note: reading such measurements at all requires #1307 — CAN_DataFrame.DataBytes
is a VLSD channel inside the composed CAN_DataFrame structure, and on master
its signal data cannot be located. That fix is independent of this PR and is
submitted separately.

Related issues

• Fixes CAN-FD Frame: Signals from second PDU in a specific Frame are missing #534 — signals from the second PDU in a dynamic CAN-FD frame were missing (only the PDU at offset 0 decoded); the per-frame header walk resolves this.
• Fixes CAN-FD Frame: Signals from PDUs are missing #854 — extract_bus_logging() returned no PDU signals ("PDU signal extraction is not added yet").
• Closes CAN- FD support #883 — CAN-FD PDU extraction support (supersedes the approach in CAN-FD or CAN PDU Extraction Support #912).
• Related to Signals not able to find using MDF object by loading the MF4 file #1012 — CAN-FD signals missing from groups; likely the same cause, unconfirmed without the source file.

[kipp-ing]

Update: validated against real measurements, three defects fixed

I got access to more real-world data — two OEM CAN-FD bus logs (5.1 M and 2.3 M CAN frames) with three production ARXML databases, 15 real container messages — and ran the decoder against it before asking anyone to merge this. Notably, the same drive exists both message-based (raw CAN frames) and signal-based (decoded on the fly by the logger toolchain), which gives an oracle that shares no code with asammdf or canmatrix.

That was worth doing: the in-memory tests were green, but real containers are messier than synthetic ones and turned up three defects that only real data reaches.

1. OverflowError aborted the whole extraction. Real containers carry opaque 216/288/400-bit blobs declared signed. Those are kept as a raw byte matrix, so as_non_byte_sized_signed_int computed 1 << 216 on them. Signals too wide for an integer dtype now skip two's complement, exactly like their unsigned counterparts already did.

2. Container padding was reported as measured data. A sender may transmit a contained PDU shorter than its declared size — the header DLC is the authority, not the database. Signals reaching past the transmitted length were being decoded out of the neighbouring PDU or the padding. They are now flagged through invalidation_bits. Real data hits this on 2.4 % of contained-PDU occurrences.

3. The header walk could run backwards. canmatrix's ARXML parser marks the synthetic Header_ID/Header_DLC signals signed, so a 0xFF padding byte decodes as a DLC of −1. The offset then moved backwards over a padded tail, rescanning the frame at misaligned positions and inventing contained PDUs out of padding. Both header fields are now read unsigned. This one is invisible when canmatrix.Frame.unpack is the only oracle, because it makes the same assumption.

Results after the fixes:

check	result
vs canmatrix.Frame.unpack, sample by sample	1 237 247 values, 0 mismatches
vs the signal-based recording of the same drive	298 of 303 signals agree on every one of 114 251 samples
non-container signals, master vs this branch	541 signals bit-identical, 0 changed
container signals gained	789 (master yields only Header_ID/Header_DLC)

The five signals in row 2 that do not agree everywhere are free-running sequence counters and CRCs whose sample instants differ between the two recordings by more than the comparison window — identical value range and cardinality.

Two regression tests were added; defect 3 is covered by the existing oracle test now that it asserts absolute values. Full description updated above.

One prerequisite fell out of this: reading these measurements at all needs #1307. CAN_DataFrame.DataBytes is a VLSD channel inside the composed CAN_DataFrame structure, and on master its signal data cannot be located, so extract_bus_logging raises before it ever reaches this code. That fix is independent of this PR and is submitted separately.