Skip to content

add dynamic AUTOSAR PDU-container extraction to CAN bus logging - #1

Draft
kipp-ing wants to merge 90 commits into
masterfrom
dynamic-pdu-container-extraction
Draft

kipp-ing wants to merge 90 commits into
masterfrom
dynamic-pdu-container-extraction

Conversation

@kipp-ing

@kipp-ing kipp-ing commented Jul 8, 2026

Copy link
Copy Markdown
Owner

What changed

extract_pdus() added to src/asammdf/blocks/bus_logging_utils.py. It decodes
AUTOSAR dynamic PDU-container CAN frames. A container frame carries a variable
sequence of contained PDUs, each prefixed by a Header_ID and Header_DLC
field. Each contained PDU's byte offset depends on the lengths of the PDUs
before it, so the payload is walked header by header. The walk keeps a per-frame
byte offset and advances it across all frames at once (one pass per header
slot), so frames with different PDU counts, orders, and lengths are handled. For
each contained PDU id the matching payload slices are gathered into one array
and its signals are extracted with the existing extract_signal. A contained
PDU's signal start bits are relative to the PDU payload, so no offset rebasing is
needed.

_extract_can_logging() in src/asammdf/blocks/mdf_v4.py routes frames where
message.is_pdu_container is true through extract_pdus(). All other frames
keep using extract_mux().

Output: each contained PDU becomes its own channel group, created by the
existing channel-group code. extract_pdus() returns the same structure as
extract_mux() with the PDU identity in the entry key, so the caller is
unchanged apart from the routing branch.

Decoding matches canmatrix.Frame.unpack() for container frames.

Signedness fix in extract_signal

extract_signal() decoded a signed signal of standard bit width (8, 16, 32, 64)
at a non-byte-aligned offset by viewing the padded byte block as i{std_size},
which does not sign-extend from the field width. Example: an 8-bit signed field
at bit offset 1 returned 225 instead of -31. The condition now also uses
as_non_byte_sized_signed_int when the field is not byte aligned. This path is
shared with extract_mux, so the change also affects normal (non-container)
extraction of such signals.

Tests

test/test_CAN_pdu_extraction.py (offline, uses canmatrix.Frame.unpack() as
the reference):

  • extract_pdus() output compared to the reference for big-endian and
    little-endian headers and for 0x00 and 0xFF padding, over frames with random
    PDU count and order.
  • Full extract_bus_logging() run on a synthesized CAN-logging MF4, checking one
    channel group per contained PDU and matching sample values.
  • Signed, non-byte-aligned signals (big and little endian) inside one PDU.
  • A container without Header_ID/Header_DLC returns no signals.

Existing test/test_CAN_bus_logging.py (OBD2 and J1939) still passes.

Not covered

  • Static PDU containers (no per-PDU header) are skipped and produce no signals.
  • PDU-internal multiplexing.
  • LIN container frames.
  • The CAN-FD frame flag path is not exercised in tests; container frames were
    built at 64-byte width without the FD flag set.

dnbmch-kb and others added 12 commits August 2, 2026 13:38
- ListData.__init__ assigned self.self.data_block_nr in both the mapped
  and the stream branch, so reading any ##LD block raised
  AttributeError: 'ListData' object has no attribute 'self'
- _append_column_oriented read signal.axes / signal.conversions /
  signal.units, none of which exist on Signal; the three locals were
  never used, so the lines are removed
- add a 4.20 column-storage round-trip regression test covering both
  paths
…self-typo

Fix AttributeError reading LDBLOCK (self.self typo) and column-oriented append crash
Container (multiplexed PDU) frames carry a variable sequence of contained
PDUs, each prefixed by a Header_ID/Header_DLC header, so a PDU's byte offset
depends on the lengths of the PDUs before it. extract_mux() only handled
is_multiplexed frames, so container frames were mis-decoded.

- extract_pdus() in bus_logging_utils.py walks the container headers per
  frame (vectorized across frames, one pass per PDU slot), gathers each
  contained PDU payload and extracts its signals. Each contained PDU becomes
  its own channel group, reusing the existing channel-group machinery; a
  PDU's signal start bits are PDU-relative so extract_signal applies
  unchanged. canmatrix.Frame.unpack is the reference algorithm.
- route is_pdu_container frames through extract_pdus in _extract_can_logging;
  all other frames keep using extract_mux.
- fix extract_signal signedness: a signed signal with a standard bit width
  (8/16/32/64) at a non-byte-aligned offset was viewed as i{std_size} on the
  padded width instead of being sign-extended from its real bit width (e.g. an
  8-bit signed field at bit offset 1 returned 225 instead of -31). This
  affected the normal extract_mux path too.
- add offline test/test_CAN_pdu_extraction.py validating extract_pdus against
  canmatrix.unpack (big/little-endian headers, 0x00/0xFF padding, unique
  multi-PDU frames, signed bit-packed signals) plus a full extract_bus_logging
  end-to-end case and a static-container skip.
Static (header-less) AUTOSAR containers have a fixed layout: canmatrix
rebases each contained PDU's signal start bits to be frame-relative and
Frame.unpack itself refuses them, so they were previously skipped. Handle
them by decoding every contained PDU straight from the full frame payload,
one channel group each (pdu.id is None -> identity keyed on the PDU name).

- factor the per-PDU signal emission out of extract_pdus into
  _emit_pdu_signals, shared by the dynamic (PDU-relative payload) and static
  (full-frame payload) paths; _contained_pdu_muxer builds the channel-group
  identity and tolerates a None header id.
- test_extract_pdus_static_container validates static extraction against a
  flat frame carrying the same frame-relative signals (byte-aligned LE/BE
  plus a non-byte-aligned signed field).
- test_extract_bus_logging_canfd_container_e2e runs the full pipeline on
  genuine CAN-FD container frames (64-byte payload, EDL flag + DataLength
  members set) to exercise the real-world container transport.
Validated `extract_pdus` against two real OEM CAN-FD bus logs (5.1 M and 2.3 M
CAN frames) and three production ARXML databases, using `canmatrix.Frame.unpack`
as the oracle. That uncovered three defects that the in-memory tests could not
reach:

1. `OverflowError` aborted the whole extraction. Real containers carry opaque
   216/288/400-bit blobs declared *signed*; those are kept as a byte matrix, so
   `as_non_byte_sized_signed_int` computed `1 << 216` on them. Signals wider
   than an integer dtype now skip two's complement, like unsigned ones already
   did.

2. Container padding was reported as measured data. A sender may transmit a
   contained PDU shorter than its declared size — the header DLC is the
   authority. Signals reaching past the transmitted length are now flagged
   through `invalidation_bits` instead of surfacing padding as a value. Real
   data hits this on 2.4 % of contained-PDU occurrences.

3. The header walk could run backwards. canmatrix's ARXML parser marks the
   synthetic `Header_ID`/`Header_DLC` signals *signed*, so a 0xFF padding byte
   decodes as a DLC of -1; the offset then moved backwards over a padded tail,
   rescanning the frame misaligned and inventing contained PDUs out of padding.
   Both header fields are now read unsigned.

Two regression tests added for 1. and 2.; 3. is covered by the existing oracle
test now that absolute values are asserted.
`doc/buslogging.rst` is the user-facing bus logging documentation and said
nothing about container I-PDUs. Add a section covering what gets decoded (one
channel group per contained PDU, addressed like any other bus logging signal),
that both the dynamic and static layouts are handled, that samples whose bytes
were not transmitted are marked through `invalidation_bits`, and the two known
limitations (multiplexed contained PDUs, LIN).

Also point the developer note at it, and note there that it is deliberately not
part of the built docs — sphinx is configured for `.rst` only.
@kipp-ing
kipp-ing force-pushed the dynamic-pdu-container-extraction branch from fa2e1be to 2190907 Compare August 15, 2026 15:18
kipp-ing and others added 17 commits August 15, 2026 17:58
test_Tabular_BaseWidget_Shortcuts.py:50 called @skipIf(sys.platform ==
"linux") without the required reason argument, raising a TypeError at
import time that aborted collection for the entire test suite on every
platform. Add a reason string.

The pyinstaller_build workflow hits the same NMake/CMAKE_C_COMPILER
failure on windows-latest that wheels.yml was fixed for: force the
Ninja generator (and install Ninja + VS Build Tools) the same way.

Verified locally:
- full-suite collection (228 items, 0 errors) after the skipIf fix,
  vs. 1 collection error aborting everything before it
- rebuilt the extension with CMAKE_GENERATOR=Ninja forced (mirrors
  what build.yml now sets); scikit-build-core confirms "Building
  project with Ninja...", extension imports fine, full non-GUI suite
  (56 passed, 2 skipped) still green against that rebuild
Use the CMake package and target names the system lz4/zstd export
…and-pyinstaller-windows

fix: unblock test collection and Windows pyinstaller build
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants