Summary
Today metal-operator carries per-vendor BMC implementations, plus an out-of-cluster mock-server used by the unit tests. What it does not have is a structured way to prove that a given BMC, whether real hardware or a new mock profile, actually behaves the way the operator expects across the full surface area it relies on (power, boot order, one-time boot overrides, BIOS/BMC settings, inventory, virtual media, etc.).
This is an umbrella issue to design and build a compatibility / conformance test framework that exercises the Redfish code path against any vendor and answers two questions:
- Feature development: when a contributor adds a new capability (e.g. BIOS settings ETag drift detection, a new OEM boot-mode), how do they verify it does not regress on vendors they do not own?
- Hardware adoption: when an operator wants to bring up
metal-operator against new or planned hardware, how do they run a conformance suite that surfaces exactly which capabilities are missing or broken on their BMC, before they wire it into a production cluster?
Motivation
The problem today
- Redfish is "a standard" in name only. Every vendor deviates in
@Oem payloads, attribute-registry quirks, ETag handling, boot-source semantics, and the set of optional resources they implement. The existing vendor files exist precisely because of these deviations.
- Unit tests in
bmc/*_test.go exercise the vendor shims against bmc/mock, but the mock itself only encodes the shapes we have already encountered. It cannot tell a new adopter whether their hardware is compatible; it only tells us our code works against our own fiction.
- There is no published notion of "what metal-operator requires of a BMC". An adopter pointing the operator at unsupported hardware discovers the gap only at runtime, via failed reconciles and cryptic Redfish errors, with no structured report of which capability was missing.
- New features are merged with vendor coverage for at most the vendors the author has access to. Dell/HPE/Lenovo/Supermicro coverage drifts because no gate forces all four to be exercised for every BMC-interface method.
Why a framework, not more unit tests
The value is not "another test"; it is a shared definition of compatibility that both the CI pipeline (against mocks) and an operator on the floor (against real BMCs) can run. Conformance is a contract; the framework
is how that contract is expressed and checked.
Goals
- A declarative capability matrix: one row per BMC-interface method (power on/off/cycle, get/set boot order, one-time boot override, get/set BIOS attributes, BIOS version, BMC version, BMC settings, inventory, virtual media, …), one column per vendor/implementation, with the expected conformance level (
MUST / SHOULD / OPTIONAL).
- A runnable conformance suite that, given a BMC endpoint + credentials (or a mock profile), executes every capability and emits a structured pass/skip/fail report keyed to the matrix.
- The same suite usable two ways:
- In CI against
bmc/mock vendor profiles: gates PRs that touch bmc/.
- Out-of-cluster against a real BMC: an adoption readiness check.
- A path to record & replay: a capture of a real BMC's Redfish traffic, turned into a mock profile, so that vendor coverage can grow without every contributor owning the hardware.
- A documented "bring-your-own-vendor" story: a new OEM is supported by authoring a mock profile (+ optional
VendorFactory override) and running the suite, not by reverse-engineering the existing tests.
Non-Goals
- Replacing the existing Ginkgo unit tests in
bmc/*_test.go. The conformance suite is a layer above them: it asserts the contract, unit tests assert the implementation.
- Certifying hardware for the project. The suite reports conformance; it does not bless specific SKUs. (A separate hardware-compatibility list may follow, fed by suite output; out of scope here.)
- Network/firmware stress or performance benchmarking.
/ref https://github.com/trevex/qemu-bmc
Summary
Today
metal-operatorcarries per-vendor BMC implementations, plus an out-of-cluster mock-server used by the unit tests. What it does not have is a structured way to prove that a given BMC, whether real hardware or a new mock profile, actually behaves the way the operator expects across the full surface area it relies on (power, boot order, one-time boot overrides, BIOS/BMC settings, inventory, virtual media, etc.).This is an umbrella issue to design and build a compatibility / conformance test framework that exercises the Redfish code path against any vendor and answers two questions:
metal-operatoragainst new or planned hardware, how do they run a conformance suite that surfaces exactly which capabilities are missing or broken on their BMC, before they wire it into a production cluster?Motivation
The problem today
@Oempayloads, attribute-registry quirks, ETag handling, boot-source semantics, and the set of optional resources they implement. The existing vendor files exist precisely because of these deviations.bmc/*_test.goexercise the vendor shims againstbmc/mock, but the mock itself only encodes the shapes we have already encountered. It cannot tell a new adopter whether their hardware is compatible; it only tells us our code works against our own fiction.Why a framework, not more unit tests
The value is not "another test"; it is a shared definition of compatibility that both the CI pipeline (against mocks) and an operator on the floor (against real BMCs) can run. Conformance is a contract; the framework
is how that contract is expressed and checked.
Goals
MUST/SHOULD/OPTIONAL).bmc/mockvendor profiles: gates PRs that touchbmc/.VendorFactoryoverride) and running the suite, not by reverse-engineering the existing tests.Non-Goals
bmc/*_test.go. The conformance suite is a layer above them: it asserts the contract, unit tests assert the implementation./ref https://github.com/trevex/qemu-bmc