One row per controller family, firmware revision and call. Record what the device did, not what the specification says it should do. The note field is the most valuable column here and the one most often left empty: it becomes the observed-behaviour record the next engineer needs.
| Vendor | Controller | Firmware | Units | Call | Behaviour | What it does | Remove |
|---|---|---|---|---|---|---|---|
| Variant | Units | Share of fleet | Calls tested | Coverage | Findings |
|---|---|---|---|---|---|
| Dell iDRAC 9 6.10.30 | 9,200 | 38.3% | 2 of 8 | 25% | clean so far |
| HPE iLO 5 2.78 | 7,400 | 30.8% | 2 of 8 | 25% | 1 diverging |
| Dell iDRAC 8 2.83 | 3,100 | 12.9% | 1 of 8 | 13% | 1 diverging |
| Supermicro BMC 1.73 | 2,400 | 10.0% | 1 of 8 | 13% | breaks on load |
| HPE iLO 4 2.82 | 1,900 | 7.9% | 1 of 8 | 13% | 1 diverging |
Divergences that recur across estates
Twelve behaviours that show up repeatedly on real hardware, with what each one does to code written from the document. The pattern worth noticing is in the right-hand column: most of these produce a plausible value or a misread status rather than an error, which is why they survive testing and appear in production.
| Layer | What the device does | What it does to code written from the document |
|---|---|---|
| Redfish | Optional schema fields absent rather than null | Parsers that check for null get an undefined key |
| Redfish | Numeric readings returned as strings | Silent type coercion, or a comparison that never fires |
| Redfish | Thermal endpoint missing entirely on older controllers | 404 read as node-down rather than as unsupported |
| Redfish | Session limits between four and eight, undocumented | Fan-out polling degrades the controllers it monitors |
| Redfish | Error bodies differ in shape between vendors | Error handling written for one vendor misreads another |
| IPMI | Sensor numbering not stable across firmware revisions | A sensor id read after an update refers to something else |
| Modbus | Register addressing off by one against the document | Every value shifted by one register, all plausible |
| Modbus | Word order swapped for 32-bit values | A plausible number rather than an error, which is worse |
| Modbus | Coils read back a state different from the write | Confirmation logic passes on a write that did not take |
| Modbus | Malformed request stops the device until power cycle | A scan takes a device out of service |
| Modbus | Documented function codes returning an exception | Feature assumed available at design time is not |
| Any | Behaviour differs between two units of one part number | Reproduction fails on the unit in front of you |
The two worst entries are the word-order swap and the off-by-one addressing, because both return a well-formed number that is simply wrong. An exception is a good outcome by comparison: it fails loudly and somebody investigates.
A worked example
A 24,000 unit estate across five controller variants, with two Redfish calls recorded against each. The lab has one of every controller and the team reports the integration as tested against the full estate.
Against the eight calls the platform actually makes, 18 percent of the matrix is recorded. More usefully: 18,925 units, 79 percent of the fleet, run a combination nobody has exercised. That figure is the one that reframes the conversation, because the lab genuinely does have one of everything, and one of everything is not coverage when coverage is measured in units.
Four of the five variants already show a divergence in the two calls that were tested, covering 14,800 units. Extrapolating that rate across the six untested calls is not a rigorous forecast, and it does not need to be: the direction is enough to justify a week of bench time.
The entry that changes the architecture is the Supermicro row. 2,400 units stop responding after five rapid reads, which is not a divergence to handle in the abstraction layer. It is a per-target scheduling constraint on the whole platform, and adding workers makes it arrive sooner. That belongs in the polling schedule before it belongs in a driver.
The arithmetic, so you can check it
A variant is a controller family on a firmware revision, because that is the unit that behaves consistently: vendor alone is too coarse and an individual unit is too fine. Coverage per variant is calls tested / calls planned, capped at one, and fleet coverage weights that by units, so untested units = total - sum(units x coverage).
The ordering is by units x (1 - coverage), which is where the next intermittent failure comes from. It deliberately ranks a large partially-covered variant above a small uncovered one, because that is where the population is.
This records what you observed, so it inherits your sampling. Behaviour can differ between two units of the same part number and the same firmware, which means a single tested unit is evidence about that unit before it is evidence about the variant. It also cannot tell you which untested call will diverge, only how much of the fleet is exposed to that question. Treat a full matrix as the absence of known divergence rather than as proof of conformance.
The reasoning is in PLC to model and fleet scale changes the failure mode. For the scheduling constraint a hostile controller imposes, use the fleet poll scheduler.
Questions
Because a standard is a starting point rather than a guarantee, and what is on your estate is a vendor's implementation of it. iDRAC 8 and 9 differ, iLO 4 and 5 differ, optional schema fields drop inconsistently, and error behaviour is not uniform even across firmware revisions of one vendor. Code that trusts the document works in the lab and fails intermittently against the real estate.
A table of what each device family and firmware revision actually did for each call you make, rather than what its documentation claims. It is the highest-value artifact in a protocol integration and it almost never exists, so every new engineer rediscovers the same divergences by hitting them in production.
Far more than teams expect, because coverage is usually measured in devices tested rather than in units carried by the variants tested. A lab with one of each controller looks like full coverage and can leave most of the fleet on a firmware revision nobody has exercised. That population is where intermittent failures come from.
Because they need completely different responses. A divergence is a parsing problem you handle in an abstraction layer. A controller that stops responding under rapid reads is a per-target scheduling constraint on the whole platform, it cannot be handled in a driver, and adding workers makes it arrive sooner rather than later.
On the variant carrying the most units at the lowest coverage, which is rarely the one that is easiest to reach. Coverage on a variant with two hundred units is worth a fraction of coverage on one with nine thousand, and the tool orders by exactly that so the decision does not depend on which device happens to be on the desk.