It was discovered that the updated ipmi checks appear to alert on Presence sensor "Entity Absent".
When investigating this issue, I found that the presence sensor was checking on all PCI slots whether there was a card present or not.
As an example:
This line created an error on all units of a cloud that didn't have PCI slot 3 filled with a card (which is a valid config):
ubuntu@os-s003:/etc/cron.d$ sudo /usr/sbin/ipmi-sensors -r 50
ID | Name | Type | Reading | Units | Event
50 | Presence | Entity Presence | N/A | N/A | 'Entity Absent'
ubuntu@os-s003:/etc/cron.d$ sudo /usr/sbin/ipmi-sensors -r 50 --entity-sensor-names
ID | Name | Type | Reading | Units | Event
50 | Add-in Card 3 Presence | Entity Presence | N/A | N/A | 'Entity Absent'
This is a present card and is not alerting:
ubuntu@os-s003:/etc/cron.d$ sudo /usr/sbin/ipmi-sensors -r 49 --entity-sensor-names
ID | Name | Type | Reading | Units | Event
49 | Add-in Card 1 Presence | Entity Presence | N/A | N/A | 'Entity Present'
This is able to be ignored as a workaround with:
ipmi_check_options="-O --exclude-record-ids=50"
This specifically tells the ipmi-sensors command to ignore sensor number 50. This works, but maybe "Entity Absent" reports on Presence sensors should be automatically binned with "--ignore-not-available-sensors" and ignored by the ipmi scripts. This unfortunately is a catch-22 as getting notification that a fan or power sensor missing is important, but missing an Add-in Card is not necessarily critical.
The code may need to add "--noentityabsent" to avoid false positives across all environments for pci slots missing cards.