Check for Megaraid (and other Physical disk/raid) problems

[Tejeev Patel]

I'm copying in an Engineers comments on an internal case as he had good points.

We are currently not monitoring the error counters or logs of MegaRAID or other controllers

These issues are typically reported to us by customers noticing amber lights in their datacenter, but we'd like to become proactive at responding to these issues.

There are plugins for megaraid checks on the net that can be used. Perhaps check this as one of the better checks for megaraid.

https://exchange.nagios.org/directory/Plugins/Hardware/Storage-Systems/RAID-Controllers/check_megaraid_sas-v2/details

We should be reporting on media/other errors, plus non-optimal logical volumes.

if you're unfamiliar with megaraid commands, I suggest starting with some basics like:

megaraid -ldpdinfo -aALL
megaraid -pdinfo -PhysDrv[X:Y] -aALL
megaraid -AdpEventLog -file errors.log -aALL

You can either have an option to enable megaraid checks in the nrpe charm, or we could do auto-detects of whether we're on metal and if there's a MegaRaid/LSI raid card in lspci

I noticed that the latest version of smartmontools includes the following error when you run "smartctl /dev/sda":

Smartctl open device: /dev/sda failed: DELL or MegaRaid controller, please try adding '-d megaraid,N'

If you see this note about megaraid, we could add the checks.

We might also want to bake in some S.M.A.R.T monitoring into the charm for non-megaraid disk errors.

apparently SMART should work with the raids, but the smartctl -d <raidtype>,N format require you know which number device you're addressing, as smartctl -d <raidtype>,0 /dev/sda is the same as /dev/sde if you keep using N==0. So, iterate through smartctl -d <raidtype>,0..30 to find all disks in smartctl if you want to scrape there for the alert.

[Xav Paice (xavpaice)]

Retargeted to the hw-health charm.

[Drew Freiberger (afreiberger)]

Megaraid checks were added into the charm, however, the megaraid software is not detecting all smart errors on the disks.

See the following private pastebin to see that smartctl shows errors on megaraid,3 (fourth logical disk, fifth physical) but no errors or smart alert has been logged in megacli output.

https://pastebin.canonical.com/p/jwfcQyK7d2/

Must add smartctl -l error -d megaraid,<X> /dev/sda to the charm to check the error log.

Have to iterate over all physical disk slot numbers from megacli ldpdinfo with smartctl.

[Xav Paice (xavpaice)]

This requires smartctl to be fully integrated to the charm, and given a config option for the controller number.

Will mark as wishlist, pending smartctl addition.