/dev/dm-15: read failed after 0 of 4096 at 0: Input/output error

[TJ]

On 12.04.2 Precise amd64, dist-upgrade yesterday (2013-02-11) installed a new kernel (3.2.0-38.46).Whilst update-grub was running a massive number of ERRORS were reported of the form:

# update-grub
Generating grub.cfg ...
Found background image: /usr/share/images/desktop-base/desktop-grub.png
Found linux image: /boot/vmlinuz-3.6.0-030600-generic
Found initrd image: /boot/initrd.img-3.6.0-030600-generic
Found linux image: /boot/vmlinuz-3.2.0-38-generic
Found initrd image: /boot/initrd.img-3.2.0-38-generic
Found linux image: /boot/vmlinuz-3.2.0-35-generic
Found initrd image: /boot/initrd.img-3.2.0-35-generic
Found linux image: /boot/vmlinuz-3.2.0-34-generic
Found initrd image: /boot/initrd.img-3.2.0-34-generic
Found linux image: /boot/vmlinuz-3.2.0-33-generic
Found initrd image: /boot/initrd.img-3.2.0-33-generic
Found linux image: /boot/vmlinuz-3.2.0-32-generic
Found initrd image: /boot/initrd.img-3.2.0-32-generic
Found linux image: /boot/vmlinuz-3.2.0-31-generic
Found initrd image: /boot/initrd.img-3.2.0-31-generic
Found memtest86+ image: /memtest86+.bin
ERROR: asr: reading /dev/dm-26[Input/output error]
ERROR: ddf1: reading /dev/dm-26[Input/output error]
ERROR: ddf1: reading /dev/dm-26[Input/output error]
ERROR: hpt37x: reading /dev/dm-26[Input/output error]
ERROR: hpt45x: reading /dev/dm-26[Input/output error]
ERROR: isw: reading /dev/dm-26[Input/output error]
ERROR: isw: reading /dev/dm-26[Input/output error]
ERROR: isw: reading /dev/dm-26[Input/output error]
ERROR: jmicron: reading /dev/dm-26[Input/output error]
ERROR: lsi: reading /dev/dm-26[Input/output error]
ERROR: nvidia: reading /dev/dm-26[Input/output error]
ERROR: pdc: reading /dev/dm-26[Input/output error]
ERROR: pdc: reading /dev/dm-26[Input/output error]
ERROR: pdc: reading /dev/dm-26[Input/output error]
...
ERROR: pdc: reading /dev/dm-26[Input/output error]
ERROR: pdc: reading /dev/dm-26[Input/output error]
ERROR: pdc: reading /dev/dm-26[Input/output error]
ERROR: sil: reading /dev/dm-26[Input/output error]
ERROR: via: reading /dev/dm-26[Input/output error]
ERROR: asr: reading /dev/dm-25[Input/output error]
ERROR: ddf1: reading /dev/dm-25[Input/output error]
ERROR: ddf1: reading /dev/dm-25[Input/output error]
ERROR: hpt37x: reading /dev/dm-25[Input/output error]
ERROR: hpt45x: reading /dev/dm-25[Input/output error]
ERROR: isw: reading /dev/dm-25[Input/output error]
ERROR: isw: reading /dev/dm-25[Input/output error]
ERROR: isw: reading /dev/dm-25[Input/output error]
ERROR: jmicron: reading /dev/dm-25[Input/output error]
ERROR: lsi: reading /dev/dm-25[Input/output error]
ERROR: nvidia: reading /dev/dm-25[Input/output error]
ERROR: pdc: reading /dev/dm-25[Input/output error]
ERROR: pdc: reading /dev/dm-25[Input/output error]
ERROR: pdc: reading /dev/dm-25[Input/output error]
...
ERROR: pdc: reading /dev/dm-25[Input/output error]
ERROR: pdc: reading /dev/dm-25[Input/output error]
ERROR: pdc: reading /dev/dm-25[Input/output error]
ERROR: sil: reading /dev/dm-25[Input/output error]
ERROR: via: reading /dev/dm-25[Input/output error]
ERROR: asr: reading /dev/dm-24[Input/output error]
ERROR: ddf1: reading /dev/dm-24[Input/output error]
ERROR: ddf1: reading /dev/dm-24[Input/output error]
ERROR: hpt37x: reading /dev/dm-24[Input/output error]
ERROR: hpt45x: reading /dev/dm-24[Input/output error]
ERROR: isw: reading /dev/dm-24[Input/output error]
ERROR: isw: reading /dev/dm-24[Input/output error]
ERROR: isw: reading /dev/dm-24[Input/output error]
ERROR: jmicron: reading /dev/dm-24[Input/output error]
ERROR: lsi: reading /dev/dm-24[Input/output error]
ERROR: nvidia: reading /dev/dm-24[Input/output error]
ERROR: pdc: reading /dev/dm-24[Input/output error]
ERROR: pdc: reading /dev/dm-24[Input/output error]
ERROR: pdc: reading /dev/dm-24[Input/output error]
...

and this pattern repeats. The captured log-file of the update-grub run contains 2154 lines.

After some research I found a hint on the RedHat bugzilla that points to dmraid being responsible. After uninstalling the 'dmraid' (1.0.0.rc16-4.1ubuntu8) package the errors are mostly gone but some failed I/O messages remain:

# update-grub
Generating grub.cfg ...
Found background image: /usr/share/images/desktop-base/desktop-grub.png
Found linux image: /boot/vmlinuz-3.6.0-030600-generic
Found initrd image: /boot/initrd.img-3.6.0-030600-generic
Found linux image: /boot/vmlinuz-3.2.0-38-generic
Found initrd image: /boot/initrd.img-3.2.0-38-generic
Found linux image: /boot/vmlinuz-3.2.0-35-generic
Found initrd image: /boot/initrd.img-3.2.0-35-generic
Found linux image: /boot/vmlinuz-3.2.0-34-generic
Found initrd image: /boot/initrd.img-3.2.0-34-generic
Found linux image: /boot/vmlinuz-3.2.0-33-generic
Found initrd image: /boot/initrd.img-3.2.0-33-generic
Found linux image: /boot/vmlinuz-3.2.0-32-generic
Found initrd image: /boot/initrd.img-3.2.0-32-generic
Found linux image: /boot/vmlinuz-3.2.0-31-generic
Found initrd image: /boot/initrd.img-3.2.0-31-generic
Found memtest86+ image: /memtest86+.bin
  /dev/dm-15: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-16: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-17: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-18: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-19: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-20: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-21: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-22: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-23: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-24: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-25: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-26: read failed after 0 of 4096 at 0: Input/output error
done

Previously these errors have not appeared.

I tracked down the source of the errors reports to "/usr/bin/os-prober" which is called from "/etc/grub.d/30_os-prober". Specifically, the lines:

        if type lvs >/dev/null 2>&1; then
                echo "$(LVM_SUPPRESS_FD_WARNINGS=1 log_output lvs --noheadings --separator : -o vg_name,lv_name |
                        sed "s|-|--|g;s|^[[:space:]]*\(.*\):\(.*\)$|/dev/mapper/\1-\2|")"
        fi

The call to "lvs" is what writes the errors.

On investigating further I discovered that the device-mapper nodes dm-{15-26} are left-overs from an external SATA hard-drive that was hot-plugged and unplugged. It seems that, for whatever reason, the drive didn't remove cleanly and device-mapper was never told to remove the device nodes.

So the upshot is that when there are hanging device-mapper nodes on the system and os-prober runs whilst dmraid is installed all those hanging nodes will be repeatedly probed, flooding the console with error messages which look extremely serious.

It might be sensible to consider some rate-limiting of the errors, or better yet, tee-ing STDERR to a log file and referring the user to that file if it is more than 0 bytes in size once the os-probe has completed.