"local disk flush failed with status -5" with drbd8 over LVM

[Paul Leonard]

This is a known issue with versions of drbd 8.0.x prior to 8.0.12 (or 8.2.6, in the unstable branch). It's related to some kernel I/O modules in pre-2.6.25 kernels failing to return correct error codes from flushes (LVM being the one that I'm using, although some buggy SCSI controllers could also experience this same problem). Example discussion on the matter in DRBD-user mailing list: http://archives.free.net.ph/thread/20080515.095952.81067f01.en.html
Releasing either drbd 8.0.12 or backporting the appropriate changes to 8.0.11 (8.0.11-0ubuntu4 maybe?) is probably the easiest fix for the problem.

Release of Ubuntu we're using: 8.04
Current version of drbdr8-utils we're using: 8.0.11-0ubuntu3
What I expected to happen: no error messages
What happened instead:
May 30 13:49:20 tron kernel: [162444.168016] drbd0: local disk flush failed with status -5
May 30 13:49:22 tron kernel: [162446.169329] drbd0: local disk flush failed with status -5
May 30 13:49:24 tron kernel: [162448.169875] drbd0: local disk flush failed with status -5
etc., every 2-5 seconds.

[Andreas Påhlsson (andreas-pahlsson-xcerion)]

I have this error once every millisecond in my logs during IO testing.

This patch is really important for Hardy in HA configurations using DRBD. Please backport or upgrade packages.

[derrick daugherty (ubuntu-launchpad-com)]

I would very much like drbd 8.0.12 as well. i was hoping to keep this new cluster as cookie-cutter as possible.

i'm drbd on top of lvm, the logs go nuts during any filesystem io. drbd syncs are error free but ocfs2 on top that brought this to my attention.

[Chuck Short (zulcss)]

Which kernel version are you using?

Thanks
chuck

[Paul Leonard (pleonard)]

We're using the Ubuntu stock kernel from the Main repository, current as of the date that I filed the bug:
pleonard@tron:~$ uname -a
Linux tron 2.6.24-17-generic #1 SMP Thu May 1 13:57:17 UTC 2008 x86_64 GNU/Linux
pleonard@tron:~$ dpkg -l | grep linux-image
ii linux-image-2.6.24-16-generic 2.6.24-16.30 Linux kernel image for version 2.6.24 on x86
ii linux-image-2.6.24-17-generic 2.6.24-17.31 Linux kernel image for version 2.6.24 on x86
ii linux-image-generic 2.6.24.17.19 Generic Linux kernel image

The changelogs for the 2.6.24-series kernels in Main and Updates pools (http://changelogs.ubuntu.com/changelogs/pool/main/l/linux/linux_2.6.24-19.34/changelog) don't seem to indicate an appropriate backport that would fix this problem, but for the sake of completeness I'll upgrade to the newest available Updates-pool rev (2.6.24.19.21) and update this ticket.

[Paul Leonard (pleonard)]

Looks like whatever patches were backported into the Update-pool kernel have fixed this problem, in lieu of the 8.0.12 release of drbd8. As of linux-image-2.6.24-19-generic=2.6.24-19.33, I'm not seeing those error messages in /var/log/syslog any more.

Please hold this bug for 24 hours, just in case, and then close it with my thanks.

On a related note, will the 8.0.12-1ubuntu1 build of drbd8 for Intrepid be backported to Hardy?

[Chuck Short (zulcss)]

Thanks for testing, with regards to 8.0.12 please ask for a backport.

https://help.ubuntu.com/community/UbuntuBackports

Regards
chuck

[derrick daugherty (ubuntu-launchpad-com)]

2.6.24-19.33 seems to have fixed it for me as well, i was on -18. i created an 8.0.12 drbd package, should i just create a drbd backport bug with the patch?

for the archives, i received a new message on mount, but instead of repeating for every io it is only at mount time:

Jun 19 11:24:49 bwvm01 kernel: [54082.234571] drbd1: local disk flush failed with status -95

[Chuck Short (zulcss)]

Please see my previous comment.

Thanks
chuck

[Andreas Påhlsson (andreas-pahlsson-xcerion)]

I'm seeing the same "local disk flush failed with status -95" as Derrick once in my logs. The original logging problem is gone as far as I can see. Nice. :)