2.6.32-47 kernel update on 10.04 breaks software RAID (+ LVM)
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
linux (Ubuntu) |
Won't Fix
|
Medium
|
Steve Conklin |
Bug Description
Been running 10.04 LTS on 8 similar AMD Opteron x86_64 servers for several years. The servers have been kept up-to-date with patches as they come out. These servers have been running 2.6.x kernels. Each server has some form of Linux software RAID running on it as well as 3Ware hardware RAID card using SATA disks. Software RAID is configured as RAID1 for all but one server running software RAID10. All servers had software raid configured to use single partitions on each disk of types of 0xFD (Linux Software Raid Autodetect). All servers were configured with LVM over the top of /dev/md0.
In past year, mysterious problems have been happening with software RAID after applying system patches. Upon reboot, server is unable to mount LVM partitions on Linux software RAID and boot is interrupted with "Continue to wait; or Press S to skip mounting or M for manual recovery" requiring intervention from an operator.
Upon pressing 'M' and logging in as root, the LVM slices on the software RAID partition are not mounted and sometimes appear to be missing from LVM. Oftentimes pvs, vgs and lvs will complain about "leaking memory". Germane to the issue, LVM will sometimes show the problem partitions as "Active" while other times during the login, they will simply be gone. With LVM and /dev/md0 unstable, there is no way to discern the true state of the partitons in question. Starting the system from alternate boot media such as CDROM or USB drive, sometimes shows the software RAID and LVM in proper state which leads to suspicion of a kernel update on the afflicted system. Historically and subjectively, best practice in this instance seems to be booting from live media and starting the array degraded mode, and backing up the array.
Changed in linux (Ubuntu): | |
importance: | Undecided → Medium |
Changed in linux (Ubuntu): | |
assignee: | nobody → Steve Conklin (sconklin) |
Prior to rebooting the system, these were the contents of /var/cache/ apt/archives showing which patches had just been applied. This system had been running and rebooting for months prior to applying these patches.
[11:42 06/12/13] apt/archives/ 0.8.2-3_ amd64.deb 1.1.4-2_ amd64.deb ng_3.1. 2-1~lucid1_ amd64.deb selector- common_ 0.5.8+langfixes ~lucid2_ all.deb 2.8.5-2~ build0. 10.04_amd64. deb server_ 2.6.32. 47.54_amd64. deb 2.6.32. 47.54_amd64. deb server_ 2.6.32. 47.54_amd64. deb 2.6.32- 47_2.6. 32-47.109_ all.deb 2.6.32- 47-server_ 2.6.32- 47.109_ amd64.deb 2.6.32- 47-server_ 2.6.32- 47.109_ amd64.deb 3.1.3-2ubuntu3. 5_amd64. deb 3.1.3-2ubuntu3. 5_amd64. deb 2.8.5-2ubuntu0. 4_amd64. deb 2%3a1.1. 1-2ubuntu0. 1_amd64. deb 1.5-2ubuntu0. 1_amd64. deb data_2% 3a1.3.2- 1ubuntu3. 1_all.deb 6_2%3a1. 3.2-1ubuntu3. 1_amd64. deb
[root@usb-live /mnt/slash]
# dir var/cache/
total 46260
-rw-r--r-- 1 root root 51876 Dec 4 2007 libdbi0_
-rw-r--r-- 1 root root 60626 Nov 6 2009 libnet1_
-rw-r--r-- 1 root root 342662 Sep 6 2010 syslog-
-rw-r--r-- 1 root root 276106 May 25 2011 language-
-rw-r--r-- 1 root root 1501242 Oct 7 2011 postfix_
-rw-r--r-- 1 root root 5138 May 7 04:33 linux-headers-
-rw-r--r-- 1 root root 5140 May 7 04:33 linux-server_
-rw-r--r-- 1 root root 5144 May 7 04:33 linux-image-
-rw-r--r-- 1 root root 10175388 May 7 04:34 linux-headers-
-rw-r--r-- 1 root root 31861814 May 7 04:34 linux-image-
-rw-r--r-- 1 root root 837616 May 7 04:34 linux-headers-
-rw-r--r-- 1 root root 335916 May 23 21:04 dhcp3-common_
-rw-r--r-- 1 root root 275594 May 23 21:04 dhcp3-client_
-rw-r--r-- 1 root root 417662 May 29 13:03 libgnutls26_
-rw-r--r-- 1 root root 43626 Jun 5 12:08 libxext6_
-rw-r--r-- 1 root root 43186 Jun 5 12:08 libxcb1_
-rw-r--r-- 1 root root 233946 Jun 5 12:08 libx11-
-rw-r--r-- 1 root root 844966 Jun 5 12:08 libx11-
drwxr-xr-x 2 root root 4096 Jun 12 10:08 partial
drwxr-xr-x 3 root root 4096 Jun 12 10:08 .
drwxr-xr-x 3 root root 4096 Jun 12 10:11 ..