Unknown LVM metadata header

[Mike Hobbs]

Binary package hint: grub2

update-grub2 appears to run successfully, but generates a lot of "error: Unknown LVM metadata header" messages in the process. LVM was created during install on a clean system (using ubuntu-9.10-beta-alternate-i386.iso).

$ sudo update-grub2
error: Unknown LVM metadata header
error: Unknown LVM metadata header
Generating grub.cfg ...
error: Unknown LVM metadata header
error: Unknown LVM metadata header
error: Unknown LVM metadata header
error: Unknown LVM metadata header
error: Unknown LVM metadata header
Found linux image: /boot/vmlinuz-2.6.31-14-generic
Found initrd image: /boot/initrd.img-2.6.31-14-generic
error: Unknown LVM metadata header
error: Unknown LVM metadata header
error: Unknown LVM metadata header
error: Unknown LVM metadata header
error: Unknown LVM metadata header
error: Unknown LVM metadata header
Found linux image: /boot/vmlinuz-2.6.31-11-generic
Found initrd image: /boot/initrd.img-2.6.31-11-generic
error: Unknown LVM metadata header
error: Unknown LVM metadata header

Here's some extra info that may help:

$ mount
/dev/mapper/root-root on / type ext4 (rw,noatime,errors=remount-ro)
proc on /proc type proc (rw)
none on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type tmpfs (rw,mode=0755)
none on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
none on /dev/shm type tmpfs (rw,nosuid,nodev)
none on /var/run type tmpfs (rw,nosuid,mode=0755)
none on /var/lock type tmpfs (rw,noexec,nosuid,nodev)
none on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
/dev/sda1 on /boot type ext2 (rw,noexec,nosuid,nodev,noatime)

$ ls -la /boot
total 26783
drwxr-xr-x 4 root root 1024 2009-10-15 11:27 .
drwxr-xr-x 22 root root 4096 2008-11-18 16:28 ..
-rw-r--r-- 1 root root 628340 2009-09-25 05:18 abi-2.6.31-11-generic
-rw-r--r-- 1 root root 629174 2009-10-15 01:09 abi-2.6.31-14-generic
-rw-r--r-- 1 root root 111218 2009-09-25 05:18 config-2.6.31-11-generic
-rw-r--r-- 1 root root 111371 2009-10-15 01:09 config-2.6.31-14-generic
drwxr-xr-x 2 root root 3072 2009-10-15 11:58 grub
-rw-r--r-- 1 root root 7297493 2008-11-17 15:49 initrd.img-2.6.31-11-generic
-rw-r--r-- 1 root root 7283454 2009-10-15 11:27 initrd.img-2.6.31-14-generic
drwxr-xr-x 2 root root 12288 2008-11-17 15:24 lost+found
-rw-r--r-- 1 root root 128796 2009-09-28 05:17 memtest86+.bin
-rw-r--r-- 1 root root 1657331 2009-09-25 05:18 System.map-2.6.31-11-generic
-rw-r--r-- 1 root root 1664737 2009-10-15 01:09 System.map-2.6.31-14-generic
-rw-r--r-- 1 root root 1196 2009-09-25 05:21 vmcoreinfo-2.6.31-11-generic
-rw-r--r-- 1 root root 1196 2009-10-15 01:12 vmcoreinfo-2.6.31-14-generic
-rw-r--r-- 1 root root 3873888 2009-09-25 05:18 vmlinuz-2.6.31-11-generic
-rw-r--r-- 1 root root 3890400 2009-10-15 01:09 vmlinuz-2.6.31-14-generic

$ sudo sfdisk -l /dev/sda

Disk /dev/sda: 974 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start End #cyls #blocks Id System
/dev/sda1 * 0+ 30 31- 248976 83 Linux
/dev/sda2 31 973 943 7574647+ 8e Linux LVM
/dev/sda3 0 - 0 0 0 Empty
/dev/sda4 0 - 0 0 0 Empty

ProblemType: Bug
Architecture: i386
Date: Thu Oct 15 11:59:41 2009
DistroRelease: Ubuntu 9.10
Package: grub-pc 1.97~beta4-1ubuntu1
ProcEnviron:
 LANG=C
 SHELL=/bin/bash
ProcVersionSignature: Ubuntu 2.6.31-14.47-generic
SourcePackage: grub2
Uname: Linux 2.6.31-14-generic i686

[Mike Hobbs (mike-hobbshouse)]

Sorry, the update-grub2 output above is not complete. Here's the complete output:

$ sudo update-grub2
error: Unknown LVM metadata header
error: Unknown LVM metadata header
Generating grub.cfg ...
error: Unknown LVM metadata header
error: Unknown LVM metadata header
error: Unknown LVM metadata header
error: Unknown LVM metadata header
error: Unknown LVM metadata header
Found linux image: /boot/vmlinuz-2.6.31-14-generic
Found initrd image: /boot/initrd.img-2.6.31-14-generic
error: Unknown LVM metadata header
error: Unknown LVM metadata header
error: Unknown LVM metadata header
error: Unknown LVM metadata header
error: Unknown LVM metadata header
error: Unknown LVM metadata header
Found linux image: /boot/vmlinuz-2.6.31-11-generic
Found initrd image: /boot/initrd.img-2.6.31-11-generic
error: Unknown LVM metadata header
error: Unknown LVM metadata header
error: Unknown LVM metadata header
error: Unknown LVM metadata header
error: Unknown LVM metadata header
error: Unknown LVM metadata header
Found memtest86+ image: /memtest86+.bin
  Incorrect metadata area header checksum
done

[Mike Hobbs (mike-hobbshouse)]

Never mind. This appears to be a bug with LVM during install. pvscan returns metadata errors as well. I'll create a new bug for the LVM package.

$ sudo pvscan
  Incorrect metadata area header checksum
  PV /dev/sda2 VG root lvm2 [7.22 GB / 724.00 MB free]
  PV /dev/sda1 lvm2 [7.22 GB]
  Total: 2 [14.45 GB] / in use: 1 [7.22 GB] / in no VG: 1 [7.22 GB]

[Felix Zielcke (fzielcke)]

Am Donnerstag, den 15.10.2009, 18:32 +0000 schrieb Mike Hobbs:
> Never mind. This appears to be a bug with LVM during install. pvscan
> returns metadata errors as well. I'll create a new bug for the LVM
> package.
>
> $ sudo pvscan
> Incorrect metadata area header checksum
> PV /dev/sda2 VG root lvm2 [7.22 GB / 724.00 MB free]
> PV /dev/sda1 lvm2 [7.22 GB]
> Total: 2 [14.45 GB] / in use: 1 [7.22 GB] / in no VG: 1 [7.22 GB]
>

There's already bug #385428 for unknown LVM metadata, but for that
submitter it's probable because the LVM itself is too old.

I just installed karmic beta1 on the default LVM setup on a 20 GiB
virtual disk. That worked fine, at least grub-probe didn't complain
about it.

So sda is the only disk in your system?
grub-probe always reads the LVM metadata on every disk it can find, not
only on the one you requested.

--
Felix Zielcke
Proud Debian Maintainer and GNU GRUB developer

[Jeff Sereno (jsereno)]

I have this "unknown LVM metadata header" error as well. Installed Karmic Beta (using the Alternate installer) onto a brand new software RAID5 setup - entire RAID volume used for LVM. Fully up to date with the current updates.

I sometimes (not always) see the error just as GRUB loads and before the Ubuntu logo appears for loading, and whenever update-grub runs.

So far I have not seen any general issues in day to day usage, but I also notice that I don't get the error with the pvscan command:

$ sudo pvscan
  PV /dev/md0 VG RAID5-2TB lvm2 [1.82 TB / 0 free]
  Total: 1 [1.82 TB] / in use: 1 [1.82 TB] / in no VG: 0 [0 ]
$

[Mike Hobbs (mike-hobbshouse)]

I discovered that the source of my problem was the way I had been messing around with partitions during install. I had initially let it configure LVM with default settings. I didn't like the way it put my LVM partition and the start of the disk and my boot partition at the end of the disk, so I deleted all the partitions and reconfigured everything manually. Well, pvcreate puts metadata in certain sectors on the disk and when I deleted the partitions and recreated the boot partition from scratch at the beginning of the disk, that metadata was still there, but now affecting the (non-LVM) boot partition. That stray metadata on my boot partition was confusing pvscan (and apparently update-grub2 as well). A simple 'pvremove /dev/sda1' cleared everything up.

[Mike Hobbs (mike-hobbshouse)]

For posterity, the actual bug is now in partman-lvm: https://bugs.launchpad.net/ubuntu/+source/partman-lvm/+bug/453276

[Mike Hobbs (mike-hobbshouse)]

Re: Jeff Sereno, have you tried pvck and vgck to verify that there really is no error?

$ sudo pvck /dev/md0
$ sudo vgck RAID5-2TB

[Trey Blancher (ectospasm)]

I had this same problem. Leftover LVM metadata from some failed installs (due to bad RAM) caused pvscan to see my /boot partition as an extra erroneously-sized pv (~950GB instead of ~250MB). Unmounting /boot, running "pvremove /dev/sda1", replacing the UUID string in /etc/fstab with /dev/hda1, and /boot is back to normal working order. I haven't done a full reboot test yet, but I'd be surprised if the error returns, or if I have boot problems.

[Svein Seldal (sveinse)]

I have a fresh installation of Lucid with manually modified lvm2 partitions. I have /dev/sda1 as ext2 /boot partition, while /dev/sda5 is a PV for the LVM2. I got the same header checksum error from all lvm2 tools and update-grub.

Doing a "pvscan" revealed that /dev/sda1 is an available PV member of the LVM, when it's only used as a standard partition for /boot. I don't know if that's good or bad, but Mike's solution worked. However I had to do some extra steps on a running system:

  - umount /dev/sda1
  - pvremove /dev/sda1
  - blkid /dev/sda1
and then replace the UUID into /etc/fstab where /dev/sda1 is mounted (/boot)
  - mount /dev/sda1

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Mark Mentovai (mark-moxienet)]

I don’t believe there’s any bug with GRUB here. In my case, I found “leftover” PV headers on /dev/sda1 and /dev/sdb1 at offset 0x200 (sector 1). I suspect that these were created in a similar way to what Mike Hobbs experienced in https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/452350 . I saw this with the 11.10 Alternate installer; I didn’t like the what “auto” did but I guess by the time I switched to “manual,” the PV headers had been written. In my current installation, /dev/sda1 and /dev/sdb1 are in a RAID mirror set that shows up at /dev/md0, which is used as an LVM PV.

The PV headers at /dev/sda1 and /dev/sdb1 offset 0x200 (grub_lvm_label_header with "LABELONE" magic) referenced metadata expected at offset 0x1000, but because /dev/sda1 and /dev/sdb1 are RAID members and version 1.2, the RAID superblock (grub_raid_super_1x or <linux/raid/md_p.h> mdp_superblock_1 with MD_SB_MAGIC magic) was written at that offset. This doesn’t match the LVM MD metadata, which would be grub_lvm_mda_header with GRUB_LVM_FMTT_MAGIC magic.

GRUB was correct to issue the “unknown LVM metadata header” error because it did find an LVM label pointing to something other than LVM metadata. The problem was that the LVM label didn’t belong there at all.

In my case, I was able to remove the label by brute force (although I wouldn’t recommend doing this) by running dd if=/dev/zero of=/dev/sda1 bs=512 count=2, and repeating for /dev/sdb1.

[Clay Jackson (clayj)]

I'm seeing the same problem in 11.10 server - and none of the suggestions around pvcreate/pv -anything seem to work.

I'm a bit reluctant to brute force things w/o a bunch more confirmation.

Help???

[Ian Macintosh (ian-macintosh)]

I saw this as well on a server where a forgotten PV label caused the error message.

In my case, I sequentially checked all the partitions and raid volumes with 'pvck' until I found the incorrectly marked one, in my case /dev/sdh1. This partition was part of an MD array, so I failed it from the raid 6 array, removed it, then wiped the label with 'pvremove /dev/sdh1' -ff. It forced me to use the -ff flag in this case, because the PV label incorrectly indicated that it belonged to an active VG. The header was wrong, so using -ff was the correct solution. I re-added /dev/sdh1 back into the array and it resynced without an issue. Problem gone and no further error messages.

In summary, I do not believe this to be a bug, but rather a correctly given error message from LVM where it detects inconsistent or missing data.

The solution I recommend is to hunt down the incorrect PV label and wipe it. Naturally, taking care not to fiddle with the device or partition if it is in use elsewhere when wiping the PV label is a good idea.

I suspect that using pvremove on a device, even when it is in use elsewhere may potentially be safe _because_ the label would not be there in the first place if that particular block was in use. YMMV.

[Matt Fischer (mfisch)]

I just did a clean install on my laptop with a simple setup of 1 VG and 2 LVs and I'm already getting the same output.

[Kenneth Wrede (kennethwrede)]

I got the problem when I upgraded from Lucid to Precise (beta, a few days before release).

When I start the computer Grub seems to run but do not show up. If I continueusly pressing enter the system is booting anyway. If I run "sudo update-grub" I'm getting the error messages described above.
I can't tell if it is two aspects of the same problem or two different.

[Jeff Sereno (jsereno)]

I just did a fresh install (not upgrade) of Precise Server 64-bit on a six 1.5TB RAID5 setup and I'm getting the same errors again.

$ sudo pvck /dev/md0
  Found label on /dev/md0, sector 1, type=LVM2 001

  Found text metadata area: offset=4096, size=2617344

$ sudo vgck -v HTPC
    Using volume group(s) on command line
    Finding volume group "HTPC"

$ sudo update-grub
error: unknown LVM metadata header.
error: unknown LVM metadata header.
error: unknown LVM metadata header.
error: unknown LVM metadata header.
Generating grub.cfg ...
error: unknown LVM metadata header.
error: unknown LVM metadata header.
error: unknown LVM metadata header.
error: unknown LVM metadata header.
error: unknown LVM metadata header.
error: unknown LVM metadata header.
error: unknown LVM metadata header.
error: unknown LVM metadata header.
error: unknown LVM metadata header.
error: unknown LVM metadata header.
error: unknown LVM metadata header.
error: unknown LVM metadata header.
error: unknown LVM metadata header.
error: unknown LVM metadata header.
error: unknown LVM metadata header.
error: unknown LVM metadata header.
error: unknown LVM metadata header.
error: unknown LVM metadata header.
error: unknown LVM metadata header.
error: unknown LVM metadata header.
error: unknown LVM metadata header.
error: unknown LVM metadata header.
Found linux image: /boot/vmlinuz-3.2.0-24-generic
Found initrd image: /boot/initrd.img-3.2.0-24-generic
error: unknown LVM metadata header.
error: unknown LVM metadata header.
error: unknown LVM metadata header.
error: unknown LVM metadata header.
error: unknown LVM metadata header.
error: unknown LVM metadata header.
error: unknown LVM metadata header.
error: unknown LVM metadata header.
Found linux image: /boot/vmlinuz-3.2.0-23-generic
Found initrd image: /boot/initrd.img-3.2.0-23-generic
error: unknown LVM metadata header.
error: unknown LVM metadata header.
error: unknown LVM metadata header.
error: unknown LVM metadata header.
error: unknown LVM metadata header.
error: unknown LVM metadata header.
error: unknown LVM metadata header.
error: unknown LVM metadata header.
error: unknown LVM metadata header.
error: unknown LVM metadata header.
Found memtest86+ image: /boot/memtest86+.bin
done

$ sudo mdadm --detail /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Mon Apr 30 21:41:04 2012
     Raid Level : raid5
     Array Size : 7325678080 (6986.31 GiB 7501.49 GB)
  Used Dev Size : 1465135616 (1397.26 GiB 1500.30 GB)
   Raid Devices : 6
  Total Devices : 6
    Persistence : Superblock is persistent

    Update Time : Mon Apr 30 22:59:13 2012
          State : active, resyncing
 Active Devices : 6
Working Devices : 6
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

  Resync Status : 0% complete

           Name : htpc:0 (local to host htpc)
           UUID : 25f65ca0:f80f7b06:4f94157c:0b27a779
         Events : 5

    Number Major Minor RaidDevice State
       0 8 1 0 active sync /dev/sda1
       1 8 17 1 active sync /dev/sdb1
       2 8 33 2 active sync /dev/sdc1
       3 8 49 ...

[Teodori Serge (teodori-serge)]

Hello I known why this error appears (in my case at least), if you just delete partition information and then recreate new ones for lvm2, it might happen that the metadata of the deleted lvm2 partitions is still present. The easiest way to correct this errors is to wipe out the old metadata definitions. Use hexedit and zero those sectorts from your devices, but this is very dangerous... You should ask lvm2 team to help you. Grub2 cannot fix this, but a work around could be written.

[Teodori Serge (teodori-serge)]

OK found how to correct it :) first you need "dd" and "hexedit".

Important you must understand this.

LVM2 stores vg metadata informations in the first 4MB of all its pvs.
After every update on a vg a new "Metadata Sequence No" is created.
Every pv has its own metadata "Metadata Areas"

So do "pvscan", "vgdisplay" and make out where the metadata area is. (for 'x' pv do 'dd')
After that copy metadata with 'dd', and analyse the output of 'dd' with the one of 'vgdisplay'.
If you are sure you have found the bad blocks open 'hexedit' on device and wipeout bad parts.

please keep me informed about any changes <email address hidden>

[iMac (imac-netstatz)]

pvremove could not find the device in my case. It said "Couldn't find device. Check your filters?" with -ff and "Physical Volume /dev/sda5 note found" without -ff. pvck and pvscan were of no use identifying the label, however I knew that I had moved LVM from /dev/sda5 to /dev/sda6 using the alternate installer (similar to comments above).

I found this post below and was able to see the label myself.
http://www.redhat.com/archives/linux-lvm/2008-February/msg00011.html

dd if=/dev/sda5 bs=1k count=1 | hexdump -e '"%_p"'

so there it is. but when I use pvremove -vvvv /dev/sda5 I can see the "Skipping md component device" rule being applied. How do I turn this off??

Interestingly, I have another device that is a raid device, that has LVM on it, and it looks the same, so I get how grub2 is getting confused.

The solution was to turn md detection off in /etc/lvm/lvm.conf and re-run the pvremove command. The labels were "successfully wiped".

[Dominic Dunlop (dominic-dunlop)]

I've been bitten too. I have a software RAID 10, /dev/md0, which is made up of /dev/sd[a-d]2. /dev/md0 is the only "physical" volume in a volume group. Looking at the first two sectors of /dev/sd[a-d]2, there are indeed volume group manager labels, but this is as I would expect: they're holding data for the first two sectors of /dev/md0 — the md driver's overhead loses a little space at the end of the volume, but none at the beginning. If I point pvck at /dev/sd[a-d]2, it correctly ignores them, as they're part of an md device. I'm not about to zap these labels, as I think they're valid. Could the problem be that update-grub (and, incidentally, the boot process, where one can see the same error message flitting by) is not as smart as mdadm, and is not ignoring partitions that are claimed by an md device?

[Kenneth Wrede (kennethwrede)]

This bug is solved for me. It dissapeard with an update.

Does someone still have problems?

[Simon Déziel (sdeziel)]

This is still occuring on my Precise server.

[Dominic Dunlop (dominic-dunlop)]

Still appearing on 12.04.2 for me.

On 27 Aug, 2013, at 20:32 , Kenneth Wrede <email address hidden> wrote:

> This bug is solved for me. It dissapeard with an update.
>
> Does someone still have problems?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/452350
>
> Title:
> Unknown LVM metadata header
>
> Status in “grub2” package in Ubuntu:
> Confirmed
> Status in “grub2” package in Debian:
> Confirmed
>
> Bug description:
> Binary package hint: grub2
>
> update-grub2 appears to run successfully, but generates a lot of
> "error: Unknown LVM metadata header" messages in the process. LVM was
> created during install on a clean system (using ubuntu-9.10-beta-
> alternate-i386.iso).
>
> $ sudo update-grub2
> error: Unknown LVM metadata header
> error: Unknown LVM metadata header
> Generating grub.cfg ...
> error: Unknown LVM metadata header
> error: Unknown LVM metadata header
> error: Unknown LVM metadata header
> error: Unknown LVM metadata header
> error: Unknown LVM metadata header
> Found linux image: /boot/vmlinuz-2.6.31-14-generic
> Found initrd image: /boot/initrd.img-2.6.31-14-generic
> error: Unknown LVM metadata header
> error: Unknown LVM metadata header
> error: Unknown LVM metadata header
> error: Unknown LVM metadata header
> error: Unknown LVM metadata header
> error: Unknown LVM metadata header
> Found linux image: /boot/vmlinuz-2.6.31-11-generic
> Found initrd image: /boot/initrd.img-2.6.31-11-generic
> error: Unknown LVM metadata header
> error: Unknown LVM metadata header
>
> Here's some extra info that may help:
>
> $ mount
> /dev/mapper/root-root on / type ext4 (rw,noatime,errors=remount-ro)
> proc on /proc type proc (rw)
> none on /sys type sysfs (rw,noexec,nosuid,nodev)
> none on /sys/fs/fuse/connections type fusectl (rw)
> none on /sys/kernel/debug type debugfs (rw)
> none on /sys/kernel/security type securityfs (rw)
> udev on /dev type tmpfs (rw,mode=0755)
> none on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
> none on /dev/shm type tmpfs (rw,nosuid,nodev)
> none on /var/run type tmpfs (rw,nosuid,mode=0755)
> none on /var/lock type tmpfs (rw,noexec,nosuid,nodev)
> none on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
> /dev/sda1 on /boot type ext2 (rw,noexec,nosuid,nodev,noatime)
>
> $ ls -la /boot
> total 26783
> drwxr-xr-x 4 root root 1024 2009-10-15 11:27 .
> drwxr-xr-x 22 root root 4096 2008-11-18 16:28 ..
> -rw-r--r-- 1 root root 628340 2009-09-25 05:18 abi-2.6.31-11-generic
> -rw-r--r-- 1 root root 629174 2009-10-15 01:09 abi-2.6.31-14-generic
> -rw-r--r-- 1 root root 111218 2009-09-25 05:18 config-2.6.31-11-generic
> -rw-r--r-- 1 root root 111371 2009-10-15 01:09 config-2.6.31-14-generic
> drwxr-xr-x 2 root root 3072 2009-10-15 11:58 grub
> -rw-r--r-- 1 root root 7297493 2008-11-17 15:49 initrd.img-2.6.31-11-generic
> -rw-r--r-- 1 root root 7283454 2009-10-15 11:27 initrd.img-2.6.31-14-generic
> drwxr-xr-x 2 root root 12288 2008-11-17 15:24 lost+found
> -rw-r--r-- 1 root root 128796 2009-09-28 05:17 memtest86+.bin
> -rw-r--r-- 1 ...

[Marcus Tomlinson (marcustomlinson)]

This release of Ubuntu is no longer receiving maintenance updates. If this is still an issue on a maintained version of Ubuntu please let us know.

[Marcus Tomlinson (marcustomlinson)]

This issue has sat incomplete for more than 60 days now. I'm going to close it as invalid. Please feel free re-open if this is still an issue for you. Thank you.