[2.0b5] deployment fails the second time LVM is deployed on virtio

Bug #1585016 reported by Lee Trager on 2016-05-24
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Critical
Mike Pontillo

Bug Description

I had a machine deployed with a large ext4 logical volume over two physical volumes. I deployed a rack controller on it for some testing. After I was done I released the machine and tried to redeploy but got the following error. I've been able to reproduce this by deploying a machine with LVM, releasing it, commissioning it again, and trying to redeploy an OS.

  Logical volume "lv0" successfully removed
  Volume group "lv0" not found
  Cannot process volume group lv0
  Volume group "vg0" successfully removed
An error occured handling 'vdb': FileNotFoundError - [Errno 2] No such file or directory: '/sys/block/lvm-pv-uuid-aIDI1l-jCya-1jLL-w3Si-Jn6W-RKPr-XP6Ssx/holders'
[Errno 2] No such file or directory: '/sys/block/lvm-pv-uuid-aIDI1l-jCya-1jLL-w3Si-Jn6W-RKPr-XP6Ssx/holders'
Installation failed with exception: Unexpected error while running command.
Command: ['curtin', 'block-meta', 'custom']
Exit code: 3
Reason: -
Stdout: b' Logical volume "lv0" successfully removed\n Volume group "lv0" not found\n Cannot process volume group lv0\n Volume group "vg0" successfully removed\nAn error occured handling \'vdb\': FileNotFoundError - [Errno 2] No such file or directory: \'/sys/block/lvm-pv-uuid-aIDI1l-jCya-1jLL-w3Si-Jn6W-RKPr-XP6Ssx/holders\'\n[Errno 2] No such file or directory: \'/sys/block/lvm-pv-uuid-aIDI1l-jCya-1jLL-w3Si-Jn6W-RKPr-XP6Ssx/holders\'\n'
Stderr: ''

Related branches

Lee Trager (ltrager) wrote :

I fixed this by wiping the partition table with

dd if=/dev/zero of=/dev/vda bs=1M count=1
dd if=/dev/zero of=/dev/vdb bs=1M count=1

Lee Trager (ltrager) wrote :

Storage commissioning output:

[
 {
  "NAME": "vda",
  "SIZE": "8589934592",
  "RO": "0",
  "RM": "0",
  "BLOCK_SIZE": "4096",
  "MODEL": "",
  "ROTA": "1",
  "PATH": "/dev/vda"
 },
 {
  "NAME": "vdb",
  "SIZE": "8589934592",
  "RO": "0",
  "RM": "0",
  "BLOCK_SIZE": "4096",
  "MODEL": "",
  "ROTA": "1",
  "PATH": "/dev/vdb",
  "ID_PATH": "/dev/disk/by-id/lvm-pv-uuid-5cPTHY-DZA2-W4Vi-qtd4-1Fgt-RiSc-8Wcc1E"
 }
]

Changed in maas:
status: New → Triaged
summary: - Commissing with LVM breaks deployments
+ [2.0b5] Commissing with LVM breaks deployments
Changed in maas:
importance: High → Critical
Changed in maas:
assignee: nobody → Mike Pontillo (mpontillo)
summary: - [2.0b5] Commissing with LVM breaks deployments
+ [2.0b5] curtin fails the second time LVM is deployed

Looks like a problem in curtin's clear_holders(); the */holders path can't be found.

summary: - [2.0b5] curtin fails the second time LVM is deployed
+ [2.0b5] curtin fails the second time LVM is deployed on virtio
Mike Pontillo (mpontillo) wrote :

Looks like after the recommission, we pick up the by-id/<lvm-id> of each disk that belonged to the LVM, but not its serial number. This confuses curtin when we got to deploy, which expects to see a /dev/<device> path (not the ephemeral LVM ID) or a serial number.

I think if we store the /dev/<device> path instead of of the by-id path, *if and only if* we didn't find a serial number on the disk, that will make MAAS more reliable when operating against a set of virtual disks.

summary: - [2.0b5] curtin fails the second time LVM is deployed on virtio
+ [2.0b5] deployment fails the second time LVM is deployed on virtio
Changed in maas:
status: Triaged → Fix Committed
Changed in maas:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers