Fuel creates boot partitions on unalocated drives

[Stepan Rogov]

Versions: Fuel 7.0, 8.0

Model: Virtio Block Device (virtblk)
Disk /dev/vdc: 376GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number Start End Size File system Name Flags
 1 1049kB 26.2MB 25.2MB primary bios_grub
 2 26.2MB 236MB 210MB primary

How to reproduce:
Deploy Fuel environment with node with at least one unallocated drive.

[Bug Checker Bot (bug-checker)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

expected result

steps to reproduce

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Vladimir Kozhukalov (kozhukalov)]

Could you please expose the source code of /usr/lib/python2.7/site-packages/fuel-agent/drivers/nailgun.py?
Looks like we fixed this long time ago [1]

[1] https://github.com/openstack/fuel-agent/blob/master/fuel_agent/drivers/nailgun.py#L327-L334

[Alexander Gordeev (a-gordeev)]

It looks like fuel-agent should skip 'unallocated drives' anyway.

> Number Start End Size File system Name Flags
> 1 1049kB 26.2MB 25.2MB primary bios_grub
> 2 26.2MB 236MB 210MB primary

this isn't a boot partition. It's BIOS_GRUB partition in which grub core.img resides in case of non-native UEFI boot.

So, your expectation is that fuel-agent shouldn't touch this drive at all, right?

Fuel-agent is never doing anything on its own mind. Perhaps, nailgun's volume manager generated partitioning layout even for 'unallocated' drive. I'm not sure.

Please share with us fuel-agent.log file from the node (usually it's stored somewhere under /var/log/docker-logs/remote/<node-id>/bootstrap/fuel-agent.log on fuel-master). This log file captures all input data which nailgun sent.

Then, we could proceed with analysis further.

As Vladimir already mentioned above, fuel-agent do contain the code which should skip unallocated drives from processing. It could be a bug in this code, but i'm not sure. Without fuel-agent log file it's hard to determine the root cause.

[Stepan Rogov (srogov)]

/dev/vdb and /dev/vdc should have no partitions, because its are unallocated

[Alexander Gordeev (a-gordeev)]

Ok. I got it.

volume manager created /boot partition on mirrored raid by default.

https://github.com/openstack/fuel-web/blob/stable/7.0/nailgun/nailgun/extensions/volume_manager/manager.py#L605-L618

If node has more than 1 disks, then boot_is_raid becomes True, and /boot partition will be placed on MD raid constructed from 200M partitions from all of the disks.

Therefore, It's not a fuel-agent's issue.

[Bug Checker Bot (bug-checker)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

expected result

steps to reproduce

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Stepan Rogov (srogov)]

Yep, but there is no any raids on this node

# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
unused devices: <none>

[Alexander Gordeev (a-gordeev)]

On the screenshot you can see only high level disk roles, for example: such as 'base system' or 'storage' .

These disk roles don't reflect to the actual low-level partitioning scheme and don't expose such details as partitions. Any partition for service needs (such as uefi system partition or even special 'bios_grub' partition) is not visible from UI perspective.

So, from UI point of view, it's true that those disks are unallocated. There're no high level roles has been assigned to any of them.

But, they did contain partitions for service needs (such as BIOS_GRUB and UEFI partition).

There's no /boot partition on /dev/vdc and /dev/vdb. The first partition is BIOS_GRUB and the second is for UEFI.

Even if volume manager asked to create /boot on MD raid, fuel-agent refused to do that and created only one plain partition named under /dev/vda3 for /boot.

The reason why fuel-agent created service partitions on them is that volume-manager from nailgun provided the default partitioning scheme which assumes that /boot partition is spread across all disks on MD raid. That's all.

[Stepan Rogov (srogov)]

I think that a user is entitled to rely on the fact that the discs will remain unallocated (without any partitions).

[Alexander Gordeev (a-gordeev)]

Ok, it seems like the code [1] for skipping disks doesn't work as expected anymore.

[1] https://github.com/openstack/fuel-agent/commit/9c06fba042660cd3701de58ca6f727949be7ee62

So, my suggestion to fuel-python team: check/debug and probably fix it. (Input data for debugging purposes could be obtained from attached fuel-agent log)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/310416

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/310416
Committed: https://git.openstack.org/cgit/openstack/fuel-agent/commit/?id=825468cb3bb1eb1e6a5e7cd665d89496998cdb54
Submitter: Jenkins
Branch: master

commit 825468cb3bb1eb1e6a5e7cd665d89496998cdb54
Author: Alexander Gordeev <email address hidden>
Date: Wed Apr 27 14:21:42 2016 +0300

    Enable unallocated drives back

    Apparently fix for this bug
    https://bugs.launchpad.net/fuel/+bug/1258347
    turned out to be non-working anyway.
    The reason was that 'lvm_meta_pool' sometimes has non-zero size.
    Due to its non-zero size the disk wasn't skipped from partitioning,
    thus creating a bunch of partitions even on unallocated drives.

    Change-Id: Ifaf8cb6a1f9d5010998b89445c3e575f3957411b
    Closes-Bug: #1566399

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/310815

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/310815
Committed: https://git.openstack.org/cgit/openstack/fuel-agent/commit/?id=6e89fb83e44a7cd3b286308b661f645378576d19
Submitter: Jenkins
Branch: stable/mitaka

commit 6e89fb83e44a7cd3b286308b661f645378576d19
Author: Alexander Gordeev <email address hidden>
Date: Wed Apr 27 14:21:42 2016 +0300

    Enable unallocated drives back

    Apparently fix for this bug
    https://bugs.launchpad.net/fuel/+bug/1258347
    turned out to be non-working anyway.
    The reason was that 'lvm_meta_pool' sometimes has non-zero size.
    Due to its non-zero size the disk wasn't skipped from partitioning,
    thus creating a bunch of partitions even on unallocated drives.

    Change-Id: Ifaf8cb6a1f9d5010998b89445c3e575f3957411b
    Closes-Bug: #1566399
    (cherry picked from commit 825468cb3bb1eb1e6a5e7cd665d89496998cdb54)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/320860

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/320890

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/320860
Committed: https://git.openstack.org/cgit/openstack/fuel-agent/commit/?id=8cfa48bbcadc5a4f3ccf45a3d38708985f02c4c1
Submitter: Jenkins
Branch: stable/8.0

commit 8cfa48bbcadc5a4f3ccf45a3d38708985f02c4c1
Author: Alexander Gordeev <email address hidden>
Date: Wed Apr 27 14:21:42 2016 +0300

    Enable unallocated drives back

    Apparently fix for this bug
    https://bugs.launchpad.net/fuel/+bug/1258347
    turned out to be non-working anyway.
    The reason was that 'lvm_meta_pool' sometimes has non-zero size.
    Due to its non-zero size the disk wasn't skipped from partitioning,
    thus creating a bunch of partitions even on unallocated drives.

    Change-Id: Ifaf8cb6a1f9d5010998b89445c3e575f3957411b
    Closes-Bug: #1566399
    (cherry picked from commit 825468cb3bb1eb1e6a5e7cd665d89496998cdb54)
    (cherry picked from commit 6e89fb83e44a7cd3b286308b661f645378576d19)

[Maksym Strukov (unbelll)]

Verified as fixed in 9.0-404

[Sergii Rizvan (srizvan)]

Steps to reproduce:
You need at least one node in the environment with more than one disk.
Go to MOS web GUI > Environment > Nodes > Click on the node with more than one disk > Configure disks > Remove all partitions from the second disk, so you can see that space of the whole disk is unnalocated.
Deploy the environment.
After OS provisioning login by SSH to the selected node and run next command (/dev/vdb here is unnalocated drive):
root@node-4:~# parted /dev/vdb print
Model: Virtio Block Device (virtblk)
Disk /dev/vdb: 376GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number Start End Size File system Name Flags
 1 1049kB 26.2MB 25.2MB primary bios_grub
 2 26.2MB 236MB 210MB primary

Here we can see, that fuel-agent have created 2 partitions, but the drive was marked as unnalocated.
After applying the patch fuel-agent doesn't create any partitions on unnalocated disks.

[TatyanaGladysheva (tgladysheva)]

Verified on MOS 8.0 + MU2 updates.

Actual results:
Before updates:
/dev/vdb - unallocated drive. After OS provisioning:
root@node-1:~# parted /dev/vdb print
Model: Virtio Block Device (virtblk)
Disk /dev/vdb: 53.7GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number Start End Size File system Name Flags
 1 1049kB 26.2MB 25.2MB primary bios_grub
 2 26.2MB 236MB 210MB primary

root@node-1:~#

After updates:
/dev/vdb - unallocated drive. After OS provisioning:
root@node-1:~# parted /dev/vdb print
Error: /dev/vdb: unrecognised disk label
root@node-1:~#

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/320890
Committed: https://git.openstack.org/cgit/openstack/fuel-agent/commit/?id=abe76d7b3ce91c438d9519c5dff84c17a0e33de3
Submitter: Jenkins
Branch: stable/7.0

commit abe76d7b3ce91c438d9519c5dff84c17a0e33de3
Author: Alexander Gordeev <email address hidden>
Date: Wed Apr 27 14:21:42 2016 +0300

    Enable unallocated drives back

    Apparently fix for this bug
    https://bugs.launchpad.net/fuel/+bug/1258347
    turned out to be non-working anyway.
    The reason was that 'lvm_meta_pool' sometimes has non-zero size.
    Due to its non-zero size the disk wasn't skipped from partitioning,
    thus creating a bunch of partitions even on unallocated drives.

    Conflicts:
            fuel_agent/tests/test_nailgun.py

    Change-Id: Ifaf8cb6a1f9d5010998b89445c3e575f3957411b
    Closes-Bug: #1566399
    (cherry picked from commit 825468cb3bb1eb1e6a5e7cd665d89496998cdb54)
    (cherry picked from commit 6e89fb83e44a7cd3b286308b661f645378576d19)
    (cherry picked from commit 8cfa48bbcadc5a4f3ccf45a3d38708985f02c4c1)

[TatyanaGladysheva (tgladysheva)]

Verified on MOS 7.0 + MU5 updates.

Actual results:
/dev/vdb - unallocated drive. After OS provisioning the following results are observed:

Before updates:
root@node-2:~# parted /dev/vdb print
Model: Virtio Block Device (virtblk)
Disk /dev/vdb: 53.7GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number Start End Size File system Name Flags
 1 1049kB 26.2MB 25.2MB primary bios_grub
 2 26.2MB 236MB 210MB primary msftdata

After updates:
root@node-10:~# parted /dev/vdb print
Error: /dev/vdb: unrecognised disk label

[TatyanaGladysheva (tgladysheva)]

Verified on 10.0 build #1569.

Actual results:
/dev/vdb - unallocated drive. After OS provisioning the following results are observed:

root@node-2:~# parted /dev/vdb print
Error: /dev/vdb: unrecognised disk label
Model: Virtio Block Device (virtblk)
Disk /dev/vdb: 53.7GB
Sector size (logical/physical): 512B/512B
Partition Table: unknown
Disk Flags:
root@node-2:~#