controller-0 install ERROR: Failed to remove PV

[Bob Church]

Brief Description
-----------------
controller-0 installation failed.
ERROR: Failed to remove PV.

Severity
--------
Critical

Steps to Reproduce
------------------
install the 2022-11-23 build

Create additional partitions, create volume groups, run wipedisk

sudo su -
sgdisk -p /dev/sda
sgdisk --new=0:0:+68G --typecode=6:ba5eba11-0000-1111-2222-000000000001 --change-name=6:" Physical Volume" /dev/sda
sgdisk --new=0:0:+100G --typecode=7:ba5eba11-0000-1111-2222-000000000001 --change-name=7:" Physical Volume" /dev/sda
sgdisk --new=0:0:+50G --typecode=8:ba5eba11-0000-1111-2222-000000000001 --change-name=8:" Physical Volume" /dev/sda
sgdisk -p /dev/sda
partprobe

grep global_filter /etc/lvm/lvm.conf
sudo sed -i 's#/dev/disk/by-path/pci-0000:00:0d.0-ata-1.0-part5|", "r|##g' /etc/lvm/lvm.conf
grep global_filter /etc/lvm/lvm.conf

pvcreate /dev/sda6
pvcreate /dev/sda7
pvcreate /dev/sda8

vgextend cgts-vg /dev/sda6
vgextend cgts-vg /dev/sda7
vgcreate nova-local /dev/sda8

vgs
pvs
lvs

reinstall

Expected Behavior
------------------
install success

Actual Behavior
----------------
install failed

Reproducibility
---------------
This is the first time to see this issue.

System Configuration
--------------------
AIO-SX

Branch/Pull Time/Commit
-----------------------
2022-11-23

Last Pass
---------
2022-11-22

Timestamp/Logs
--------------
2022-11-24 04:30:02.748 kickstart pre-part info: All-in-one-lowlatency Initial partition table for /dev/sdc is:
Error: /dev/sdc: unrecognised disk label
Model: Kingston DataTraveler 3.0 (scsi)
Disk /dev/sdc: 14755MiB
Sector size (logical/physical): 512B/51[ 23.711574] random: vgs: uninitialized urandom read (4 bytes read)
2B
Partition Table: unknown
Disk Flags:
2022-11-24 04:30:02.806 kickstart pre-part info: All-in-one-lowlatency Volume Groups : cgts-vg ; 6 10 11 12
2022-11-24 04:30:02.971 kickstart pre-part info: All-in-one-lowla[ 23.875571] random: vgchange: uninitialized urandom read (4 bytes read)
tency Found ' cgts-vg' volume groups
2022-11-24 04:30:02.972 k[ 23.888452] device-mapper: uevent: version 1.0.3
[ 23.888537] device-mapper: ioctl: 4.43.0-ioctl (2020-10-01) initialised: <email address hidden>
ickstart pre-part info: All-in-one-lowlatency ... disabling cgts-vg
0 logical volume(s) in volume group "cgts-vg" n[ 23.921841] random: vgs: uninitialized urandom read (4 bytes read)
ow active
File descriptor 7 (pipe:[16401]) leaked on vgs invocation. Parent PID 1019: /bin/sh
WARNING: Couldn't find device with uuid H9vvyc-qeDW-hiEi-Rdq1-7pC3-vDR1-ZUU0KS.
WARNING: VG cgts-vg is missing PV H9vvyc-qeDW-hiEi-Rdq1-7pC3-vDR1-ZUU0KS (last written to /dev/sda6).
File descriptor 7 (pipe:[16401]) leaked on pvs invocation. Parent PID 1021: /bin/sh
WARNING: Couldn't find device with uuid H9vvyc-qeDW-hiEi-Rdq1-7pC3-vDR1-ZUU0KS.
WARNING: VG cgts-vg is missing PV H9vvyc-qeDW-hiEi-Rdq1-7pC3-vDR1-ZUU0KS (last written to /dev/sda6).
2022-11-24 04:30:03.096 kickstart pre-part info: All-in-one-lowlatency Removing LVs on cgts-vg.
File descriptor 7 (pipe:[16401]) leaked on lvremove invocation. Parent PID 1026: /bin/sh
WARNING: Couldn't find device with uuid H9vvyc-qeDW-hiEi-Rdq1-7pC3-vDR1-ZUU0KS.
WARNING: VG cgts-vg is missing PV H9vvyc-qeDW-hiEi-Rdq1-7pC3-vDR1-ZUU0KS (last written to /dev/sda6).
WARNING: Couldn't find device with uuid H9vvyc-qeDW-hiEi-Rdq1-7pC3-vDR1-ZUU0KS.
WARNING: Couldn't find device with uuid H9vvyc-qeDW-hiEi-Rdq1-7pC3-vDR1-ZUU0KS.
WARNING: Couldn't find device with uuid H9vvyc-qeDW-hiEi-Rdq1-7pC3-vDR1-ZUU0KS.
WARNING: Couldn't find device with uuid H9vvyc-qeDW-hiEi-Rdq1-7pC3-vDR1-ZUU0KS.
WARNING: Couldn't find device with uuid H9vvyc-qeDW-hiEi-Rdq1-7pC3-vDR1-ZUU0KS.
WARNING: Couldn't find device with uuid H9vvyc-qeDW-hiEi-Rdq1-7pC3-vDR1-ZUU0KS.
WARNING: Couldn't find device with uuid H9vvyc-qeDW-hiEi-Rdq1-7pC3-vDR1-ZUU0KS.
WARNING: Couldn't find device with uuid H9vvyc-qeDW-hiEi-Rdq1-7pC3-vDR1-ZUU0KS.
WARNING: Couldn't find device with uuid H9vvyc-qeDW-hiEi-Rdq1-7pC3-vDR1-ZUU0KS.
WARNING: Couldn't find device with uuid H9vvyc-qeDW-hiEi-Rdq1-7pC3-vDR1-ZUU0KS.
WARNING: Couldn't find device with uuid H9vvyc-qeDW-hiEi-Rdq1-7pC3-vDR1-ZUU0KS.
WARNING: Couldn't find device with uuid H9vvyc-qeDW-hiEi-Rdq1-7pC3-vDR1-ZUU0KS.
WARNING: Couldn't find device with uuid H9vvyc-qeDW-hiEi-Rdq1-7pC3-vDR1-ZUU0KS.
WARNING: Couldn't find device with uuid H9vvyc-qeDW-hiEi-Rdq1-7pC3-vDR1-ZUU0KS.
WARNING: Couldn't find device with uuid H9vvyc-qeDW-hiEi-Rdq1-7pC3-vDR1-ZUU0KS.
Logical volume "root-lv" successfully removed
Logical volume "var-lv" successfully removed
Logical volume "log-lv" successfully removed
Logical volume "scratch-lv" successfully removed
Logical volume "pgsql-lv" successfully removed
Logical volume "rabbit-lv" successfully removed
Logical volume "platform-lv" successfully removed
Logical volume "extension-lv" successfully removed
Logical volume "docker-lv" successfully removed
Logical volume "etcd-lv" successfully removed
Logical volume "dockerdistribution-lv" successfully removed
Logical volume "backup-lv" successfully removed
Logical volume "kubelet-lv" successfully removed
Logical volume "ceph-mon-lv" successfully removed
File descriptor 7 (pipe:[16401]) leaked on pvs invocation. Parent PID 1028: /bin/sh
WARNING: Couldn't find device with uuid H9vvyc-qeDW-hiEi-Rdq1-7pC3-vDR1-ZUU0KS.
WARNING: VG cgts-vg is missing PV H9vvyc-qeDW-hiEi-Rdq1-7pC3-vDR1-ZUU0KS (last written to [unknown]).
2022-11-24 04:30:03.234 kickstart pre-part warn: All-in-one-lowlatency VG cgts-vg has PVs: /dev/sda5 [unknown], removing them.
2022-11-24 04:30:03.235 kickstart pre-part info: All-in-one-lowlatency Removing PV /dev/sda5.
File descriptor 7 (pipe:[16401]) leaked on pvremove invocation. Parent PID 1034: /bin/sh
WARNING: PV /dev/sda5 is used by VG cgts-vg.
WARNING: Wiping physical volume label from /dev/sda5 of volume group "cgts-vg".
Labels on physical volume "/dev/sda5" successfully wiped.
2022-11-24 04:30:03.402 kickstart pre-part info: All-in-one-lowlatency Removing PV [unknown].
File descriptor 7 (pipe:[16401]) leaked on pvremove invocation. Parent PID 1038: /bin/sh
No device found for [unknown].
[ 24.489788] random: crng init done
[ 24.489791] random: 5 urandom warning(s) missed due to ratelimiting
File descriptor 7 (pipe:[16401]) leaked on pvremove invocation. Parent PID 1043: /bin/sh
No device found for [unknown].
File descriptor 7 (pipe:[16401]) leaked on pvremove invocation. Parent PID 1048: /bin/sh
No device found for [unknown].
File descriptor 7 (pipe:[16401]) leaked on pvremove invocation. Parent PID 1053: /bin/sh
No device found for [unknown].
File descriptor 7 (pipe:[16401]) leaked on pvremove invocation. Parent PID 1058: /bin/sh
No device found for [unknown].
2022-11-24 04:30:03.557 kickstart pre-part warn: All-in-one-lowlatency Error running command 'pvremove -force --force --yes [unknown] 6>& 10>&- 11>&- 12>&-'. Try 1 of 5 retry in 0.5s.
2022-11-24 04:30:03.558 kickstart pre-part warn: All-in-one-lowlatency ret_code: 5, stdout: ''.
2022-11-24 04:30:04.216 kickstart pre-part warn: All-in-one-lowlatency Error running command 'pvremove -force --force --yes [unknown] 6>& 10>&- 11>&- 12>&-'. Try 2 of 5 retry in 0.5s.
2022-11-24 04:30:04.217 kickstart pre-part warn: All-in-one-lowlatency ret_code: 5, stdout: ''.
2022-11-24 04:30:04.877 kickstart pre-part warn: All-in-one-lowlatency Error running command 'pvremove -force --force --yes [unknown] 6>& 10>&- 11>&- 12>&-'. Try 3 of 5 retry in 0.5s.
2022-11-24 04:30:04.878 kickstart pre-part warn: All-in-one-lowlatency ret_code: 5, stdout: ''.
2022-11-24 04:30:05.534 kickstart pre-part warn: All-in-one-lowlatency Error running command 'pvremove -force --force --yes [unknown] 6>& 10>&- 11>&- 12>&-'. Try 4 of 5 retry in 0.5s.
2022-11-24 04:30:05.535 kickstart pre-part warn: All-in-one-lowlatency ret_code: 5, stdout: ''.
2022-11-24 04:30:06.191 kickstart pre-part warn: All-in-one-lowlatency Error running command 'pvremove -force --force --yes [unknown] 6>& 10>&- 11>&- 12>&-'. Try 5 of 5 retry in 0.5s.
2022-11-24 04:30:06.192 kickstart pre-part warn: All-in-one-lowlatency ret_code: 5, stdout: ''.
Installation failed.
2022-11-24 04:30:06.695 kickstart pre-part warn: All-in-one-lowlatency Installation Failed: ERROR: Failed to remove PV.
... dropping to shell ; exit to reboot ...
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell
[?2004hroot@(none):/#

Test Activity
-------------
installation

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/metal/+/865979

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/metal/+/865980

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/metal/+/865981

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/metal/+/865979
Committed: https://opendev.org/starlingx/metal/commit/651bd7656603f6532ba6ae71f26bc4baa2e01353
Submitter: "Zuul (22348)"
Branch: master

commit 651bd7656603f6532ba6ae71f26bc4baa2e01353
Author: Robert Church <email address hidden>
Date: Mon Nov 28 19:55:12 2022 -0600

    Ensure magic strings that are visible for libblkid are erased

    In the case when the root disk partition table is wiped but individual
    partitions are not wiped correctly, this will leave previous physical
    volume metadata intact on the disk.

    When a new LVM partition is created and assigned as a newly created
    physical volume the old LVM metadata on the disk partition will prevent
    the cgts-vg volume group from being created.

    This update will wipe all the magic strings present in the new physical
    volume partition established by the kickstart by executing 'wipefs -a'
    prior to creating the cgts-vg.

    Test Plan:
    PASS - Successfully install an ISO with this change on a system that did
           not cleanup the LVM metatadata from a previous install. Log in to
           the installed system and confirm that the cgts-vg is properly
           configured.

    Change-Id: I63f4235a27cb40a4283f0f4c34f63564a4f18cdd
    Partial-Bug: #1998204
    Signed-off-by: Robert Church <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/metal/+/865980
Committed: https://opendev.org/starlingx/metal/commit/b0066dcd270117cd12360ee6469f1e7ac0f4d4e7
Submitter: "Zuul (22348)"
Branch: master

commit b0066dcd270117cd12360ee6469f1e7ac0f4d4e7
Author: Robert Church <email address hidden>
Date: Tue Nov 29 00:34:53 2022 -0600

    Remove all volume groups by UUID

    In cases when wipedisk isn't run or isn't working correctly,
    pre-existing volume groups, physical volumes, and logical volumes will
    be present on the root disk. Depending on the sizes and layout of the
    previous install along with partial or aborted cleanup activities, this
    may lead [unknown] PVs with duplicate volume group names.

    Adjust the cleanup logic to:
    - Discover existing volume groups by UUID so that duplicate volume
      groups (i.e two occurrences of cgts-vg) can be handled individually.
    - Ignore [unknown] physical volumes in a volume group as they cannnot be
      removed. Cleaning up existing physical volumes across all volume
      groups will resolve any [unknown] physical volumes.

    In addition, unify if/then for/do syntax in the %pre-part hook

    Test Plan:
    PASS - create a scenario with multiple partitions along with a
           nova-local and cgts-vg volume group that result in an [unknown]
           physical volume and a duplicate cgts-vg. Do not wipe the disks
           and install an ISO with the above changes. Observe proper cleanup
           and install.
    PASS - Perform consecutive installs without wipedisk and observe proper
           cleanup and install

    Change-Id: Idf845cf00ca3c009d72dedef0805a77d94fa3d97
    Partial-Bug: #1998204
    Signed-off-by: Robert Church <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/metal/+/865981
Committed: https://opendev.org/starlingx/metal/commit/1796ed874017ab018c1cad34f950c6204fce8f80
Submitter: "Zuul (22348)"
Branch: master

commit 1796ed874017ab018c1cad34f950c6204fce8f80
Author: Robert Church <email address hidden>
Date: Tue Nov 29 03:31:50 2022 -0600

    Update wipedisk for LVM based rootfs

    Now that the root filesystem is based on an LVM logical volume, discover
    the root disk by searching for the boot partition.

    Changes include:
     - remove detection of rootfs_part/rootfs and adjust rootfs related
       references with boot_disk.
     - run bashate on the script and resolve indentation and syntax related
       errors. Leave long-line errors alone for improved readability.

    Test Plan:
    PASS - run 'wipedisk', answer prompts, and ensure all partitions are
           cleaned up except for the platform backup partition
    PASS - run 'wipedisk --include-backup', answer prompts, and ensure all
           partitions are cleaned up
    PASS - run 'wipedisk --include-backup --force' and ensure all partitions
           are cleaned up

    Change-Id: I036ce745353b6a26bc2615ffc6e3b8955b4dd1ec
    Closes-Bug: #1998204
    Signed-off-by: Robert Church <email address hidden>