/tmp/tmp.OzEfVL1XcE/boot/efi: mount point does not exist

[Ananya Banerjee]

TASK [modify-image : Mount image] fails in ovb master jobs with mount /dev/nbd0p1 /tmp/tmp.OzEfVL1XcE/boot/efi\nmount: /tmp/tmp.OzEfVL1XcE/boot/efi: mount point does not exist."

Logs:
https://logserver.rdoproject.org/54/31954/54/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master/21715de/job-output.txt
https://logserver.rdoproject.org/97/44397/2/check/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-master/bf3f8c4/job-output.txt

We see this starting from 2022-08-02 17:22:37
It might be related to https://review.opendev.org/c/openstack/tripleo-image-elements/+/846807

[Harald Jensås (harald-jensas)]

Cleaned up the output from TASK [modify-image : Debug image mount] so that is readable:

2022-08-03 05:40:47.847574 | primary | "stderr": "+ type tripleo-mount-image
+ tripleo-mount-image -a /home/zuul/overcloud-hardened-uefi-full.raw -m /tmp/tmp.M5OCp8JR8x
+ grep '\"format\": \"raw\"'
+ qemu-img info --output json /home/zuul/overcloud-hardened-uefi-full.raw
+ image_format='--format raw'
+ qemu-nbd --format raw --connect /dev/nbd0 /home/zuul/overcloud-hardened-uefi-full.raw
+ vgscan
+ vgchange --refresh
+ vgchange -ay
+ root_device=
+ boot_device=
+ efi_device=
+ timeout 5 sh -c 'while ! ls /dev/nbd0p* ; do sleep 1; done'
++ ls -1 /dev/nbd0p1 /dev/nbd0p2 /dev/nbd0p3 /dev/nbd0p4
+ devices='/dev/nbd0p1
/dev/nbd0p2
/dev/nbd0p3
/dev/nbd0p4'
++ echo /dev/nbd0p1 /dev/nbd0p2 /dev/nbd0p3 /dev/nbd0p4
++ wc -w
+ device_count=4
+ '[' 4 == 0 ']'
+ '[' 4 == 1 ']'
+ for device in ${devices}
+ lsblk --nodeps -P --output-all /dev/nbd0p1
++ lsblk --nodeps --noheadings --output LABEL /dev/nbd0p1
+ label=MKFS_ESP
++ lsblk --nodeps --noheadings --output PARTTYPENAME /dev/nbd0p1
+ part_type_name='EFI System'
++ lsblk --nodeps --noheadings --output PARTLABEL /dev/nbd0p1
+ part_label=ESP
+ [[ EFI System == \\B\\I\\O\\S\\ \\b\\o\\o\\t ]]
+ [[ EFI System == \\P\\o\\w\\e\\r\\P\\C\\ \\P\\R\\e\\P\\ \\b\\o\\o\\t ]]
+ '[' -z '' ']'
+ [[ EFI System == \\E\\F\\I\\ \\S\\y\\s\\t\\e\\m ]]
+ efi_device=/dev/nbd0p1
+ continue
+ for device in ${devices}
+ lsblk --nodeps -P --output-all /dev/nbd0p2
++ lsblk --nodeps --noheadings --output LABEL /dev/nbd0p2
+ label=
++ lsblk --nodeps --noheadings --output PARTTYPENAME /dev/nbd0p2
+ part_type_name='BIOS boot'
++ lsblk --nodeps --noheadings --output PARTLABEL /dev/nbd0p2
+ part_label=BSP
+ [[ BIOS boot == \\B\\I\\O\\S\\ \\b\\o\\o\\t ]]
+ continue
+ for device in ${devices}
+ lsblk --nodeps -P --output-all /dev/nbd0p3
++ lsblk --nodeps --noheadings --output LABEL /dev/nbd0p3
+ label=mkfs_boot
++ lsblk --nodeps --noheadings --output PARTTYPENAME /dev/nbd0p3
+ part_type_name='Linux extended boot'
++ lsblk --nodeps --noheadings --output PARTLABEL /dev/nbd0p3
+ part_label=boot
+ [[ Linux extended boot == \\B\\I\\O\\S\\ \\b\\o\\o\\t ]]
+ [[ Linux extended boot == \\P\\o\\w\\e\\r\\P\\C\\ \\P\\R\\e\\P\\ \\b\\o\\o\\t ]]
+ '[' -z /dev/nbd0p1 ']'
+ '[' -z '' ']'
+ [[ mkfs_boot == \\b\\o\\o\\t ]]
+ '[' -z '' ']'
+ root_device=/dev/nbd0p3
+ continue
+ for device in ${devices}
+ lsblk --nodeps -P --output-all /dev/nbd0p4
++ lsblk --nodeps --noheadings --output LABEL /dev/nbd0p4
+ label=
++ lsblk --nodeps --noheadings --output PARTTYPENAME /dev/nbd0p4
+ part_type_name='Linux filesystem'
++ lsblk --nodeps --noheadings --output PARTLABEL /dev/nbd0p4
+ part_label=root
+ [[ Linux filesystem == \\B\\I\\O\\S\\ \\b\\o\\o\\t ]]
+ [[ Linux filesystem == \\P\\o\\w\\e\\r\\P\\C\\ \\P\\R\\e\\P\\ \\b\\o\\o\\t ]]
+ '[' -z /dev/nbd0p1 ']'
+ '[' -z '' ']'
+ [[ '' == \\b\\o\\o\\t ]]
+ '[' -z /dev/nbd0p3 ']'
+ '[' -z /dev/nbd0p3 ']'
+ '[' -b /dev/mapper/vg-lv_root ']'
+ mount /dev/mapper/vg-lv_root /tmp/tmp.M5OCp8JR8x
+ for m in $MOUNTS
+ device=/dev/mapper/vg-lv_var
+ path=/var
+ mount_volume /dev/mapper/vg-lv_var /tmp/tmp.M5OCp8JR8x/var
+ '[' -b /dev/mapper/...

[Harald Jensås (harald-jensas)]

It looks like the boot partition may not have been detected.
Afict /dev/nbd0p3 is the boot partition, but the label is 'mkfs_boot' and the partition label is 'boot'. The tripleo-mount-image code looks for `${label} == "boot"`, since the label in `mkfs_boot` it is not detected, and the boot device is never mounted. It needs to mount the boot partition before mounting the EFI partition, if not the /boot/efi: mount point does not exist.

+ for device in ${devices}
+ lsblk --nodeps -P --output-all /dev/nbd0p3
++ lsblk --nodeps --noheadings --output LABEL /dev/nbd0p3
+ label=mkfs_boot
++ lsblk --nodeps --noheadings --output PARTTYPENAME /dev/nbd0p3
+ part_type_name='Linux extended boot'
++ lsblk --nodeps --noheadings --output PARTLABEL /dev/nbd0p3
+ part_label=boot

[1] https://opendev.org/openstack/tripleo-common/src/branch/master/scripts/tripleo-mount-image#L150-L157
[2] https://opendev.org/openstack/tripleo-common/src/branch/master/scripts/tripleo-mount-image#L183-L190

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-common/+/851984

[Harald Jensås (harald-jensas)]

The label "mkfs_boot" is weird.

For reference, the build log for the image use by the failing job is:
https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-9-buildimage-overcloud-hardened-uefi-full-master/c38061a/build.log

I'll try to go trough that for clues.

[Harald Jensås (harald-jensas)]

diskimage-builder is actively setting the label 'mkfs_boot'.

2022-08-03 01:05:30.343 | DEBUG diskimage_builder.block_device.level2.mkfs [-] Creating fs command [['mkfs', '-t', 'ext4', '-L', 'mkfs_boot', '-U', 'c91f24e4-9364-4797-bfa7-534b09471084', '-q', '/dev/mapper/loop0p3']] create /home/zuul/workspace/venv/lib64/python3.9/site-packages/diskimage_builder/block_device/level2/mkfs.py:132
2022-08-03 01:05:30.343 | INFO diskimage_builder.block_device.utils [-] Calling [sudo mkfs -t ext4 -L mkfs_boot -U c91f24e4-9364-4797-bfa7-534b09471084 -q /dev/mapper/loop0p3]

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-9-buildimage-overcloud-hardened-uefi-full-master/c38061a/build.log

[Steve Baker (steve-stevebaker)]

I'd prefer this 4 month old change[1] which checks for Bootloader Spec standard GUIDs for /boot and /boot/efi, then it doesn't matter what the label is, as long as it is a spec compliant image.

There is another change before that in the series which is 6 months old[2], so it would be good if that landed too :)

[1] https://review.opendev.org/c/openstack/tripleo-common/+/835774
[2] https://review.opendev.org/c/openstack/tripleo-common/+/828412

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Harald Jensås <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-common/+/851984
Reason: https://review.opendev.org/c/openstack/tripleo-common/+/835774 is a better fix.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-common/+/835774
Committed: https://opendev.org/openstack/tripleo-common/commit/e6b0a7bf34e5b15f32de9ed51bf914f990ee585a
Submitter: "Zuul (22348)"
Branch: master

commit e6b0a7bf34e5b15f32de9ed51bf914f990ee585a
Author: Steve Baker <email address hidden>
Date: Tue Mar 1 14:12:02 2022 +1300

    Support standard partition type GUIDs

    This change adds GUID detection for mounting /boot and /boot/efi.

    Closes-Bug: #1983467
    Change-Id: I0c1b6670fa57e61748f9d890db1173ad442f6680

[Marios Andreou (marios-b)]

periodic fs20 master finally has some green runs at [1] - from the periodic run today in [2]

I think we can move this to fix-released.

[1] https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-master
[2] https://review.rdoproject.org/zuul/build/b3be182de41e4cfe831770a0fd5acfc9

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-common/+/855838

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-common/+/855838
Committed: https://opendev.org/openstack/tripleo-common/commit/0f80917934a84efa58b9913d72ebcbb350cdb41b
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 0f80917934a84efa58b9913d72ebcbb350cdb41b
Author: Steve Baker <email address hidden>
Date: Tue Mar 1 14:12:02 2022 +1300

    Support standard partition type GUIDs

    This change adds GUID detection for mounting /boot and /boot/efi.

    Closes-Bug: #1983467
    Change-Id: I0c1b6670fa57e61748f9d890db1173ad442f6680
    (cherry picked from commit e6b0a7bf34e5b15f32de9ed51bf914f990ee585a)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common 17.0.0 release.