tripleo-buildimage-overcloud-hardened-full-centos-8 fails with dracut module 'network-legacy' cannot be found or installed

[Rabi Mishra]

Noticed at:
https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_0b5/763384/10/check/tripleo-buildimage-overcloud-hardened-full-centos-8/0b5af36/build.log

traceback:

2020-12-09 09:45:22.275 | ++ basename //boot/vmlinuz-4.18.0-240.1.1.el8_3.x86_64
2020-12-09 09:45:22.276 | + KERNEL=vmlinuz-4.18.0-240.1.1.el8_3.x86_64
2020-12-09 09:45:22.276 | + KERNEL_VERSION=4.18.0-240.1.1.el8_3.x86_64
2020-12-09 09:45:22.276 | +++ ls //boot/initramfs-4.18.0-240.1.1.el8_3.x86_64.img
2020-12-09 09:45:22.278 | ++ basename //boot/initramfs-4.18.0-240.1.1.el8_3.x86_64.img
2020-12-09 09:45:22.279 | + RAMDISK=initramfs-4.18.0-240.1.1.el8_3.x86_64.img
2020-12-09 09:45:22.279 | + '[' '!' initramfs-4.18.0-240.1.1.el8_3.x86_64.img ']'
2020-12-09 09:45:22.280 | + echo vmlinuz-4.18.0-240.1.1.el8_3.x86_64:initramfs-4.18.0-240.1.1.el8_3.x86_64.img
2020-12-09 09:45:22.281 | DEBUG:root:Calling: ['dracut', '--force', '--add', 'crypt lvm', '-f', '/boot/initramfs-4.18.0-240.1.1.el8_3.x86_64.img', '4.18.0-240.1.1.el8_3.x86_64']
2020-12-09 09:45:22.313 | dracut: No '/dev/log' or 'logger' included for syslog logging
2020-12-09 09:45:22.963 | dracut: dracut module 'network-legacy' cannot be found or installed.
2020-12-09 09:45:22.972 | Traceback (most recent call last):
2020-12-09 09:45:22.972 | File "/tmp/in_target.d/finalise.d/50-dracut-regenerate", line 91, in <module>
2020-12-09 09:45:22.972 | main()
2020-12-09 09:45:22.972 | File "/tmp/in_target.d/finalise.d/50-dracut-regenerate", line 85, in main
2020-12-09 09:45:22.972 | raise e
2020-12-09 09:45:22.972 | subprocess.CalledProcessError: Command '['dracut', '--force', '--add', 'crypt lvm', '-f', '/boot/initramfs-4.18.0-240.1.1.el8_3.x86_64.img', '4.18.0-240.1.1.el8_3.x86_64']' returned non-zero exit status 1.
2020-12-09 09:45:22.989 | ++ diskimage_builder/lib/img-functions:run_in_target:59 : check_break after-error run_in_target bash
2020-12-09 09:45:22.993 | ++ diskimage_builder/lib/common-functions:check_break:143 : egrep -e '(,|^)after-error(,|$)' -q
2020-12-09 09:45:22.993 | ++ diskimage_builder/lib/common-functions:check_break:143 : echo ''
2020-12-09 09:45:23.001 | + diskimage_builder/lib/img-functions:run_in_target:1 : trap_cleanup
2020-12-09 09:45:23.005 | + diskimage_builder/lib/img-functions:trap_cleanup:36 : exitval=1
2020-12-09 09:45:23.008 | + diskimage_builder/lib/img-functions:trap_cleanup:37 : cleanup
2020-12-09 09:45:23.011 | + diskimage_builder/lib/img-functions:cleanup:42 : unmount_image

[wes hayutin (weshayutin)]

https://zuul.openstack.org/builds?job_name=tripleo-buildimage-overcloud-hardened-full-centos-8

[wes hayutin (weshayutin)]

https://review.opendev.org/c/openstack/tripleo-heat-templates/+/708543 thanks chandan

[chandan kumar (chkumar246)]

2020-12-09 09:35:52.672 | dracut-config-generic x86_64049-95.git20200804.el8 BaseOS 54 k
2020-12-09 09:35:52.672 | dracut-config-rescue x86_64049-95.git20200804.el8 BaseOS 56 k
2020-12-09 09:35:52.672 | dracut-network x86_64049-95.git20200804.el8 BaseOS 102 k
2020-12-09 09:35:52.672 | dracut-squash x86_64049-95.git20200804.el8 BaseOS 56 k

It is getting installed there

[Harald Jensås (harald-jensas)]

https://ebc2a3061166f5a55254-8af958d7dd3c07fe325f83683b2d5c0b.ssl.cf5.rackcdn.com/766232/2/check/tripleo-buildimage-overcloud-full-centos-8/c117efc/overcloud-full.log

2020-12-09 15:25:33.660 | + echo vmlinuz-4.18.0-240.1.1.el8_3.x86_64:initramfs-4.18.0-240.1.1.el8_3.x86_64.img
2020-12-09 15:25:33.661 | DEBUG:root:Calling: ['dracut', '--force']
2020-12-09 15:25:33.694 | dracut: No '/dev/log' or 'logger' included for syslog logging
2020-12-09 15:25:33.728 | dracut: Cannot find module directory /lib/modules/4.18.0-193.28.1.el8_2.x86_64/
2020-12-09 15:25:33.728 | dracut: and --no-kernel was not specified
2020-12-09 15:25:33.730 | Traceback (most recent call last):
2020-12-09 15:25:33.730 | File "/tmp/in_target.d/finalise.d/50-dracut-regenerate", line 102, in <module>
2020-12-09 15:25:33.730 | main()
2020-12-09 15:25:33.730 | File "/tmp/in_target.d/finalise.d/50-dracut-regenerate", line 84, in main
2020-12-09 15:25:33.730 | raise e

" Cannot find module directory /lib/modules/4.18.0-193.28.1.el8_2.x86_64/ "
I've seen something similar before, if the OS building the image is running an older kernel I think. Booted OS is running BOOT_IMAGE: /boot/vmlinuz-4.18.0-193.28.1.el8_2.x86_64 kernel, but el8_3 is in the built image. So it tries to lookup lib/modules/4.18.0-193.28.1.el8_2.x86_64/, but it should lookup lib/modules/4.18.0-240.1.1.el8_3.x86_64/. I keep a "reboot" step in my playbooks to get around it.

[wes hayutin (weshayutin)]

https://review.opendev.org/c/openstack/tripleo-ci/+/766297

[wes hayutin (weshayutin)]

problem is here
https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/elements/dracut-regenerate/finalise.d/50-dracut-regenerate#L74-L85

The upstream nodes should be at CentOS 8.3, they are for other centos jobs. Not enough logging here to tell.

[yatin (yatinkarel)]

with 8.3 image job pass https://review.rdoproject.org/r/#/c/31423/, so need to update the image used on images.rdoproject.org.

[Sandeep Yadav (sandeepyadav93)]

Hi,

Ops team have updated image to 8.3 on images.rdoproject.org.

~~~
https://images.rdoproject.org/CentOS-8-x86_64-GenericCloud.qcow2.md5
~~~

Failing job(periodic-tripleo-centos-8-buildimage-overcloud-hardened-full-master) passed in testproject

https://review.rdoproject.org/r/#/c/28442/

~~~
periodic-tripleo-centos-8-buildimage-overcloud-hardened-full-master https://review.rdoproject.org/zuul/build/75f96f4012134055a7f91978d889d536 : SUCCESS in 21m 34s
~~~

[Sandeep Yadav (sandeepyadav93)]

Hello,

We have also proposed patch[1] for jobs to use the new image.

[1] Remove md5sum hardcoding from environment file.
https://review.opendev.org/c/openstack/tripleo-quickstart/+/768717

[Lon Hohberger (lhh)]

This caused an issue with dracut-regenerate if the kernel utilized in the image build didn't match the host kernel. The following change fixes it:

https://review.opendev.org/c/openstack/diskimage-builder/+/770241