Error: Unable to find a match: cephadm failing on ovb-1ctlr_1comp-featureset002-master

[Bhagyashri Shewale]

2020-12-24 02:09:27.387 | > Error: Unable to find a match: cephadm
2020-12-24 02:09:27.387 | returncode: 1
2020-12-24 02:09:27.387 | install-packages failed with returncode 1
2020-12-24 02:09:27.407 | ++ diskimage_builder/lib/img-functions:run_in_target:59 : check_break after-error run_in_target bash
2020-12-24 02:09:27.412 | ++ diskimage_builder/lib/common-functions:check_break:143 : echo ''
2020-12-24 02:09:27.413 | ++ diskimage_builder/lib/common-functions:check_break:143 : egrep -e '(,|^)after-error(,|$)' -q
2020-12-24 02:09:27.418 | + diskimage_builder/lib/img-functions:run_in_target:1 : trap_cleanup
2020-12-24 02:09:27.422 | + diskimage_builder/lib/img-functions:trap_cleanup:36 : exitval=1
2020-12-24 02:09:27.425 | + diskimage_builder/lib/img-functions:trap_cleanup:37 : cleanup
2020-12-24 02:09:27.429 | + diskimage_builder/lib/img-functions:cleanup:42 : unmount_image
2020-12-24 02:09:27.432 | + diskimage_builder/lib/img-functions:unmount_image:22 : sync
2020-12-24 02:09:27.569 | + diskimage_builder/lib/img-functions:unmount_image:26 : unmount_dir /home/zuul/dib_build.rsCDtsT3/mnt
2020-12-24 02:09:27.572 | + diskimage_builder/lib/common-functions:unmount_dir:414 : local dir=/home/zuul/dib_build.rsCDtsT3/mnt
2020-12-24 02:09:27.576 | + diskimage_builder/lib/common-functions:unmount_dir:415 : local real_dir
2020-12-24 02:09:27.579 | + diskimage_builder/lib/common-functions:unmount_dir:416 : local mnts
2020-12-24 02:09:27.583 | + diskimage_builder/lib/common-functions:unmount_dir:417 : local split_mounts
2020-12-24 02:09:27.587 | + diskimage_builder/lib/common-functions:unmount_dir:418 : local found_mountpoint
2020-12-24 02:09:27.590 | + diskimage_builder/lib/common-functions:unmount_dir:420 : '[' '!' -d /home/zuul/dib_build.rsCDtsT3/mnt ']'
2020-12-24 02:09:27.594 | ++ diskimage_builder/lib/common-functions:unmount_dir:427 : readlink -e /home/zuul/dib_build.rsCDtsT3/mnt
2020-12-24 02:09:27.598 | + diskimage_builder/lib/common-functions:unmount_dir:427 : real_dir=/home/zuul/dib_build.rsCDtsT3/mnt
2020-12-24 02:09:27.602 | + diskimage_builder/lib/common-functions:unmount_dir:430 : IFS='|'
2020-12-24 02:09:27.605 | + diskimage_builder/lib/common-functions:unmount_dir:430 : read -ra split_mounts
2020-12-24 02:09:27.610 | ++ diskimage_builder/lib/common-functions:unmount_dir:435 : awk '{print $2}'
2020-12-24 02:09:27.610 | ++ diskimage_builder/lib/common-functions:unmount_dir:435 : grep '^/home/zuul/dib_build.rsCDtsT3/mnt/'
2020-12-24 02:09:27.612 | ++ diskimage_builder/lib/common-functions:unmount_dir:435 : sort -r
2020-12-24 02:09:27.617 | + diskimage_builder/lib/common-functions:unmount_dir:435 : mnts='/home/zuul/dib_build.rsCDtsT3/mnt/tmp/yum
2020-12-24 02:09:27.617 | /home/zuul/dib_build.rsCDtsT3/mnt/tmp/in_target.d
2020-12-24 02:09:27.617 | /home/zuul/dib_build.rsCDtsT3/mnt/sys
2020-12-24 02:09:27.617 | /home/zuul/dib_build.rsCDtsT3/mnt/proc
2020-12-24 02:09:27.617 | /home/zuul/dib_build.rsCDtsT3/mnt/dev/pts
2020-12-24 02:09:27.617 | /home/zuul/dib_build.rsCDtsT3/mnt/dev'
2020-12-24 02:09:27.621 | + diskimage_builder/lib/common-functions:unmount_dir:436 : for m in $mnts
2020-12-24 02:09:27.624 | + diskimage_builder/lib/common-functions:unmount_dir:438 : found_mountpoint=false
2020-12-24 02:09:27.627 | + diskimage_builder/lib/common-functions:unmount_dir:439 : for mountpoint in "${split_mounts[@]}"
2020-12-24 02:09:27.631 | + diskimage_builder/lib/common-functions:unmount_dir:440 : [[ / != \/ ]]
2020-12-24 02:09:27.634 | + diskimage_builder/lib/common-functions:unmount_dir:448 : '[' false == false ']'
2020-12-24 02:09:27.637 | + diskimage_builder/lib/common-functions:unmount_dir:450 : echo 'Unmount /home/zuul/dib_build.rsCDtsT3/mnt/tmp/yum'
2020-12-24 02:09:27.637 | Unmount /home/zuul/dib_build.rsCDtsT3/mnt/tmp/yum
2020-12-24 02:09:27.641 | + diskimage_builder/lib/common-functions:unmount_dir:451 : sudo umount -fl /home/zuul/dib_build.rsCDtsT3/mnt/tmp/yum
2020-12-24 02:09:27.693 | + diskimage_builder/lib/common-functions:unmount_dir:436 : for m in $mnts
2020-12-24 02:09:27.697 | + diskimage_builder/lib/common-functions:unmount_dir:438 : found_mountpoint=false
2020-12-24 02:09:27.700 | + diskimage_builder/lib/common-functions:unmount_dir:439 : for mountpoint in "${split_mounts[@]}"
2020-12-24 02:09:27.704 | + diskimage_builder/lib/common-functions:unmount_dir:440 : [[ / != \/ ]]
2020-12-24 02:09:27.708 | + diskimage_builder/lib/common-functions:unmount_dir:448 : '[' false == false ']'
2020-12-24 02:09:27.712 | + diskimage_builder/lib/common-functions:unmount_dir:450 : echo 'Unmount /home/zuul/dib_build.rsCDtsT3/mnt/tmp/in_target.d'
2020-12-24 02:09:27.712 | Unmount /home/zuul/dib_build.rsCDtsT3/mnt/tmp/in_target.d
2020-12-24 02:09:27.716 | + diskimage_builder/lib/common-functions:unmount_dir:451 : sudo umount -fl /home/zuul/dib_build.rsCDtsT3/mnt/tmp/in_target.d
2020-12-24 02:09:27.753 | + diskimage_builder/lib/common-functions:unmount_dir:436 : for m in $mnts
2020-12-24 02:09:27.757 | + diskimage_builder/lib/common-functions:unmount_dir:438 : found_mountpoint=false
2020-12-24 02:09:27.761 | + diskimage_builder/lib/common-functions:unmount_dir:439 : for mountpoint in "${split_mounts[@]}"
2020-12-24 02:09:27.765 | + diskimage_builder/lib/common-functions:unmount_dir:440 : [[ / != \/ ]]
2020-12-24 02:09:27.769 | + diskimage_builder/lib/common-functions:unmount_dir:448 : '[' false == false ']'
2020-12-24 02:09:27.772 | + diskimage_builder/lib/common-functions:unmount_dir:450 : echo 'Unmount /home/zuul/dib_build.rsCDtsT3/mnt/sys'
2020-12-24 02:09:27.773 | Unmount /home/zuul/dib_build.rsCDtsT3/mnt/sys
2020-12-24 02:09:27.776 | + diskimage_builder/lib/common-functions:unmount_dir:451 : sudo umount -fl /home/zuul/dib_build.rsCDtsT3/mnt/sys
2020-12-24 02:09:27.813 | + diskimage_builder/lib/common-functions:unmount_dir:436 : for m in $mnts
2020-12-24 02:09:27.817 | + diskimage_builder/lib/common-functions:unmount_dir:438 : found_mountpoint=false
2020-12-24 02:09:27.821 | + diskimage_builder/lib/common-functions:unmount_dir:439 : for mountpoint in "${split_mounts[@]}"
2020-12-24 02:09:27.825 | + diskimage_builder/lib/common-functions:unmount_dir:440 : [[ / != \/ ]]
2020-12-24 02:09:27.829 | + diskimage_builder/lib/common-functions:unmount_dir:448 : '[' false == false ']'
2020-12-24 02:09:27.833 | + diskimage_builder/lib/common-functions:unmount_dir:450 : echo 'Unmount /home/zuul/dib_build.rsCDtsT3/mnt/proc'
2020-12-24 02:09:27.833 | Unmount /home/zuul/dib_build.rsCDtsT3/mnt/proc
2020-12-24 02:09:27.837 | + diskimage_builder/lib/common-functions:unmount_dir:451 : sudo umount -fl /home/zuul/dib_build.rsCDtsT3/mnt/proc
2020-12-24 02:09:27.875 | + diskimage_builder/lib/common-functions:unmount_dir:436 : for m in $mnts
2020-12-24 02:09:27.879 | + diskimage_builder/lib/common-functions:unmount_dir:438 : found_mountpoint=false
2020-12-24 02:09:27.884 | + diskimage_builder/lib/common-functions:unmount_dir:439 : for mountpoint in "${split_mounts[@]}"
2020-12-24 02:09:27.887 | + diskimage_builder/lib/common-functions:unmount_dir:440 : [[ / != \/ ]]
2020-12-24 02:09:27.891 | + diskimage_builder/lib/common-functions:unmount_dir:448 : '[' false == false ']'
2020-12-24 02:09:27.895 | + diskimage_builder/lib/common-functions:unmount_dir:450 : echo 'Unmount /home/zuul/dib_build.rsCDtsT3/mnt/dev/pts'
2020-12-24 02:09:27.895 | Unmount /home/zuul/dib_build.rsCDtsT3/mnt/dev/pts
2020-12-24 02:09:27.899 | + diskimage_builder/lib/common-functions:unmount_dir:451 : sudo umount -fl /home/zuul/dib_build.rsCDtsT3/mnt/dev/pts
2020-12-24 02:09:27.938 | + diskimage_builder/lib/common-functions:unmount_dir:436 : for m in $mnts
2020-12-24 02:09:27.942 | + diskimage_builder/lib/common-functions:unmount_dir:438 : found_mountpoint=false
2020-12-24 02:09:27.946 | + diskimage_builder/lib/common-functions:unmount_dir:439 : for mountpoint in "${split_mounts[@]}"
2020-12-24 02:09:27.950 | + diskimage_builder/lib/common-functions:unmount_dir:440 : [[ / != \/ ]]
2020-12-24 02:09:27.953 | + diskimage_builder/lib/common-functions:unmount_dir:448 : '[' false == false ']'
2020-12-24 02:09:27.957 | + diskimage_builder/lib/common-functions:unmount_dir:450 : echo 'Unmount /home/zuul/dib_build.rsCDtsT3/mnt/dev'
2020-12-24 02:09:27.957 | Unmount /home/zuul/dib_build.rsCDtsT3/mnt/dev
2020-12-24 02:09:27.961 | + diskimage_builder/lib/common-functions:unmount_dir:451 : sudo umount -fl /home/zuul/dib_build.rsCDtsT3/mnt/dev
2020-12-24 02:09:28.003 | + diskimage_builder/lib/img-functions:unmount_image:27 : '[' -n '' ']'
2020-12-24 02:09:28.007 | + diskimage_builder/lib/img-functions:unmount_image:30 : '[' -n '' ']'
2020-12-24 02:09:28.010 | + diskimage_builder/lib/img-functions:cleanup:43 : /usr/bin/python3 /usr/lib/python3.6/site-packages/diskimage_builder/lib/dib-block-device.py umount
2020-12-24 02:09:29.055 | DEBUG diskimage_builder.block_device.blockdevice [-] Creating BlockDevice object __init__ /usr/lib/python3.6/site-packages/diskimage_builder/block_device/blockdevice.py:229
2020-12-24 02:09:29.055 | DEBUG diskimage_builder.block_device.blockdevice [-] Params [{'config': '/home/zuul/dib_build.rsCDtsT3/block-device/config.yaml', 'image-dir': '/home/zuul/dib_image.NcYXR5Nz', 'root-fs-type': 'xfs', 'root-label': None, 'mount-base': '/home/zuul/dib_build.rsCDtsT3/mnt', 'build-dir': '/home/zuul/dib_build.rsCDtsT3'}] __init__ /usr/lib/python3.6/site-packages/diskimage_builder/block_device/blockdevice.py:232
2020-12-24 02:09:29.056 | INFO diskimage_builder.block_device.blockdevice [-] State already cleaned - no way to do anything here
2020-12-24 02:09:29.096 | + diskimage_builder/lib/img-functions:cleanup:44 : cleanup_build_dir
2020-12-24 02:09:29.099 | + diskimage_builder/lib/common-functions:cleanup_build_dir:216 : timeout 10 sh -c ' while ! sudo rm -rf /home/zuul/dib_build.rsCDtsT3/built; do sleep 1; done'
2020-12-24 02:09:29.144 | + diskimage_builder/lib/common-functions:cleanup_build_dir:220 : sudo rm -rf /home/zuul/dib_build.rsCDtsT3/mnt
2020-12-24 02:09:30.950 | + diskimage_builder/lib/common-functions:cleanup_build_dir:221 : kill_chroot_processes /home/zuul/dib_build.rsCDtsT3
2020-12-24 02:09:31.054 | + diskimage_builder/lib/common-functions:cleanup_build_dir:222 : tmpfs_check 0
2020-12-24 02:09:31.058 | + diskimage_builder/lib/common-functions:tmpfs_check:23 : local echo_message=0
2020-12-24 02:09:31.062 | + diskimage_builder/lib/common-functions:tmpfs_check:24 : '[' 0 == 0 ']'
2020-12-24 02:09:31.067 | + diskimage_builder/lib/common-functions:tmpfs_check:25 : '[' -r /proc/meminfo ']'
2020-12-24 02:09:31.072 | ++ diskimage_builder/lib/common-functions:tmpfs_check:26 : awk '/^MemTotal/ { print $2 }' /proc/meminfo
2020-12-24 02:09:31.079 | + diskimage_builder/lib/common-functions:tmpfs_check:26 : total_kB=8150348
2020-12-24 02:09:31.082 | + diskimage_builder/lib/common-functions:tmpfs_check:29 : RAM_NEEDED=14
2020-12-24 02:09:31.086 | + diskimage_builder/lib/common-functions:tmpfs_check:30 : '[' 8150348 -lt 14680064 ']'
2020-12-24 02:09:31.089 | + diskimage_builder/lib/common-functions:tmpfs_check:31 : '[' 0 == 1 ']'
2020-12-24 02:09:31.093 | + diskimage_builder/lib/common-functions:tmpfs_check:34 : return 1
2020-12-24 02:09:31.096 | + diskimage_builder/lib/common-functions:cleanup_build_dir:230 : rm -rf --one-file-system /home/zuul/dib_build.rsCDtsT3
2020-12-24 02:09:31.106 | + diskimage_builder/lib/img-functions:cleanup:45 : cleanup_image_dir
2020-12-24 02:09:31.109 | + diskimage_builder/lib/common-functions:cleanup_image_dir:234 : kill_chroot_processes /home/zuul/dib_image.NcYXR5Nz
2020-12-24 02:09:31.174 | + diskimage_builder/lib/common-functions:cleanup_image_dir:235 : tmpfs_check 0
2020-12-24 02:09:31.178 | + diskimage_builder/lib/common-functions:tmpfs_check:23 : local echo_message=0
2020-12-24 02:09:31.181 | + diskimage_builder/lib/common-functions:tmpfs_check:24 : '[' 0 == 0 ']'
2020-12-24 02:09:31.184 | + diskimage_builder/lib/common-functions:tmpfs_check:25 : '[' -r /proc/meminfo ']'
2020-12-24 02:09:31.188 | ++ diskimage_builder/lib/common-functions:tmpfs_check:26 : awk '/^MemTotal/ { print $2 }' /proc/meminfo
2020-12-24 02:09:31.194 | + diskimage_builder/lib/common-functions:tmpfs_check:26 : total_kB=8150348
2020-12-24 02:09:31.197 | + diskimage_builder/lib/common-functions:tmpfs_check:29 : RAM_NEEDED=14
2020-12-24 02:09:31.201 | + diskimage_builder/lib/common-functions:tmpfs_check:30 : '[' 8150348 -lt 14680064 ']'
2020-12-24 02:09:31.204 | + diskimage_builder/lib/common-functions:tmpfs_check:31 : '[' 0 == 1 ']'
2020-12-24 02:09:31.207 | + diskimage_builder/lib/common-functions:tmpfs_check:34 : return 1
2020-12-24 02:09:31.210 | + diskimage_builder/lib/common-functions:cleanup_image_dir:241 : rm -rf --one-file-system /home/zuul/dib_image.NcYXR5Nz
2020-12-24 02:09:31.215 | + diskimage_builder/lib/img-functions:trap_cleanup:38 : exit 1

[1]: https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-main&job_name=periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-master

[2]: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-master/4733550/logs/undercloud/home/zuul/build.log.gz

[3]: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-master/4733550/logs/undercloud/home/zuul/repo_setup.log.gz

[Sandeep Yadav (sandeepyadav93)]

Hello,

* This issue started on 17th Dec

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-master&project=openstack/tripleo-ci

* In the last good run, the job was not trying to install cephadm

~~~
https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-master/94596cb/logs/undercloud/home/zuul/build.log.gz
~~~

Following patch which adds cephadm in the overcloud image looks related to the issue:-

https://review.opendev.org/c/openstack/tripleo-puppet-elements/+/765512

[Sandeep Yadav (sandeepyadav93)]

Hello,

periodic-tripleo-centos-8-buildimage-overcloud-full-master worked fine because it had yum.repos.d/tripleo-centos-ceph-octopus.repo which provides cephadm package.

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-overcloud-full-master/3945594/build.log

~~~
2020-12-28 00:44:10.278 | > cephadm x86_64 2:15.2.5-1.el8 tripleo-centos-ceph-octopus 73 k
~~~

[Sandeep Yadav (sandeepyadav93)]

Patch is up: https://review.opendev.org/c/openstack/tripleo-quickstart/+/768599

[Sandeep Yadav (sandeepyadav93)]

Ceph Octopus repository doesn't have ceph-ansible package, If we completely replace nautilius repos with Octopus repos.

Run of install_packages.sh(trying to install during ceph-ansible[1]) fails [2].

We understand in the long term ceph-ansible will be replaced by cephadm, but looking at patch [3] which is still a WIP, Looks like we couldn't switch ceph-ansible to cephadm completely yet.

We need more inputs from Ceph Team on this but till then to unblock us we plan to keep ceph nautilius repos as it is and just install cephadm from octopus repo (Looks like atleast package installation will work as we don't notice any hard dependencies on cephadm and ceph binaries[4])

[1]

~~~
sudo $(command -v dnf || command -v yum) install -y python3-tripleoclient
~~~

[2]
https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_fe5/768599/2/check/tripleo-ci-centos-8-content-provider/fe52e6f/logs/undercloud/home/zuul/install_packages.sh.log

~~~
2020-12-28 12:03:02 | Last metadata expiration check: 0:00:37 ago on Mon 28 Dec 2020 12:02:25 PM UTC.
2020-12-28 12:03:02 | No match for argument: ceph-ansible
2020-12-28 12:03:02 | Error: Unable to find a match: ceph-ansible
~~~

[3] https://review.opendev.org/c/openstack/tripleo-heat-templates/+/767294

[4]

~~~
$ rpm -qp --requires https://cbs.centos.org/kojifiles/packages/ceph/15.2.5/1.el8/x86_64/cephadm-15.2.5-1.el8.x86_64.rpm
/bin/sh
/bin/sh
/usr/libexec/platform-python
lvm2
python3
rpmlib(CompressedFileNames) <= 3.0.4-1
rpmlib(FileDigests) <= 4.6.0-1
rpmlib(PayloadFilesHavePrefix) <= 4.0-1
rpmlib(PayloadIsXz) <= 5.2-1
~~~

[Sandeep Yadav (sandeepyadav93)]

Merge of https://review.opendev.org/c/openstack/tripleo-quickstart/+/768599 fixes the issue. In future when ceph-ansible to cephadm integration completes we will switch to octopus repos completely for master branch.