tripleo

tripleo-quicktart is not able to provision VMs

Bug #1948001 reported by Juan Badia Payno
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Unassigned
tripleo yoga-1

Bug Description

Deploy tripleo with tripleo-quickstart is failing on the conver_image.sh.

How to reproduce
----------------
Download tripleo-quickstart
~]# cd tripleo-quickstart
~]# ./quickstart.sh -n -X -R master-tripleo-ci --tags all -T none --extra-vars undercloud_disk=70 --nodes config/nodes/3ctlr_2comp_3ceph.yml -p quickstart.yml 127.0.0.2

The ansible error:

TASK [convert-image : convert image] *********************************************************************************************************************************************************************************************************
task path: /root/tripleo-quickstart/roles/convert-image/tasks/main.yml:37
Thursday 21 October 2021 11:16:12 +0200 (0:00:23.537) 0:25:28.547 ******
fatal: [127.0.0.2]: FAILED! => {
    "changed": true,
    "cmd": [
        "virt-customize",
        "-v",
        "--smp",
        "2",
        "-m",
        "2048",
        "-a",
        "/home/stack/undercloud.qcow2",
        "--run",
        "convert_image.sh"
    ],
    "delta": "0:01:31.165601",
    "end": "2021-10-21 11:17:43.975713",
    "rc": 1,
    "start": "2021-10-21 11:16:12.810112"
}

STDOUT:

[ 0.0] Examining the guest ...
[ 5.7] Setting a random seed
[ 5.7] Running: convert_image.sh

STDERR:

[Due to some erros I will add the log on the comments]

Revision history for this message
Juan Badia Payno (jbadiapa) wrote :
Download full text (265.1 KiB)

TASK [convert-image : convert image] *********************************************************************************************************************************************************************************************************
task path: /root/tripleo-quickstart/roles/convert-image/tasks/main.yml:37
Thursday 21 October 2021 11:16:12 +0200 (0:00:23.537) 0:25:28.547 ******
fatal: [127.0.0.2]: FAILED! => {
    "changed": true,
    "cmd": [
        "virt-customize",
        "-v",
        "--smp",
        "2",
        "-m",
        "2048",
        "-a",
        "/home/stack/undercloud.qcow2",
        "--run",
        "convert_image.sh"
    ],
    "delta": "0:01:31.165601",
    "end": "2021-10-21 11:17:43.975713",
    "rc": 1,
    "start": "2021-10-21 11:16:12.810112"
}

STDOUT:

[ 0.0] Examining the guest ...
[ 5.7] Setting a random seed
[ 5.7] Running: convert_image.sh

STDERR:

libguestfs: launch: program=virt-customize
libguestfs: launch: version=1.40.2rhel=8,release=27.module_el8.4.0+783+f8734d30,libvirt
libguestfs: launch: backend registered: unix
libguestfs: launch: backend registered: uml
libguestfs: launch: backend registered: libvirt
libguestfs: launch: backend registered: direct
libguestfs: launch: backend=direct
libguestfs: launch: tmpdir=/tmp/libguestfsPbWHaT
libguestfs: launch: umask=0002
libguestfs: launch: euid=1005
libguestfs: begin testing qemu features
libguestfs: checking for previously cached test results of /usr/libexec/qemu-kvm, in /var/tmp/.guestfs-1005
libguestfs: loading previously cached test results
libguestfs: qemu version: 4.2
libguestfs: qemu mandatory locking: yes
libguestfs: qemu KVM: enabled
libguestfs: finished testing qemu features
/usr/libexec/qemu-kvm \
    -global virtio-blk-pci.scsi=off \
    -no-user-config \
    -enable-fips \
    -nodefaults \
    -display none \
    -machine accel=kvm:tcg \
    -cpu host \
    -smp 2 \
    -m 2048 \
    -no-reboot \
    -rtc driftfix=slew \
    -no-hpet \
    -global kvm-pit.lost_tick_policy=discard \
    -kernel /usr/local/src/appliance/kernel \
    -initrd /usr/local/src/appliance/initrd \
    -object rng-random,filename=/dev/urandom,id=rng0 \
    -device virtio-rng-pci,rng=rng0 \
    -device virtio-scsi-pci,id=scsi \
    -drive file=/home/stack/undercloud.qcow2,cache=writeback,discard=unmap,id=hd0,if=none \
    -device scsi-hd,drive=hd0 \
    -drive file=/usr/local/src/appliance/root,snapshot=on,id=appliance,cache=unsafe,if=none,format=raw \
    -device scsi-hd,drive=appliance \
    -device virtio-serial-pci \
    -serial stdio \
    -device sga \
    -chardev socket,path=/run/user/1005/libguestfsHGmTvD/guestfsd.sock,id=channel0 \
    -device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 \
    -netdev user,id=usernet,net=169.254.0.0/16 \
    -device virtio-net-pci,netdev=usernet \
    -append "panic=1 console=ttyS0 edd=off udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1 cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable 8250.nr_uarts=1 root=/dev/sdb selinux=0 guestfs_verbose=1 guestfs_network=1 TERM=linux"
libguestfs: responding to serial console Device Status Report
\x1b[1;256r\x1b[256;25...

Revision history for this message
Juan Badia Payno (jbadiapa) wrote :
Download full text (4.3 KiB)

A piece of the log regarding the convert_image.sh

Complete!
+ '[' '!' -f /etc/sysconfig/network-scripts/ifcfg-eth0 ']'
+ cat
+ useradd stack
+ echo 'stack ALL=(root) NOPASSWD:ALL'
+ chmod 0440 /etc/sudoers.d/stack
+ mkdir /home/stack/.ssh
+ echo '127.0.0.1 undercloud localhost'
+ echo HOSTNAME=undercloud
+ echo undercloud
+ chown -R stack:stack /home/stack/
+ dd if=/dev/zero of=/swapfile bs=1024 count=4194304
dd: error writing '/swapfile': No space left on device
1153861+0 records in
1153860+0 records out
1181552640 bytes (1.2 GB, 1.1 GiB) copied, 6.44038 s, 183 MB/s
libguestfs: trace: download = 0
virt-customize: error: convert_image.sh: command exited with an error

This seems to be the problem. However deleting all the swapfile thing only postpones the error a little bit.

As the following error showed up:

TASK [setup/undercloud : Inject additional images] *******************************************************************************************************************************************************************************************
task path: /root/tripleo-quickstart/roles/libvirt/setup/undercloud/tasks/main.yml:88
Tuesday 26 October 2021 17:44:59 +0200 (0:00:00.025) 0:10:18.171 *******
failed: [127.0.0.2] (item=ironic-python-agent.initramfs) => {
    "ansible_loop_var": "item",
    "changed": true,
    "cmd": [
        "virt-customize",
        "-a",
        "/home/stack/undercloud.qcow2",
        "--upload",
        "/home/stack/ironic-python-agent.initramfs:/home/stack/ironic-python-agent.initramfs",
        "--run-command",
        "chown stack:stack /home/stack/ironic-python-agent.initramfs"
    ],
    "delta": "0:00:06.831224",
    "end": "2021-10-26 17:45:06.797714",
    "item": "ironic-python-agent.initramfs",
    "rc": 1,
    "start": "2021-10-26 17:44:59.966490"
}

STDOUT:

[ 0.0] Examining the guest ...
[ 4.2] Setting a random seed
[ 4.2] Uploading: /home/stack/ironic-python-agent.initramfs to /home/stack/ironic-python-agent.initramfs

STDERR:

virt-customize: error: libguestfs error: upload: write error:
/home/stack/ironic-python-agent.initramfs

If reporting bugs, run virt-customize with debugging enabled and include
the complete output:

  virt-customize -v -x [...]

MSG:

non-zero return code
failed: [127.0.0.2] (item=ironic-python-agent.kernel) => {
    "ansible_loop_var": "item",
    "changed": true,
    "cmd": [
        "virt-customize",
        "-a",
        "/home/stack/undercloud.qcow2",
        "--upload",
        "/home/stack/ironic-python-agent.kernel:/home/stack/ironic-python-agent.kernel",
        "--run-command",
        "chown stack:stack /home/stack/ironic-python-agent.kernel"
    ],
    "delta": "0:00:04.385507",
    "end": "2021-10-26 17:45:11.502365",
    "item": "ironic-python-agent.kernel",
    "rc": 1,
    "start": "2021-10-26 17:45:07.116858"
}

STDOUT:

[ 0.0] Examining the guest ...
[ 4.2] Setting a random seed
[ 4.2] Uploading: /home/stack/ironic-python-agent.kernel to /home/stack/ironic-python-agent.kernel

STDERR:

virt-customize: error: libguestfs error: upload:
/home/stack/ironic-python-agent.kernel: No space left on device

If reporting bugs, run virt-custom...

Read more...

Revision history for this message
Juan Badia Payno (jbadiapa) wrote :

Not sure if this[1] have anything to do with the change.

[1] - https://review.opendev.org/c/openstack/tripleo-common/+/798041

Ronelle Landy (rlandy)
Changed in tripleo:
milestone: none → yoga-1
importance: Undecided → Critical
status: New → Triaged
Revision history for this message
Steve Baker (steve-stevebaker) wrote :

What is the value of overcloud_as_undercloud in this environment?

Can a link be provided to logs of CI job failing with this issue?

I don't think it is related to this issue, but having overcloud_as_undercloud:true with the whole-disk image when the *overcloud* image is modified by quickstart might also be a problem because the LVM volumes will have the same name in the overcloud image and the host (undercloud based on overcloud).

This bug may be another reason why overcloud_as_undercloud:true won't work with overcloud-hardened-uefi-full.qcow2 as the source image. The solution may be to *also* download overcloud-full.tar when
overcloud_as_undercloud:true so that overcloud-full.qcow2 can again be used as the source image of undercloud.qcow2.

Or maybe this is the time to revisit the value that overcloud_as_undercloud provides?

Revision history for this message
Juan Badia Payno (jbadiapa) wrote :

Steve,

The value of the overcloud_as_undercloud=true as can be seen at [1]

The proposal executing the overcloud_as_undercloud=false does work for that specific part, but it failed with a different error at(there is no undercloud.qcow2 file):

TASK [setup/undercloud : Copy instackenv.json to appliance] *********************************************************************************
task path: /root/tripleo-quickstart/roles/libvirt/setup/undercloud/tasks/main.yml:101
Tuesday 02 November 2021 04:24:09 -0400 (0:00:00.024) 0:09:55.740 ******
fatal: [127.0.0.2]: FAILED! => {
    "changed": true,
    "cmd": [
        "virt-customize",
        "-a",
        "/home/stack/undercloud.qcow2",
        "--upload",
        "/home/stack/instackenv.json:/home/stack/instackenv.json",
        "--run-command",
        "chown stack:stack /home/stack/instackenv.json"
    ],
    "delta": "0:00:00.014920",
    "end": "2021-11-02 04:24:09.788398",
    "rc": 1,
    "start": "2021-11-02 04:24:09.773478"
}

STDOUT:

[ 0.0] Examining the guest ...

STDERR:

virt-customize: error: libguestfs error: /home/stack/undercloud.qcow2: No
such file or directory

thx,

[1].- https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/CentOS-8/master.yml#L3

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-quickstart (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/818010

Changed in tripleo:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-quickstart (master)

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart/+/818010
Committed: https://opendev.org/openstack/tripleo-quickstart/commit/5b7b4f205dfb84513153b990aaff7900a7517439
Submitter: "Zuul (22348)"
Branch: master

commit 5b7b4f205dfb84513153b990aaff7900a7517439
Author: Steve Baker <email address hidden>
Date: Tue Nov 16 09:07:38 2021 +1300

    Do not use whole-disk images on libvirt environments

    Some quickstart development will be required for libvirt environments
    around fetching/uploading the correct image and supporting
    overcloud_as_undercloud when the default overcloud image is
    overcloud-hardened-uefi-full.qcow2.

    Now that the default on master is whole_disk_images:true, this change
    sets it to false for libvirt environments so they continue to work
    with overcloud-full for now.

    Change-Id: I1affd48fc450250360d4eae8b30d18330a49739d
    Blueprint: whole-disk-default
    Closes-Bug: #1950383
    Closes-Bug: #1948001

Changed in tripleo:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.