StarlingX

virsh only listing one volume, even though there was an additional volume attached after instantiation

Bug #1834194 reported by Wendy Mitchell
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Fix Released
Medium
Austin Sun

Bug Description

Brief Description
-----------------
only one volume listed in virsh even though 2 attached

Severity
--------
standard

Steps to Reproduce
------------------
1. Create a flavor with eg. 1 vcpu, 1024 RAM, 2GB Root Disk, hw:mem_page_size 2048
2. Create a bootable volume
eg.
openstack --os-username 'tenant1' --os-password '<insertpwd>' --os-project-name tenant1 --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne volume create --image=188ba197-1006-4ef2-99ac-3329760c794f --bootable --size=2 vol-tenant1-vol_wendy-5

3. Launch the instance from volume

Volumes Attached appears to list vol-tenant1-vol_wendy-5 on /dev/vda

4. Create an additional volume and attach it so volumes attached appears to be 2
os-extended-volumes:volumes_attached | [{"id": "81ae169c-a989-4997-afc3-a7d25b7fcc6a", "delete_on_termination": false}, {"id": "61654dda-5ae6-4c40-b5a7-7c3c90781518", "delete_on_termination": false}]

$ nova show 08ea15f6-1284-4383-9a5f-8c13a41d9c17
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | AUTO |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | compute-1 |
| OS-EXT-SRV-ATTR:hostname | ttttttt |
| OS-EXT-SRV-ATTR:hypervisor_hostname | compute-1 |
| OS-EXT-SRV-ATTR:instance_name | instance-0000034e |
| OS-EXT-SRV-ATTR:kernel_id | |
| OS-EXT-SRV-ATTR:launch_index | 0 |
| OS-EXT-SRV-ATTR:ramdisk_id | |
| OS-EXT-SRV-ATTR:reservation_id | r-whmgw8d6 |
| OS-EXT-SRV-ATTR:root_device_name | /dev/vda |
| OS-EXT-SRV-ATTR:user_data | - |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2019-06-25T13:02:04.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2019-06-25T13:01:50Z |
| description | - |
| flavor:disk | 2 |
| flavor:ephemeral | 0 |
| flavor:extra_specs | {"hw:mem_page_size": "2048"} |
| flavor:original_name | flv_rootdisktest2 |
| flavor:ram | 1024 |
| flavor:swap | 0 |
| flavor:vcpus | 1 |
| hostId | 9a30d3bab229e5b0ec63017cbcf3690b110db0f8d73b9a30206ffb9f |
| host_status | UP |
| id | 08ea15f6-1284-4383-9a5f-8c13a41d9c17 |
| image | Attempt to boot from volume - no image supplied |
| key_name | - |
| locked | False |
| metadata | {} |
| name | ttttttt |
| os-extended-volumes:volumes_attached | [{"id": "81ae169c-a989-4997-afc3-a7d25b7fcc6a", "delete_on_termination": false}, {"id": "61654dda-5ae6-4c40-b5a7-7c3c90781518", "delete_on_termination": false}] |
| progress | 0 |
| security_groups | default |
| server_groups | [] |
| status | ACTIVE |
| tags | [] |
| tenant1-mgmt-net network | 192.168.85.52 |
| tenant1-net1 network | 172.16.1.131 |
| tenant_id | db4395f7baaa4de28a5417c659b28acd |
| trusted_image_certificates | - |
| updated | 2019-06-25T13:02:04Z |
| user_id | af0923a7f8a641d1902e3552be9f74f2

ssh to the host that the instance resides and run virsh domblklist to see how many volumes appear to be attached.

compute-1:~$ sudo virsh dumpxml instance-0000034e
<domain type='kvm' id='137'>
  <name>instance-0000034e</name>
  <uuid>08ea15f6-1284-4383-9a5f-8c13a41d9c17</uuid>
  <metadata>
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">
      <nova:package version="19.0.1"/>
      <nova:name>ttttttt</nova:name>
      <nova:creationTime>2019-06-25 13:01:56</nova:creationTime>
      <nova:flavor name="flv_rootdisktest2">
        <nova:memory>1024</nova:memory>
        <nova:disk>2</nova:disk>
        <nova:swap>0</nova:swap>
        <nova:ephemeral>0</nova:ephemeral>
        <nova:vcpus>1</nova:vcpus>
      </nova:flavor>
      <nova:owner>
        <nova:user uuid="af0923a7f8a641d1902e3552be9f74f2">admin</nova:user>
        <nova:project uuid="db4395f7baaa4de28a5417c659b28acd">tenant1</nova:project>
      </nova:owner>
    </nova:instance>
  </metadata>
  <memory unit='KiB'>1048576</memory>
  <currentMemory unit='KiB'>1048576</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size='2048' unit='KiB' nodeset='0'/>
    </hugepages>
  </memoryBacking>
  <vcpu placement='static'>1</vcpu>
  <cputune>
    <shares>1024</shares>
    <vcpupin vcpu='0' cpuset='3-23,51-71'/>
    <emulatorpin cpuset='3-23,51-71'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0'/>
    <memnode cellid='0' mode='strict' nodeset='0'/>
  </numatune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <sysinfo type='smbios'>
    <system>
      <entry name='manufacturer'>OpenStack Foundation</entry>
      <entry name='product'>OpenStack Nova</entry>
      <entry name='version'>19.0.1</entry>
      <entry name='serial'>08ea15f6-1284-4383-9a5f-8c13a41d9c17</entry>
      <entry name='uuid'>08ea15f6-1284-4383-9a5f-8c13a41d9c17</entry>
      <entry name='family'>Virtual Machine</entry>
    </system>
  </sysinfo>
  <os>
    <type arch='x86_64' machine='pc-i440fx-3.0'>hvm</type>
    <boot dev='hd'/>
    <smbios mode='sysinfo'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu>
    <topology sockets='1' cores='1' threads='1'/>
    <numa>
      <cell id='0' cpus='0' memory='1048576' unit='KiB' memAccess='shared'/>
    </numa>
  </cpu>
  <clock offset='utc'>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='writeback' discard='unmap'/>
      <auth username='cinder'>
        <secret type='ceph' uuid='457eb676-33da-42ec-9a8c-9293d545c337'/>
      </auth>
      <source protocol='rbd' name='cinder-volumes/81ae169c-a989-4997-afc3-a7d25b7fcc6a'>
        <host name='192.168.204.3' port='6789'/>
        <host name='192.168.204.4' port='6789'/>
        <host name='192.168.204.30' port='6789'/>
      </source>
      <target dev='vda' bus='virtio'/>
      <serial>81ae169c-a989-4997-afc3-a7d25b7fcc6a</serial>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='piix3-uhci'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <interface type='vhostuser'>
      <mac address='fa:16:3e:1d:2a:d7'/>
      <source type='unix' path='/var/run/openvswitch/vhu87c44b08-9d' mode='server'/>
      <target dev='vhu87c44b08-9d'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='fa:16:3e:1c:72:80'/>
      <source type='unix' path='/var/run/openvswitch/vhu9bb53f28-3c' mode='server'/>
      <target dev='vhu9bb53f28-3c'/>
      <model type='virtio'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/0'/>
      <log file='/var/lib/nova/instances/08ea15f6-1284-4383-9a5f-8c13a41d9c17/console.log' append='off'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/0'>
      <source path='/dev/pts/0'/>
      <log file='/var/lib/nova/instances/08ea15f6-1284-4383-9a5f-8c13a41d9c17/console.log' append='off'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'>
      <alias name='input1'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input2'/>
    </input>
    <graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+0:+0</label>
    <imagelabel>+0:+0</imagelabel>
  </seclabel>
</domain>

compute-1:~$ sudo virsh domblklist instance-0000034e
Target Source
------------------------------------------------
vda cinder-volumes/81ae169c-a989-4997-afc3-a7d25b7fcc6a

Expected Behaviour
------------------
Expect virsh domblkist instance-0000034e to report more than one volume attached

Actual Behaviour
----------------
Virsh unexpectedly only reports one volume attached

Reproducibility
---------------
100%

System Configuration
--------------------
standard eg. 2+3 system

Branch/Pull Time/Commit
-----------------------
BUILD_ID="20190622T013000Z"

Timestamp/Logs
--------------

Wendy Mitchell (wmitchellwr)
summary: - virsh only listing one volume whereas there was an additional volume
- attached after instantiation
+ virsh only listing one volume, even though there was an additional
+ volume attached after instantiation
Revision history for this message
Wendy Mitchell (wmitchellwr) wrote :

Regression failure due to this LP
FAIL test_evacuate_vms_with_inst_backing[local_image]
FAIL test_evacuate_vms_with_inst_backing[remote]

Numan Waheed (nwaheed)
tags: added: stx.regression stx.retestneeded
Revision history for this message
Wendy Mitchell (wmitchellwr) wrote :

Marking as regression affecting.

The following testcases all fail due to this issue:

FAIL 20190624 00:02:49 test_evacuate_vms_with_inst_backing[local_image]
FAIL 20190624 00:10:08 test_evacuate_vms_with_inst_backing[remote]

FAIL 20190624 02:04:16 test_live_migrate_vm_positive[local_image-0-0-None-2-image_with_vol-False]
FAIL 20190624 02:06:25 test_live_migrate_vm_positive[local_image-0-0-dedicated-1-image_with_vol-True]
FAIL 20190624 02:08:14 test_live_migrate_vm_positive[local_image-1-512-dedicated-2-image_with_vol-True]
FAIL 20190624 02:10:29 test_live_migrate_vm_positive[local_image-1-512-dedicated-1-image_with_vol-False]

Revision history for this message
Ghada Khalil (gkhalil) wrote :

Marking as stx.2.0 gating - issue w/ nova, libvirt or cinder?? Needs further investigation

tags: added: stx.2.0 stx.distro.openstack
Changed in starlingx:
importance: Undecided → Medium
status: New → Triaged
assignee: nobody → yong hu (yhu6)
Revision history for this message
Ghada Khalil (gkhalil) wrote :

Assigning to the distro.openstack team for investigation/resolution

yong hu (yhu6)
Changed in starlingx:
assignee: yong hu (yhu6) → Austin Sun (sunausti)
Revision history for this message
Austin Sun (sunausti) wrote :

based on test I made. The volume did not show up in tenant vm. from nova-compute log:
{"log":"2019-06-29 11:20:30.060 862779 INFO nova.compute.manager [req-72dcc04a-762f-4f5a-bea8-4514c21dc655 26223c26194e4d8599ad111b56930775 3ad5ee22bb1b4c279587cc6aeafd1d05 - default default] [instance: c54f7c86-6b59-48b9-839f-07be9ded8cec] Attaching volume c70fdb87-daf2-4f1b-b9b2-e053271b562b to /dev/vdb\n","stream":"stdout","time":"2019-06-29T11:20:30.060708716Z"}
{"log":"2019-06-29 11:20:30.060 862779 INFO nova.compute.manager [req-72dcc04a-762f-4f5a-bea8-4514c21dc655 26223c26194e4d8599ad111b56930775 3ad5ee22bb1b4c279587cc6aeafd1d05 - default default] [instance: c54f7c86-6b59-48b9-839f-07be9ded8cec] Attaching volume c70fdb87-daf2-4f1b-b9b2-e053271b562b to /dev/vdb\n","stream":"stdout","time":"2019-06-29T11:20:30.060798543Z"

it does not show this volume was successfully attached.

Revision history for this message
Austin Sun (sunausti) wrote :

more finding , after run 'openstack server add volume vm-boot mytest' , the libvirt has some error log ,
2019-07-04 00:30:56.244+0000: 180219: warning : qemuDomainObjTaint:7640 : Domain id=3 name='instance-00000004' uuid=7cfcc485-f1b9-42ec-aa84-08457b52bd1d is tainted: high-privileges
2019-07-04 00:49:16.189+0000: 180217: error : qemuDomainGetSecretAESAlias:788 : invalid argument: encrypted secret alias requires valid source alias

Revision history for this message
Austin Sun (sunausti) wrote :

https://bugzilla.redhat.com/show_bug.cgi?id=1672620

this was libvirt 4.7.0 regression, and was fixed 4.9.0 (4.8.0 was still have issue).

we have two options to fix this issue.
1) upgrading libvirt to 4.9.0
2) revert 192fdaa614e3800255048a8a70c1292ccf18397a as 4.9.0 did.

Changed in starlingx:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to integ (master)

Fix proposed to branch: master
Review: https://review.opendev.org/669323

Revision history for this message
Austin Sun (sunausti) wrote :

once https://github.com/starlingx-staging/stx-libvirt/pull/6 and https://review.opendev.org/669323 merged, and new docker images of stx-base , stx-libvirt and stx-nova were built, this issue should be fixed. I have verified locally with this change.

Revision history for this message
Wendy Mitchell (wmitchellwr) wrote :

Impacting regression tests
Lab: WP_3_7
Load: 20190706T013000Z
test_live_migrate_vm_positive[local_image-0-0-None-2-image_with_vol-False]
test_live_migrate_vm_positive[local_image-0-0-dedicated-1-image_with_vol-True]
test_live_migrate_vm_positive[local_image-1-512-dedicated-2-image_with_vol-True]
test_live_migrate_vm_positive[local_image-1-512-dedicated-1-image_with_vol-False]
test_cold_migrate_vm[local_image-0-0-None-1-image_with_vol-confirm]
test_cold_migrate_vm[local_image-0-0-dedicated-2-image_with_vol-revert]
test_evacuate_vms_with_inst_backing[local_image]

Revision history for this message
Wendy Mitchell (wmitchellwr) wrote :

The following nova (remote) regression testcases failed in the automated regression suite this week due to this issue:
Lab: WCP_113_121
Load: 20190713T013000Z

test_evacuate_vms.py::TestDefaultGuest::()::test_evacuate_vms_with_inst_backing[remote]
test_migrate_vms.py::test_live_migrate_vm_positive[remote-0-512-dedicated-2-image_with_vol-False]
test_migrate_vms.py::test_live_migrate_vm_negative[remote-0-0-image_with_vol-True-LiveMigErr.BLOCK_MIG_UNSUPPORTED]
test_migrate_vms.py::test_cold_migrate_vm[remote-0-0-None-2-image_with_vol-confirm]
test_migrate_vms.py::test_cold_migrate_vm[remote-1-0-None-2-image_with_vol-revert]

Revision history for this message
Austin Sun (sunausti) wrote :

with "20190717T013000Z" image. https://github.com/starlingx-staging/stx-libvirt/pull/6 was merged.

test result is positive.

    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none' discard='unmap'/>
      <source file='/var/lib/nova/instances/4a973e47-ff4d-4361-9ceb-a61befeffe97/disk'/>
      <backingStore type='file' index='1'>
        <format type='raw'/>
        <source file='/var/lib/nova/instances/_base/636047594352490faa5c41b54bb6535f9ec92ced'/>
        <backingStore/>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='writeback' discard='unmap'/>
      <auth username='cinder'>
        <secret type='ceph' uuid='457eb676-33da-42ec-9a8c-9293d545c337'/>
      </auth>
      <source protocol='rbd' name='cinder-volumes/1e3cc88a-e0ff-4ca6-ab44-f8475a470018'>
        <host name='192.178.204.3' port='6789'/>
      </source>
      <target dev='vdb' bus='virtio'/>
      <serial>1e3cc88a-e0ff-4ca6-ab44-f8475a470018</serial>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>

Revision history for this message
Austin Sun (sunausti) wrote :

please verify it after 20190717T013000Z

Revision history for this message
yong hu (yhu6) wrote :

https://github.com/starlingx-staging/stx-libvirt/pull/6 was merged, so marked as "fix released"

Changed in starlingx:
assignee: Austin Sun (sunausti) → Wendy Mitchell (wmitchellwr)
status: In Progress → Fix Released
Revision history for this message
Wendy Mitchell (wmitchellwr) wrote :

The following previously failing testcases are passing in regression.

Lab: WP_3_7
Load: 20190720T013000Z

PASS 20190721 00:08:32 test_live_migrate_vm_positive[local_image-0-0-None-2-image_with_vol-False]
PASS 20190721 00:11:05 test_live_migrate_vm_positive[local_image-0-0-dedicated-1-image_with_vol-True]
PASS 20190721 00:13:35 test_live_migrate_vm_positive[local_image-1-512-dedicated-2-image_with_vol-True]
PASS 20190721 00:16:29 test_live_migrate_vm_positive[local_image-1-512-dedicated-1-image_with_vol-False]

PASS 20190721 00:24:13 test_cold_migrate_vm[local_image-0-0-None-1-volume-confirm]
PASS 20190721 00:26:36 test_cold_migrate_vm[local_image-0-0-dedicated-2-volume-confirm]

Except for this one which did not actually evacuate (separate issue)
FAIL 20190720 21:22:30 test_evacuate_vms_with_inst_backing[local_image]
Details: Following VMs stayed on the same host compute-1:

tags: removed: stx.retestneeded
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to integ (master)

Reviewed: https://review.opendev.org/669323
Committed: https://git.openstack.org/cgit/starlingx/integ/commit/?id=3d5300ec250b3f6c3900fc56a3933ea4e61735f2
Submitter: Zuul
Branch: master

commit 3d5300ec250b3f6c3900fc56a3933ea4e61735f2
Author: Sun Austin <email address hidden>
Date: Fri Jul 5 18:43:13 2019 +0800

    change tis patch version to 101 for below PR

    https://github.com/starlingx-staging/stx-libvirt/pull/6

    Revert "qemu: hotplug: Prepare disk source in
    qemuDomainAttachDeviceDiskLive"
    This reverts commit 192fdaa614e3800255048a8a70c1292ccf18397a.
    According to https://bugzilla.redhat.com/show_bug.cgi?id=1672620
    reverting this commit will fix
    https://bugs.launchpad.net/starlingx/+bug/1834194 and
    reverting is official merged in libvirt-4.9.0 too.

    Partial-Bug: #1834194

    Change-Id: I0ea21e0b39ef016ecff113bb107b0667a2105424
    Signed-off-by: Sun Austin <email address hidden>

Revision history for this message
Ghada Khalil (gkhalil) wrote :

Bugs should remain assigned to the developer who provided a fix. They shouldn't be assigned back to the reporter.

Changed in starlingx:
assignee: Wendy Mitchell (wmitchellwr) → Austin Sun (sunausti)
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.