| 2020-05-25 06:24:49 |
alpha23 |
description |
Using Rocky centos/source containers
Docker version - 18.06.1-ce
Hosts are Ubuntu 18.04.4 LTS
Receiving the following defect on nova_compute which only occurs when attempting to attach an existing and unattached volume to a VM.
- Instances are created on the same host without issue and access to the virtual host is accessible via Virtual Machine Manager. For testing purposes, I was able to attach the same volume via VMM.
- All compute, block, and network services are enabled and up.
- 'nova hypervisor-list' shows the hypervisor up
- The same volume and other volumes attach to VMs on a different hypervisor without issue.
- The only difference between hypervisor hostnames that I see between the one that is failing and the one that is not is that the one that is failing is showing a FQDN. Note below, it specifies only the hostname, not the FQDN.
- From nova_compute, both the host name or the FQDN resolve correctly to the same IP and I can ping either.
- virt_type is kvm
- compute_driver is libvirt.LibvirtDriver
- kernel on host that is failing is 4.15.0-101-generic
Is this a settings issue, a permissions issue, a driver issue, or something else?
020-05-24 23:52:54.692 7 DEBUG oslo_concurrency.lockutils [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] Lock "d51fddde-1b0c-4365-b1ec-ad508d2d0bed" released by "nova.compute.manager.do_reserve" :: held 0.154s inner /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:285
2020-05-24 23:52:55.469 7 DEBUG oslo_concurrency.lockutils [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] Lock "d51fddde-1b0c-4365-b1ec-ad508d2d0bed" acquired by "nova.compute.manager.do_attach_volume" :: waited 0.000s inner /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:273
2020-05-24 23:52:55.470 7 INFO nova.compute.manager [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] Attaching volume cadca51f-ff52-4b12-9777-4f86514e3eef to /dev/vdb
2020-05-24 23:52:55.471 7 DEBUG nova.objects.instance [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] Lazy-loading 'flavor' on Instance uuid d51fddde-1b0c-4365-b1ec-ad508d2d0bed obj_load_attr /var/lib/kolla/venv/lib/python2.7/site-packages/nova/objects/instance.py:1111
2020-05-24 23:52:55.738 7 DEBUG os_brick.utils [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] ==> get_connector_properties: call u"{'execute': None, 'my_ip': '192.168.3.252', 'enforce_multipath': True, 'host': 'st1', 'root_helper': 'sudo nova-rootwrap /etc/nova/rootwrap.conf', 'multipath': False}" trace_logging_wrapper /var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/utils.py:146
2020-05-24 23:52:56.428 7 DEBUG os_brick.initiator.linuxfc [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] No Fibre Channel support detected on system. get_fc_hbas /var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/initiator/linuxfc.py:157
2020-05-24 23:52:56.429 7 DEBUG os_brick.initiator.linuxfc [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] No Fibre Channel support detected on system. get_fc_hbas /var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/initiator/linuxfc.py:157
2020-05-24 23:52:56.430 7 DEBUG os_brick.utils [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] <== get_connector_properties: return (691ms) {'initiator': u'iqn.1994-05.com.redhat:cddc7297faec', 'ip': u'192.168.3.252', 'platform': u'x86_64', 'host': u'st1', 'do_local_attach': False, 'os_type': u'linux2', 'multipath': False} trace_logging_wrapper /var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/utils.py:170
2020-05-24 23:52:56.432 7 DEBUG nova.virt.block_device [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] Updating existing volume attachment record: 9a69f304-8468-47f6-8b59-5edd1e3dd75f _volume_attach /var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/block_device.py:535
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] Driver failed to attach volume cadca51f-ff52-4b12-9777-4f86514e3eef at /dev/vdb: HypervisorUnavailable: Connection to the hypervisor is broken on host: st1
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] Traceback (most recent call last):
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/block_device.py", line 563, in _volume_attach
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] device_type=self['device_type'], encryption=encryption)
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1481, in attach_volume
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] encryption=encryption)
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1264, in _connect_volume
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] vol_driver.connect_volume(connection_info, instance)
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/volume/fs.py", line 117, in connect_volume
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] self._mount_options(connection_info))
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/volume/mount.py", line 409, in mount
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] with __manager__.get_state() as mount_state:
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] return self.gen.next()
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/volume/mount.py", line 89, in get_state
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] raise exception.HypervisorUnavailable(host=CONF.host)
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] HypervisorUnavailable: Connection to the hypervisor is broken on host: st1
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] |
Using Rocky centos/source containers
Docker version - 18.06.1-ce
Hosts are Ubuntu 18.04.4 LTS
Receiving the following defect on nova_compute which only occurs when attempting to attach an existing and unattached volume to a VM.
- Instances are created on the same host without issue and access to the virtual host is accessible via Virtual Machine Manager. For testing purposes, I was able to attach the same volume via VMM.
- All compute, block, and network services are enabled and up.
- 'nova hypervisor-list' shows the hypervisor up
- The same volume and other volumes attach to VMs on a different hypervisor without issue.
- The only difference between hypervisor hostnames that I see between the one that is failing and the one that is not is that the one that is failing is showing a FQDN. Note below, it specifies only the hostname, not the FQDN.
- From nova_compute, both the host name or the FQDN resolve correctly to the same IP and I can ping either.
- virt_type is kvm
- compute_driver is libvirt.LibvirtDriver
- kernel on both hosts (failing and not failing to attach) are 4.15.0-101-generic
Is this a settings issue, a permissions issue, a driver issue, or something else?
020-05-24 23:52:54.692 7 DEBUG oslo_concurrency.lockutils [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] Lock "d51fddde-1b0c-4365-b1ec-ad508d2d0bed" released by "nova.compute.manager.do_reserve" :: held 0.154s inner /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:285
2020-05-24 23:52:55.469 7 DEBUG oslo_concurrency.lockutils [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] Lock "d51fddde-1b0c-4365-b1ec-ad508d2d0bed" acquired by "nova.compute.manager.do_attach_volume" :: waited 0.000s inner /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:273
2020-05-24 23:52:55.470 7 INFO nova.compute.manager [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] Attaching volume cadca51f-ff52-4b12-9777-4f86514e3eef to /dev/vdb
2020-05-24 23:52:55.471 7 DEBUG nova.objects.instance [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] Lazy-loading 'flavor' on Instance uuid d51fddde-1b0c-4365-b1ec-ad508d2d0bed obj_load_attr /var/lib/kolla/venv/lib/python2.7/site-packages/nova/objects/instance.py:1111
2020-05-24 23:52:55.738 7 DEBUG os_brick.utils [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] ==> get_connector_properties: call u"{'execute': None, 'my_ip': '192.168.3.252', 'enforce_multipath': True, 'host': 'st1', 'root_helper': 'sudo nova-rootwrap /etc/nova/rootwrap.conf', 'multipath': False}" trace_logging_wrapper /var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/utils.py:146
2020-05-24 23:52:56.428 7 DEBUG os_brick.initiator.linuxfc [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] No Fibre Channel support detected on system. get_fc_hbas /var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/initiator/linuxfc.py:157
2020-05-24 23:52:56.429 7 DEBUG os_brick.initiator.linuxfc [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] No Fibre Channel support detected on system. get_fc_hbas /var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/initiator/linuxfc.py:157
2020-05-24 23:52:56.430 7 DEBUG os_brick.utils [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] <== get_connector_properties: return (691ms) {'initiator': u'iqn.1994-05.com.redhat:cddc7297faec', 'ip': u'192.168.3.252', 'platform': u'x86_64', 'host': u'st1', 'do_local_attach': False, 'os_type': u'linux2', 'multipath': False} trace_logging_wrapper /var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/utils.py:170
2020-05-24 23:52:56.432 7 DEBUG nova.virt.block_device [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] Updating existing volume attachment record: 9a69f304-8468-47f6-8b59-5edd1e3dd75f _volume_attach /var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/block_device.py:535
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] Driver failed to attach volume cadca51f-ff52-4b12-9777-4f86514e3eef at /dev/vdb: HypervisorUnavailable: Connection to the hypervisor is broken on host: st1
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] Traceback (most recent call last):
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/block_device.py", line 563, in _volume_attach
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] device_type=self['device_type'], encryption=encryption)
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1481, in attach_volume
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] encryption=encryption)
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1264, in _connect_volume
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] vol_driver.connect_volume(connection_info, instance)
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/volume/fs.py", line 117, in connect_volume
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] self._mount_options(connection_info))
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/volume/mount.py", line 409, in mount
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] with __manager__.get_state() as mount_state:
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] return self.gen.next()
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/volume/mount.py", line 89, in get_state
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] raise exception.HypervisorUnavailable(host=CONF.host)
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] HypervisorUnavailable: Connection to the hypervisor is broken on host: st1
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] |
|
| 2020-05-25 06:28:01 |
alpha23 |
description |
Using Rocky centos/source containers
Docker version - 18.06.1-ce
Hosts are Ubuntu 18.04.4 LTS
Receiving the following defect on nova_compute which only occurs when attempting to attach an existing and unattached volume to a VM.
- Instances are created on the same host without issue and access to the virtual host is accessible via Virtual Machine Manager. For testing purposes, I was able to attach the same volume via VMM.
- All compute, block, and network services are enabled and up.
- 'nova hypervisor-list' shows the hypervisor up
- The same volume and other volumes attach to VMs on a different hypervisor without issue.
- The only difference between hypervisor hostnames that I see between the one that is failing and the one that is not is that the one that is failing is showing a FQDN. Note below, it specifies only the hostname, not the FQDN.
- From nova_compute, both the host name or the FQDN resolve correctly to the same IP and I can ping either.
- virt_type is kvm
- compute_driver is libvirt.LibvirtDriver
- kernel on both hosts (failing and not failing to attach) are 4.15.0-101-generic
Is this a settings issue, a permissions issue, a driver issue, or something else?
020-05-24 23:52:54.692 7 DEBUG oslo_concurrency.lockutils [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] Lock "d51fddde-1b0c-4365-b1ec-ad508d2d0bed" released by "nova.compute.manager.do_reserve" :: held 0.154s inner /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:285
2020-05-24 23:52:55.469 7 DEBUG oslo_concurrency.lockutils [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] Lock "d51fddde-1b0c-4365-b1ec-ad508d2d0bed" acquired by "nova.compute.manager.do_attach_volume" :: waited 0.000s inner /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:273
2020-05-24 23:52:55.470 7 INFO nova.compute.manager [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] Attaching volume cadca51f-ff52-4b12-9777-4f86514e3eef to /dev/vdb
2020-05-24 23:52:55.471 7 DEBUG nova.objects.instance [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] Lazy-loading 'flavor' on Instance uuid d51fddde-1b0c-4365-b1ec-ad508d2d0bed obj_load_attr /var/lib/kolla/venv/lib/python2.7/site-packages/nova/objects/instance.py:1111
2020-05-24 23:52:55.738 7 DEBUG os_brick.utils [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] ==> get_connector_properties: call u"{'execute': None, 'my_ip': '192.168.3.252', 'enforce_multipath': True, 'host': 'st1', 'root_helper': 'sudo nova-rootwrap /etc/nova/rootwrap.conf', 'multipath': False}" trace_logging_wrapper /var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/utils.py:146
2020-05-24 23:52:56.428 7 DEBUG os_brick.initiator.linuxfc [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] No Fibre Channel support detected on system. get_fc_hbas /var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/initiator/linuxfc.py:157
2020-05-24 23:52:56.429 7 DEBUG os_brick.initiator.linuxfc [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] No Fibre Channel support detected on system. get_fc_hbas /var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/initiator/linuxfc.py:157
2020-05-24 23:52:56.430 7 DEBUG os_brick.utils [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] <== get_connector_properties: return (691ms) {'initiator': u'iqn.1994-05.com.redhat:cddc7297faec', 'ip': u'192.168.3.252', 'platform': u'x86_64', 'host': u'st1', 'do_local_attach': False, 'os_type': u'linux2', 'multipath': False} trace_logging_wrapper /var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/utils.py:170
2020-05-24 23:52:56.432 7 DEBUG nova.virt.block_device [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] Updating existing volume attachment record: 9a69f304-8468-47f6-8b59-5edd1e3dd75f _volume_attach /var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/block_device.py:535
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] Driver failed to attach volume cadca51f-ff52-4b12-9777-4f86514e3eef at /dev/vdb: HypervisorUnavailable: Connection to the hypervisor is broken on host: st1
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] Traceback (most recent call last):
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/block_device.py", line 563, in _volume_attach
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] device_type=self['device_type'], encryption=encryption)
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1481, in attach_volume
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] encryption=encryption)
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1264, in _connect_volume
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] vol_driver.connect_volume(connection_info, instance)
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/volume/fs.py", line 117, in connect_volume
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] self._mount_options(connection_info))
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/volume/mount.py", line 409, in mount
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] with __manager__.get_state() as mount_state:
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] return self.gen.next()
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/volume/mount.py", line 89, in get_state
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] raise exception.HypervisorUnavailable(host=CONF.host)
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] HypervisorUnavailable: Connection to the hypervisor is broken on host: st1
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] |
Using Rocky centos/source containers
Docker version - 18.06.1-ce
Hosts are Ubuntu 18.04.4 LTS
Receiving the following defect on nova_compute which only occurs when attempting to attach an existing and unattached volume to a VM.
- Instances are created on the same host without issue and access to the virtual host is accessible via Virtual Machine Manager. For testing purposes, I was able to attach the same volume via VMM.
- All compute, block, and network services are enabled and up.
- 'nova hypervisor-list' shows the hypervisor up
- The same volume and other volumes attach to VMs on a different hypervisor without issue.
- The only difference between hypervisor hostnames that I see between the one that is failing and the one that is not is that the one that is failing is showing a FQDN. Note below, it specifies only the hostname, not the FQDN.
- From nova_compute, both the host name or the FQDN resolve correctly to the same IP and I can ping either.
- virt_type is kvm
- compute_driver is libvirt.LibvirtDriver
- kernel on both hosts (failing and not failing to attach) are 4.15.0-101-generic
Is this a settings issue, a permissions issue, a driver issue, or something else?
From nova-compute.log:
020-05-24 23:52:54.692 7 DEBUG oslo_concurrency.lockutils [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] Lock "d51fddde-1b0c-4365-b1ec-ad508d2d0bed" released by "nova.compute.manager.do_reserve" :: held 0.154s inner /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:285
2020-05-24 23:52:55.469 7 DEBUG oslo_concurrency.lockutils [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] Lock "d51fddde-1b0c-4365-b1ec-ad508d2d0bed" acquired by "nova.compute.manager.do_attach_volume" :: waited 0.000s inner /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:273
2020-05-24 23:52:55.470 7 INFO nova.compute.manager [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] Attaching volume cadca51f-ff52-4b12-9777-4f86514e3eef to /dev/vdb
2020-05-24 23:52:55.471 7 DEBUG nova.objects.instance [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] Lazy-loading 'flavor' on Instance uuid d51fddde-1b0c-4365-b1ec-ad508d2d0bed obj_load_attr /var/lib/kolla/venv/lib/python2.7/site-packages/nova/objects/instance.py:1111
2020-05-24 23:52:55.738 7 DEBUG os_brick.utils [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] ==> get_connector_properties: call u"{'execute': None, 'my_ip': '192.168.3.252', 'enforce_multipath': True, 'host': 'st1', 'root_helper': 'sudo nova-rootwrap /etc/nova/rootwrap.conf', 'multipath': False}" trace_logging_wrapper /var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/utils.py:146
2020-05-24 23:52:56.428 7 DEBUG os_brick.initiator.linuxfc [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] No Fibre Channel support detected on system. get_fc_hbas /var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/initiator/linuxfc.py:157
2020-05-24 23:52:56.429 7 DEBUG os_brick.initiator.linuxfc [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] No Fibre Channel support detected on system. get_fc_hbas /var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/initiator/linuxfc.py:157
2020-05-24 23:52:56.430 7 DEBUG os_brick.utils [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] <== get_connector_properties: return (691ms) {'initiator': u'iqn.1994-05.com.redhat:cddc7297faec', 'ip': u'192.168.3.252', 'platform': u'x86_64', 'host': u'st1', 'do_local_attach': False, 'os_type': u'linux2', 'multipath': False} trace_logging_wrapper /var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/utils.py:170
2020-05-24 23:52:56.432 7 DEBUG nova.virt.block_device [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] Updating existing volume attachment record: 9a69f304-8468-47f6-8b59-5edd1e3dd75f _volume_attach /var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/block_device.py:535
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] Driver failed to attach volume cadca51f-ff52-4b12-9777-4f86514e3eef at /dev/vdb: HypervisorUnavailable: Connection to the hypervisor is broken on host: st1
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] Traceback (most recent call last):
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/block_device.py", line 563, in _volume_attach
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] device_type=self['device_type'], encryption=encryption)
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1481, in attach_volume
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] encryption=encryption)
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1264, in _connect_volume
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] vol_driver.connect_volume(connection_info, instance)
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/volume/fs.py", line 117, in connect_volume
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] self._mount_options(connection_info))
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/volume/mount.py", line 409, in mount
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] with __manager__.get_state() as mount_state:
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] return self.gen.next()
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/volume/mount.py", line 89, in get_state
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] raise exception.HypervisorUnavailable(host=CONF.host)
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] HypervisorUnavailable: Connection to the hypervisor is broken on host: st1
2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] |
|
