This issue actually appears to be unrelated to where the exception is thrown. Instead it is related to nova_compute start (relevant code below). The code loops through each guest drive and if the drive is not connect (in my case, there was a CDROM that was unconnected that was unrelated to the instance under test), throws an exception and then prevents all cinder volume attachments. I did not go through and track down what nova compute is actually doing after the fact; however, this was not an issue with Queens. Once the CDROM device was removed, the exception initially exhibited goes away.
Why is the code not skipping over drives that do not have a drive path but instead throws an exception, never complete whatever initialization is required, and then re-appears later with an exception that is unrelated to the root cause?
As a note, the code in Queens worked without issue.
2020-05-31 22:21:55.146 9 DEBUG nova.virt.libvirt.volume.mount [req-acf5eab2-20c2-477c-a902-dfa067ab3137 - - - - -] Discovered volume volume-cd814d75-4b53-4258-b133-1617d3bfecb8 in use for existing mountpoint /var/lib/nova/mnt/f7af0a8752d0f8b2542c2254fb348896 __init__ /var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/volume/mount.py:239
2020-05-31 22:21:55.148 9 ERROR nova.virt.libvirt.host [req-acf5eab2-20c2-477c-a902-dfa067ab3137 - - - - -] Exception handling connection event: AttributeError: 'NoneType' object has no attribute 'rfind'
2020-05-31 22:21:55.148 9 ERROR nova.virt.libvirt.host Traceback (most recent call last):
2020-05-31 22:21:55.148 9 ERROR nova.virt.libvirt.host File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 148, in _dispatch_conn_event
2020-05-31 22:21:55.148 9 ERROR nova.virt.libvirt.host handler()
2020-05-31 22:21:55.148 9 ERROR nova.virt.libvirt.host File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 414, in handler
2020-05-31 22:21:55.148 9 ERROR nova.virt.libvirt.host return self._conn_event_handler(*args, **kwargs)
2020-05-31 22:21:55.148 9 ERROR nova.virt.libvirt.host File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 477, in _handle_conn_event
2020-05-31 22:21:55.148 9 ERROR nova.virt.libvirt.host self._set_host_enabled(enabled, reason)
2020-05-31 22:21:55.148 9 ERROR nova.virt.libvirt.host File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3842, in _set_host_enabled
2020-05-31 22:21:55.148 9 ERROR nova.virt.libvirt.host mount.get_manager().host_up(self._host)
2020-05-31 22:21:55.148 9 ERROR nova.virt.libvirt.host File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/volume/mount.py", line 134, in host_up
2020-05-31 22:21:55.148 9 ERROR nova.virt.libvirt.host self.state = _HostMountState(host, self.generation)
2020-05-31 22:21:55.148 9 ERROR nova.virt.libvirt.host File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/volume/mount.py", line 229, in __init__
2020-05-31 22:21:55.148 9 ERROR nova.virt.libvirt.host mountpoint = os.path.dirname(disk.source_path)
2020-05-31 22:21:55.148 9 ERROR nova.virt.libvirt.host File "/usr/lib64/python2.7/posixpath.py", line 129, in dirname
2020-05-31 22:21:55.148 9 ERROR nova.virt.libvirt.host i = p.rfind('/') + 1
2020-05-31 22:21:55.148 9 ERROR nova.virt.libvirt.host AttributeError: 'NoneType' object has no attribute 'rfind'
2020-05-31 22:21:55.148 9 ERROR nova.virt.libvirt.host
This issue actually appears to be unrelated to where the exception is thrown. Instead it is related to nova_compute start (relevant code below). The code loops through each guest drive and if the drive is not connect (in my case, there was a CDROM that was unconnected that was unrelated to the instance under test), throws an exception and then prevents all cinder volume attachments. I did not go through and track down what nova compute is actually doing after the fact; however, this was not an issue with Queens. Once the CDROM device was removed, the exception initially exhibited goes away.
Why is the code not skipping over drives that do not have a drive path but instead throws an exception, never complete whatever initialization is required, and then re-appears later with an exception that is unrelated to the root cause?
As a note, the code in Queens worked without issue.
2020-05-31 22:21:55.146 9 DEBUG nova.virt. libvirt. volume. mount [req-acf5eab2- 20c2-477c- a902-dfa067ab31 37 - - - - -] Discovered volume volume- cd814d75- 4b53-4258- b133-1617d3bfec b8 in use for existing mountpoint /var/lib/ nova/mnt/ f7af0a8752d0f8b 2542c2254fb3488 96 __init__ /var/lib/ kolla/venv/ lib/python2. 7/site- packages/ nova/virt/ libvirt/ volume/ mount.py: 239 libvirt. host [req-acf5eab2- 20c2-477c- a902-dfa067ab31 37 - - - - -] Exception handling connection event: AttributeError: 'NoneType' object has no attribute 'rfind' libvirt. host Traceback (most recent call last): libvirt. host File "/var/lib/ kolla/venv/ lib/python2. 7/site- packages/ nova/virt/ libvirt/ host.py" , line 148, in _dispatch_ conn_event libvirt. host handler() libvirt. host File "/var/lib/ kolla/venv/ lib/python2. 7/site- packages/ nova/virt/ libvirt/ host.py" , line 414, in handler libvirt. host return self._conn_ event_handler( *args, **kwargs) libvirt. host File "/var/lib/ kolla/venv/ lib/python2. 7/site- packages/ nova/virt/ libvirt/ driver. py", line 477, in _handle_conn_event libvirt. host self._set_ host_enabled( enabled, reason) libvirt. host File "/var/lib/ kolla/venv/ lib/python2. 7/site- packages/ nova/virt/ libvirt/ driver. py", line 3842, in _set_host_enabled libvirt. host mount.get_ manager( ).host_ up(self. _host) libvirt. host File "/var/lib/ kolla/venv/ lib/python2. 7/site- packages/ nova/virt/ libvirt/ volume/ mount.py" , line 134, in host_up libvirt. host self.state = _HostMountState (host, self.generation) libvirt. host File "/var/lib/ kolla/venv/ lib/python2. 7/site- packages/ nova/virt/ libvirt/ volume/ mount.py" , line 229, in __init__ libvirt. host mountpoint = os.path. dirname( disk.source_ path) libvirt. host File "/usr/lib64/ python2. 7/posixpath. py", line 129, in dirname libvirt. host i = p.rfind('/') + 1 libvirt. host AttributeError: 'NoneType' object has no attribute 'rfind' libvirt. host
2020-05-31 22:21:55.148 9 ERROR nova.virt.
2020-05-31 22:21:55.148 9 ERROR nova.virt.
2020-05-31 22:21:55.148 9 ERROR nova.virt.
2020-05-31 22:21:55.148 9 ERROR nova.virt.
2020-05-31 22:21:55.148 9 ERROR nova.virt.
2020-05-31 22:21:55.148 9 ERROR nova.virt.
2020-05-31 22:21:55.148 9 ERROR nova.virt.
2020-05-31 22:21:55.148 9 ERROR nova.virt.
2020-05-31 22:21:55.148 9 ERROR nova.virt.
2020-05-31 22:21:55.148 9 ERROR nova.virt.
2020-05-31 22:21:55.148 9 ERROR nova.virt.
2020-05-31 22:21:55.148 9 ERROR nova.virt.
2020-05-31 22:21:55.148 9 ERROR nova.virt.
2020-05-31 22:21:55.148 9 ERROR nova.virt.
2020-05-31 22:21:55.148 9 ERROR nova.virt.
2020-05-31 22:21:55.148 9 ERROR nova.virt.
2020-05-31 22:21:55.148 9 ERROR nova.virt.
2020-05-31 22:21:55.148 9 ERROR nova.virt.