Not able to launch VMs with contrail 3.2.3.2 b6 centos Mitaka - Issue due to libvirt version change in CentOS , workaround available

[Fawad]

Setup:
contrail version: 3.2.3.2 b6 centos Mitaka

Feature enabled:
1) SSL
2) RBAC
3) Keystone v3

Seeing the following error (/var/log/nova/nova-compute.log) while launching the VM:

2017-07-24 16:14:11.303 4194 ERROR nova.compute.manager [instance: d1bb55a7-d2ca-4459-8fe8-1fc318212e5b] block_device_info=block_device_info)
2017-07-24 16:14:11.303 4194 ERROR nova.compute.manager [instance: d1bb55a7-d2ca-4459-8fe8-1fc318212e5b] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4923, in _create_domain_and_network
2017-07-24 16:14:11.303 4194 ERROR nova.compute.manager [instance: d1bb55a7-d2ca-4459-8fe8-1fc318212e5b] xml, pause=pause, power_on=power_on)
2017-07-24 16:14:11.303 4194 ERROR nova.compute.manager [instance: d1bb55a7-d2ca-4459-8fe8-1fc318212e5b] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4854, in _create_domain
2017-07-24 16:14:11.303 4194 ERROR nova.compute.manager [instance: d1bb55a7-d2ca-4459-8fe8-1fc318212e5b] guest.launch(pause=pause)
2017-07-24 16:14:11.303 4194 ERROR nova.compute.manager [instance: d1bb55a7-d2ca-4459-8fe8-1fc318212e5b] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 142, in launch
2017-07-24 16:14:11.303 4194 ERROR nova.compute.manager [instance: d1bb55a7-d2ca-4459-8fe8-1fc318212e5b] self._encoded_xml, errors='ignore')
2017-07-24 16:14:11.303 4194 ERROR nova.compute.manager [instance: d1bb55a7-d2ca-4459-8fe8-1fc318212e5b] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-07-24 16:14:11.303 4194 ERROR nova.compute.manager [instance: d1bb55a7-d2ca-4459-8fe8-1fc318212e5b] self.force_reraise()
2017-07-24 16:14:11.303 4194 ERROR nova.compute.manager [instance: d1bb55a7-d2ca-4459-8fe8-1fc318212e5b] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-07-24 16:14:11.303 4194 ERROR nova.compute.manager [instance: d1bb55a7-d2ca-4459-8fe8-1fc318212e5b] six.reraise(self.type_, self.value, self.tb)
2017-07-24 16:14:11.303 4194 ERROR nova.compute.manager [instance: d1bb55a7-d2ca-4459-8fe8-1fc318212e5b] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 137, in launch
2017-07-24 16:14:11.303 4194 ERROR nova.compute.manager [instance: d1bb55a7-d2ca-4459-8fe8-1fc318212e5b] return self._domain.createWithFlags(flags)
2017-07-24 16:14:11.303 4194 ERROR nova.compute.manager [instance: d1bb55a7-d2ca-4459-8fe8-1fc318212e5b] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2017-07-24 16:14:11.303 4194 ERROR nova.compute.manager [instance: d1bb55a7-d2ca-4459-8fe8-1fc318212e5b] result = proxy_call(self._autowrap, f, *args, **kwargs)
2017-07-24 16:14:11.303 4194 ERROR nova.compute.manager [instance: d1bb55a7-d2ca-4459-8fe8-1fc318212e5b] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2017-07-24 16:14:11.303 4194 ERROR nova.compute.manager [instance: d1bb55a7-d2ca-4459-8fe8-1fc318212e5b] rv = execute(f, *args, **kwargs)
2017-07-24 16:14:11.303 4194 ERROR nova.compute.manager [instance: d1bb55a7-d2ca-4459-8fe8-1fc318212e5b] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2017-07-24 16:14:11.303 4194 ERROR nova.compute.manager [instance: d1bb55a7-d2ca-4459-8fe8-1fc318212e5b] six.reraise(c, e, tb)
2017-07-24 16:14:11.303 4194 ERROR nova.compute.manager [instance: d1bb55a7-d2ca-4459-8fe8-1fc318212e5b] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2017-07-24 16:14:11.303 4194 ERROR nova.compute.manager [instance: d1bb55a7-d2ca-4459-8fe8-1fc318212e5b] rv = meth(*args, **kwargs)
2017-07-24 16:14:11.303 4194 ERROR nova.compute.manager [instance: d1bb55a7-d2ca-4459-8fe8-1fc318212e5b] File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1065, in createWithFlags
2017-07-24 16:14:11.303 4194 ERROR nova.compute.manager [instance: d1bb55a7-d2ca-4459-8fe8-1fc318212e5b] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2017-07-24 16:14:11.303 4194 ERROR nova.compute.manager [instance: d1bb55a7-d2ca-4459-8fe8-1fc318212e5b] libvirtError: Cannot find '' in path: No such file or directory
2017-07-24 16:14:11.303 4194 ERROR nova.compute.manager [instance: d1bb55a7-d2ca-4459-8fe8-1fc318212e5b]

Seems like there is an issue with the latest libvirt version where it doesn’t take empty parameters.

Workaround:

1) Go to compute node:

    - Changed "conf.script = “” to conf.script = None (line 61) in /usr/lib/python2.7/site-packages/nova/virt/libvirt/designer.py
    - service libvirtd restart
    - service openstack-nova-compute restart
2) Everything should work fine

[root@contrail-node23 ~]# virsh version
Compiled against library: libvirt 2.0.0
Using library: libvirt 2.0.0
Using API: QEMU 2.0.0
Running hypervisor: QEMU 1.5.3

Please let me know if you need anything else

[Hari Prasad Killi (haripk)]

Needs https://github.com/openstack/nova/commit/a41d265a19b7bcb1af8fc179bf864e00023c6cc6 from upstream, this seems to be fixed in Newton and Ocata. As this depends on other changes, we may have to wait for a sync.

Also need to decide if this fix has to be put in Mitaka.