Deploy Nova libvirt with Kolla kubernetes get error for libvirt

[Kevin Zhao]

I've already change the nova-libvirt yaml file, comment the "rpm -e qemu-system-x86 || true;"
=================================================================================
linux@kolla-kubernetes-arm64:~/kolla-bringup/k8s/kolla-kubernetes$ git diff helm/microservice/nova-libvirt-daemonset/templates/nova-libvirt.yaml
diff --git a/helm/microservice/nova-libvirt-daemonset/templates/nova-libvirt.yaml b/helm/microservice/nova-libvirt-daemonset/templates/nova-libvirt.yaml
index 58caaaf..e851cf0 100644
--- a/helm/microservice/nova-libvirt-daemonset/templates/nova-libvirt.yaml
+++ b/helm/microservice/nova-libvirt-daemonset/templates/nova-libvirt.yaml
@@ -91,8 +91,12 @@ spec:
               modprobe rbd;
               # Following is added to workaround bug
               # https://bugs.launchpad.net/kolla/+bug/1649009
+{{- if $localVals.kolla_base_distro }}
               rpm -e qemu-system-x86 || true;
+{{- end }}
+{{- if $localVals.ceph_backend }}
               cp -a {{ $containerConfigDirectory }}/secrets /etc/libvirt;
+{{- end }}
               kolla_start
           securityContext:
             privileged: true

=================================================================================
Use "kubectl logs nova-compute** -n kolla" I can see the libvirtd starts.
But check the libvirtd I find the deployment failed
still got the error as below:

Image is 5.0.1, Pike,
Libvirt version is 3.6.0

Steps to Reproduce: Minimized, easy-to-follow steps that will trigger the bug. Include any special setup steps.
1.helm install --debug kolla-kubernetes/helm/service/nova-compute --namespace kolla --name nova-compute --values /tmp/cloud.yaml
2. wait and get the logs from nova-libvirt container
3. docker inspect <nova libvirt container id>
4. Get the logs directory from host()
            {
                "Type": "bind",
                "Source": "/var/lib/kubelet/pods/31346c57-cb5c-11e7-bfb3-525400a49c3b/volumes/kubernetes.io~empty-dir/kolla-logs",
                "Destination": "/var/log/kolla",
                "Mode": "",
                "RW": true,
                "Propagation": ""
            },

5. Check the libvirtd logs
...

Container image:
haerwu/debian-source-nova-libvirt 5.0.1 2ae38ecc5e5c 3 days ago 1.02 GB

Actual Results: What happened after performing the above steps.
Check the logs from the container.
2017-11-17 05:57:57.385+0000: 18161: info : libvirt version: 3.6.0, package: 1~bpo9+1 (Riku Voipio <email address hidden> Mon, 14 Aug 2017 14:02:02 +0300)
2017-11-17 05:57:57.385+0000: 18161: info : hostname: kolla-kubernetes-arm64
2017-11-17 05:57:57.385+0000: 18161: error : daemonSetupNetworking:502 : unsupported configuration: No server certificate path set to match server key
server certificate path set to match server key

[Kevin Zhao (kevin-zhao)]

If not apply the patch in the above:

error log:
/bin/bash: line 3: rpm: command not found
cp: cannot stat '/var/lib/kolla/config_files/secrets': No such file or directory
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Kolla config strategy set to: COPY_ONCE
INFO:__main__:Copying service configuration files
INFO:__main__:Deleting /etc/libvirt/libvirtd.conf
INFO:__main__:Copying /var/lib/kolla/config_files/libvirtd.conf to /etc/libvirt/libvirtd.conf
INFO:__main__:Setting permission for /etc/libvirt/libvirtd.conf
INFO:__main__:Deleting /etc/libvirt/qemu.conf
INFO:__main__:Copying /var/lib/kolla/config_files/qemu.conf to /etc/libvirt/qemu.conf
INFO:__main__:Setting permission for /etc/libvirt/qemu.conf
INFO:__main__:Writing out command to execute
Running command: '/usr/sbin/libvirtd --listen'

If applied and re deploy:
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Kolla config strategy set to: COPY_ONCE
INFO:__main__:Copying service configuration files
INFO:__main__:Deleting /etc/libvirt/libvirtd.conf
INFO:__main__:Copying /var/lib/kolla/config_files/libvirtd.conf to /etc/libvirt/libvirtd.conf
INFO:__main__:Setting permission for /etc/libvirt/libvirtd.conf
INFO:__main__:Deleting /etc/libvirt/qemu.conf
INFO:__main__:Copying /var/lib/kolla/config_files/qemu.conf to /etc/libvirt/qemu.conf
INFO:__main__:Setting permission for /etc/libvirt/qemu.conf
INFO:__main__:Writing out command to execute
Running command: '/usr/sbin/libvirtd --listen'

But libvirtd error is always there

[Rich Wellum (richwellum)]

I also see this in stable/pike, running kolla-kubernetes.

[Kevin Zhao (kevin-zhao)]

https://review.openstack.org/#/c/522738/

[Oliver Walsh (owalsh)]

Looks like libvirt TLS is enabled (listen_tls=1 in libvirtd.conf) but certificates have not been setup.

Looking at https://github.com/openstack/kolla-kubernetes/blob/df1f9905adb05e4d261ac78c69f36d3b8c13cf1c/helm/test/devenv/templates/nova-libvirt.yaml#L20

ca_file="" so I assume no attempt is being made to configure TLS. Setting listen_tls=0 will probably get this working.

However setting listen_tcp=1, auth_tcp="none", listen_addr = "0.0.0.0" is a major security issue. Anybody on the network can virsh! This was a CVE in tripleo/rhos-director.

[Rich Wellum (richwellum)]

Some debug.

I added a sleep to nova-libvirt container so I could exec in and run kolla_start manually.

()[root@ubunturich /]# kolla_start
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Kolla config strategy set to: COPY_ONCE
INFO:__main__:Copying service configuration files
INFO:__main__:Deleting /etc/libvirt/libvirtd.conf
INFO:__main__:Copying /var/lib/kolla/config_files/libvirtd.conf to /etc/libvirt/libvirtd.conf
INFO:__main__:Setting permission for /etc/libvirt/libvirtd.conf
INFO:__main__:Deleting /etc/libvirt/qemu.conf
INFO:__main__:Copying /var/lib/kolla/config_files/qemu.conf to /etc/libvirt/qemu.conf
INFO:__main__:Setting permission for /etc/libvirt/qemu.conf
INFO:__main__:Writing out command to execute
Running command: '/usr/sbin/libvirtd --listen'

()[root@ubunturich /]# cat /var/log/kolla/libvirt/libvirtd.log
2017-11-28 04:45:14.274+0000: 9203: info : libvirt version: 3.2.0, package: 14.el7_4.3 (CentOS BuildSystem <http://bugs.centos.org>, 2017-09-07-11:27:44, c1bm.rdu2.centos.org)
2017-11-28 04:45:14.274+0000: 9203: info : hostname: ubunturich
2017-11-28 04:45:14.274+0000: 9203: error : daemonSetupNetworking:554 : unsupported configuration: No server certificate path set to match server key
()[root@ubunturich /]#

[Oliver Walsh (owalsh)]

Doh, I was looking at the test env in comment #4.

Could you include the contents of /etc/libvirtd.conf as it definitely seems related to the TLS config.

[Jeffrey Zhang (jeffrey4l)]

This is fixed by https://review.openstack.org/#/c/492552/ in kolla-ansible.

[Kevin Zhao (kevin-zhao)]

Hi Jeffery, Thanks for reminding
I will port this fix to kolla-kubernetes to verify