Ceph Rados libraries not installing using OSA (Liberty)

[Grant Morley]

Using OSA Liberty for our openstack setup doesn't seem to work when you try to use Ceph as the backend for Nova compute hosts. We are using the standard set up from "http://docs.openstack.org/developer/openstack-ansible/liberty/install-guide/"

When You have Ceph enabled and try to launch an instance you are presented with the following error from the nova-compute logs:

2016-10-24 09:13:33.935 57025 INFO nova.compute.manager [req-2d61f59a-7dc2-4550-b52e-e977ada22e29 41c60f65ae914681b6a6ca27a42ff780 324844c815084205995aff10b03a85e1 - - -] [instance: 057fdf1d-b0f0-4153-8144-df9758bdca29] Starting instance...
2016-10-24 09:13:34.342 57025 WARNING oslo_config.cfg [-] Option "username" from group "neutron" is deprecated. Use option "user-name" from group "neutron".
2016-10-24 09:13:34.550 57025 INFO nova.virt.libvirt.driver [req-2d61f59a-7dc2-4550-b52e-e977ada22e29 41c60f65ae914681b6a6ca27a42ff780 324844c815084205995aff10b03a85e1 - - -] [instance: 057fdf1d-b0f0-4153-8144-df9758bdca29] Creating image
2016-10-24 09:13:34.551 57025 ERROR nova.compute.manager [req-2d61f59a-7dc2-4550-b52e-e977ada22e29 41c60f65ae914681b6a6ca27a42ff780 324844c815084205995aff10b03a85e1 - - -] [instance: 057fdf1d-b0f0-4153-8144-df9758bdca29] Instance failed to spawn
2016-10-24 09:13:34.551 57025 ERROR nova.compute.manager [instance: 057fdf1d-b0f0-4153-8144-df9758bdca29] Traceback (most recent call last):
2016-10-24 09:13:34.551 57025 ERROR nova.compute.manager [instance: 057fdf1d-b0f0-4153-8144-df9758bdca29] File "/openstack/venvs/nova-12.0.16/lib/python2.7/site-packages/nova/compute/manager.py", line 2156, in _build_resources
2016-10-24 09:13:34.551 57025 ERROR nova.compute.manager [instance: 057fdf1d-b0f0-4153-8144-df9758bdca29] yield resources
2016-10-24 09:13:34.551 57025 ERROR nova.compute.manager [instance: 057fdf1d-b0f0-4153-8144-df9758bdca29] File "/openstack/venvs/nova-12.0.16/lib/python2.7/site-packages/nova/compute/manager.py", line 2009, in _build_and_run_instance
2016-10-24 09:13:34.551 57025 ERROR nova.compute.manager [instance: 057fdf1d-b0f0-4153-8144-df9758bdca29] block_device_info=block_device_info)
2016-10-24 09:13:34.551 57025 ERROR nova.compute.manager [instance: 057fdf1d-b0f0-4153-8144-df9758bdca29] File "/openstack/venvs/nova-12.0.16/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2527, in spawn
2016-10-24 09:13:34.551 57025 ERROR nova.compute.manager [instance: 057fdf1d-b0f0-4153-8144-df9758bdca29] admin_pass=admin_password)
2016-10-24 09:13:34.551 57025 ERROR nova.compute.manager [instance: 057fdf1d-b0f0-4153-8144-df9758bdca29] File "/openstack/venvs/nova-12.0.16/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2939, in _create_image
2016-10-24 09:13:34.551 57025 ERROR nova.compute.manager [instance: 057fdf1d-b0f0-4153-8144-df9758bdca29] backend = image('disk')
2016-10-24 09:13:34.551 57025 ERROR nova.compute.manager [instance: 057fdf1d-b0f0-4153-8144-df9758bdca29] File "/openstack/venvs/nova-12.0.16/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2884, in image
2016-10-24 09:13:34.551 57025 ERROR nova.compute.manager [instance: 057fdf1d-b0f0-4153-8144-df9758bdca29] fname + suffix, image_type)
2016-10-24 09:13:34.551 57025 ERROR nova.compute.manager [instance: 057fdf1d-b0f0-4153-8144-df9758bdca29] File "/openstack/venvs/nova-12.0.16/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 967, in image
2016-10-24 09:13:34.551 57025 ERROR nova.compute.manager [instance: 057fdf1d-b0f0-4153-8144-df9758bdca29] return backend(instance=instance, disk_name=disk_name)
2016-10-24 09:13:34.551 57025 ERROR nova.compute.manager [instance: 057fdf1d-b0f0-4153-8144-df9758bdca29] File "/openstack/venvs/nova-12.0.16/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 748, in __init__
2016-10-24 09:13:34.551 57025 ERROR nova.compute.manager [instance: 057fdf1d-b0f0-4153-8144-df9758bdca29] rbd_user=self.rbd_user)
2016-10-24 09:13:34.551 57025 ERROR nova.compute.manager [instance: 057fdf1d-b0f0-4153-8144-df9758bdca29] File "/openstack/venvs/nova-12.0.16/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py", line 117, in __init__
2016-10-24 09:13:34.551 57025 ERROR nova.compute.manager [instance: 057fdf1d-b0f0-4153-8144-df9758bdca29] raise RuntimeError(_('rbd python libraries not found'))
2016-10-24 09:13:34.551 57025 ERROR nova.compute.manager [instance: 057fdf1d-b0f0-4153-8144-df9758bdca29] RuntimeError: rbd python libraries not found

Looking at the python libraries it appears that the rbd and rados libraries do not exist which in turn means they cannot be linked to the venv:

source /openstack/venvs/nova-12.0.16/bin/activate
(nova-12.0.16)root@compute-3:~# pip freeze....

python-cinderclient==1.5.0
python-editor==0.5
python-glanceclient==1.2.0
python-keystoneclient==1.7.2
python-memcached==1.57
python-mimeparse==0.1.4
python-neutronclient==3.1.1
python-novaclient==2.35.0
pytz==2015.7
PyYAML==3.12
repoze.lru==0.6
requests==2.8.1
retrying==1.3.3
rfc3986==0.3.1
Routes==2.2
simplejson==3.8.1

As you can see there are no rados or rbd packages installed. If you try and run a manual "pip install rbd" you get the following error:

Ignoring indexes: https://pypi.python.org/simple
Collecting rbd
  Could not find a version that satisfies the requirement rbd (from versions: )
No matching distribution found for rbd

[Jesse Pretorius (jesse-pretorius)]

Odd - for Mitaka we symlink the python libs from the ceph packages into the venv: https://github.com/openstack/openstack-ansible/blob/stable/mitaka/playbooks/roles/ceph_client/tasks/ceph_install_python_libs.yml

But I can't seem to find that for Liberty - and I'm pretty sure we did the same thing. It's been a very long time since I looked at anything in Liberty though.

[Matt Thompson (mattt416)]

The linking also happens in liberty, but it happens in the glance/cinder/nova roles instead. For example:

https://github.com/openstack/openstack-ansible/blob/liberty/playbooks/roles/os_nova/tasks/nova_compute_kvm_install.yml#L108-L123

Grant, some questions:

1. Do you have /openstack/venvs/nova-12.0.16/lib/python2.7/site-packages/{rados.py,rbd.py} symlinks and if so do they point to existing files?
2. Do you have either cinder_backends_rbd_inuse|bool or nova_libvirt_images_rbd_pool set in your overrides file?
3. Do you have either nova_virt_type == 'kvm' or nova_virt_type == 'qemu' set in your overrides file?

Thanks!

--Matt

[Grant Morley (adogrant)]

Hi thanks for getting back to me.

Because I couldn't get round the issue I ended up, upgrading to Mitaka which does seem to have helped and we are no longer getting the issue.

In answer to your question Matt,

1. We didn't actually have any symlinks in /openstack/venvs/nova-12.0.16/lib/python2.7/site-packages/ for any ceph related libs which was a bit odd.

2. We had nova_libvirt_images_rbd_pool set in the overrides file.

3. We had nova_virt_type == 'kvm in the overrides file.

Sorry I know this isn't a great deal of help now we have upgraded to Mitaka, however as we were a bit stuck and the pending EOL of Liberty, we thought it would be best to upgrade.

[Launchpad Janitor (janitor)]

[Expired for openstack-ansible because there has been no activity for 60 days.]