openstack-ansible rbd python libraries not found

[Tadas Ustinavičius]

Hello, I'm deploying openstack via openstack-ansible.
Install went fine, - I've got a working openstack cluster.
After that i've decided to make some changes (changed VIP network external ip) - after re-running playbooks, I'm getting errors in all nova compute nodes:

2017-01-09 09:15:37.668 23007 ERROR nova.compute.manager [req-4dbdf38f-2786-48cd-a866-18e7f85ac88b - - - - -] No compute node record for host ostack-ibm2
2017-01-09 09:15:37.671 23007 INFO nova.compute.resource_tracker [req-4dbdf38f-2786-48cd-a866-18e7f85ac88b - - - - -] Auditing locally available compute resources for node ostack-ibm2.mgmt.domain.com
2017-01-09 09:15:37.672 23007 ERROR nova.compute.manager [req-4dbdf38f-2786-48cd-a866-18e7f85ac88b - - - - -] Error updating resources for node ostack-ibm2.mgmt.domain.com.
2017-01-09 09:15:37.672 23007 ERROR nova.compute.manager Traceback (most recent call last):
2017-01-09 09:15:37.672 23007 ERROR nova.compute.manager File "/openstack/venvs/nova-14.0.3/lib/python2.7/site-packages/nova/compute/manager.py", line 6415, in update_available_resource_for_node
2017-01-09 09:15:37.672 23007 ERROR nova.compute.manager rt.update_available_resource(context)
2017-01-09 09:15:37.672 23007 ERROR nova.compute.manager File "/openstack/venvs/nova-14.0.3/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 511, in update_available_resource
2017-01-09 09:15:37.672 23007 ERROR nova.compute.manager resources = self.driver.get_available_resource(self.nodename)
2017-01-09 09:15:37.672 23007 ERROR nova.compute.manager File "/openstack/venvs/nova-14.0.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5280, in get_available_resource
2017-01-09 09:15:37.672 23007 ERROR nova.compute.manager disk_info_dict = self._get_local_gb_info()
2017-01-09 09:15:37.672 23007 ERROR nova.compute.manager File "/openstack/venvs/nova-14.0.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4916, in _get_local_gb_info
2017-01-09 09:15:37.672 23007 ERROR nova.compute.manager info = LibvirtDriver._get_rbd_driver().get_pool_info()
2017-01-09 09:15:37.672 23007 ERROR nova.compute.manager File "/openstack/venvs/nova-14.0.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 994, in _get_rbd_driver
2017-01-09 09:15:37.672 23007 ERROR nova.compute.manager rbd_user=CONF.libvirt.rbd_user)
2017-01-09 09:15:37.672 23007 ERROR nova.compute.manager File "/openstack/venvs/nova-14.0.3/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py", line 130, in __init__
2017-01-09 09:15:37.672 23007 ERROR nova.compute.manager raise RuntimeError(_('rbd python libraries not found'))
2017-01-09 09:15:37.672 23007 ERROR nova.compute.manager RuntimeError: rbd python libraries not found
2017-01-09 09:15:37.672 23007 ERROR nova.compute.manager

This happens in I'm using openstack-ansible 14.0.3. Tried to update to I'm using openstack-ansible 14.0.4 and to I'm using openstack-ansible 14.0.5 afterwards. Problem persists. Has anyone had the same problem? Thank you.

[Tadas Ustinavičius (tadas-u)]

It seems that repositories are missing python-cephlibs package:

pip search rados
cradox (1.3.2) - Python libraries for the Ceph librados library with use cython instead of ctypes
radosgw-agent (1.2.7) - Synchronize users and data between radosgw clusters
rgwadmin (1.1.5) - Python Rados Gateway Admin API
radula (0.7.11) - RadosGW client for Ceph S3 storage
thumbor_ceph (1) - Thumbor Ceph Rados extensions
python-cephlibs (0.94.5.post1) - Python bindings for rados and rbd client libraries.
ceph_api (0.4.0) - A wrapper around librados json commands
interlegis.portalmodelo.transparency (1.0rc3) - Tipo de conteúdo do Portal Modelo para publicação de dados estruturados em formato CSV
 pip install python-cephlibs
Ignoring indexes: https://pypi.python.org/simple

Collecting python-cephlibs
  Could not find a version that satisfies the requirement python-cephlibs (from versions: )
No matching distribution found for python-cephlibs

 pip install -vvv python-cephlibs :

https://paste.ubuntu.com/23790869/

[Tadas Ustinavičius (tadas-u)]

After manually copying rados.py to /openstack/venvs/nova-14.0.5/lib/python2.7 this problem disappeared, but I've bumped into new one:

2017-01-16 10:28:56.957 6929 INFO nova.compute.resource_tracker [req-58ed2649-a3a1-4006-8fc1-4f76dd5659df - - - - -] Auditing locally available compute resources for node ostack-ibm1.domain.com
2017-01-16 10:28:56.980 6929 ERROR nova.compute.manager [req-58ed2649-a3a1-4006-8fc1-4f76dd5659df - - - - -] Error updating resources for node ostack-ibm1.mgmt.midas.lt.
2017-01-16 10:28:56.980 6929 ERROR nova.compute.manager Traceback (most recent call last):
2017-01-16 10:28:56.980 6929 ERROR nova.compute.manager File "/openstack/venvs/nova-14.0.5/lib/python2.7/site-packages/nova/compute/manager.py", line 6415, in update_available_resource_for_node
2017-01-16 10:28:56.980 6929 ERROR nova.compute.manager rt.update_available_resource(context)
2017-01-16 10:28:56.980 6929 ERROR nova.compute.manager File "/openstack/venvs/nova-14.0.5/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 511, in update_available_resource
2017-01-16 10:28:56.980 6929 ERROR nova.compute.manager resources = self.driver.get_available_resource(self.nodename)
2017-01-16 10:28:56.980 6929 ERROR nova.compute.manager File "/openstack/venvs/nova-14.0.5/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5310, in get_available_resource
2017-01-16 10:28:56.980 6929 ERROR nova.compute.manager disk_info_dict = self._get_local_gb_info()
2017-01-16 10:28:56.980 6929 ERROR nova.compute.manager File "/openstack/venvs/nova-14.0.5/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4946, in _get_local_gb_info
2017-01-16 10:28:56.980 6929 ERROR nova.compute.manager info = LibvirtDriver._get_rbd_driver().get_pool_info()
2017-01-16 10:28:56.980 6929 ERROR nova.compute.manager File "/openstack/venvs/nova-14.0.5/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py", line 367, in get_pool_info
2017-01-16 10:28:56.980 6929 ERROR nova.compute.manager with RADOSClient(self) as client:
2017-01-16 10:28:56.980 6929 ERROR nova.compute.manager File "/openstack/venvs/nova-14.0.5/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py", line 105, in __init__
2017-01-16 10:28:56.980 6929 ERROR nova.compute.manager self.cluster, self.ioctx = driver._connect_to_rados(pool)
2017-01-16 10:28:56.980 6929 ERROR nova.compute.manager File "/openstack/venvs/nova-14.0.5/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py", line 136, in _connect_to_rados
2017-01-16 10:28:56.980 6929 ERROR nova.compute.manager client.connect()
2017-01-16 10:28:56.980 6929 ERROR nova.compute.manager File "/openstack/venvs/nova-14.0.5/lib/python2.7/rados.py", line 429, in connect
2017-01-16 10:28:56.980 6929 ERROR nova.compute.manager raise make_ex(ret, "error connecting to the cluster")
2017-01-16 10:28:56.980 6929 ERROR nova.compute.manager ObjectNotFound: error connecting to the cluster

Though running python test script which tries to get info from ceph cluster works fine.
I've found, that nova user is unable to read keyring file from /etc/ceph/ directory.
After changing permis...

[Jean-Philippe Evrard (jean-philippe-evrard)]

Please include ansible run log with debug enabled and maybe nova log from startup of nova-compute with debugging enabled.

Please also see our triage conversation in our IRC logs.

[Jean-Philippe Evrard (jean-philippe-evrard)]

See also our conversation here: http://eavesdrop.openstack.org/irclogs/%23openstack-ansible/%23openstack-ansible.2017-01-17.log.html#t2017-01-17T16:20:10

[Tadas Ustinavičius (tadas-u)]

Openstack-ansible debug log:

https://ring.lt/paste/ansible.log

Below are nova debug logs on different error states and how do I fix them:

Nova debug log:
https://paste.ubuntu.com/23820692/

root@ostack-ibm1:~# ln -s /usr/lib/python2.7/dist-packages/rados.x86_64-linux-gnu.so /openstack/venvs/nova-14.0.5/lib/python2.7/site-packages/rados.x86_64-linux-gnu.so

root@ostack-ibm1:~# ln -s /usr/lib/python2.7/dist-packages/rbd.x86_64-linux-gnu.so /openstack/venvs/nova-14.0.5/lib/python2.7/site-packages/rbd.x86_64-linux-gnu.so

Nova debug log:
https://paste.ubuntu.com/23820705/

root@ostack-ibm1:~# ls -la /etc/ceph/|grep key
-rw-r----- 1 root cephkeys 204 Dec 20 12:40 ceph.client.ostack.keyring

root@ostack-ibm1:~# chmod 666 /etc/ceph/ceph.client.ostack.keyring

After this nova compute starts without any problems.

[Launchpad Janitor (janitor)]

[Expired for openstack-ansible because there has been no activity for 60 days.]