Ceph::Nova_compute: missing dependency on libvirt service => broken deployment (undeterministic)

[Tatyanka]

Steps:
Build verification test for Ubuntu

Case: ha_neutron_ceph
1. Create cluster
2. Add 3 nodes with controller and ceph OSD roles
3. Add 1 node with ceph OSD roles
4. Add 2 nodes with compute and ceph OSD roles
5. Deploy the cluster
6. Check ceph status

Actual result:
Deployment fail
with http://paste.openstack.org/show/406047/

VERSION:
  feature_groups:
    - experimental
  production: "docker"
  release: "7.0"
  openstack_version: "2015.1.0-7.0"
  api: "1.0"
  build_number: "97"
  build_id: "2015-07-28_06-50-15"
  nailgun_sha: "fd7392bb8d9d640631d9805bf1fbe3124e98db04"
  python-fuelclient_sha: "c2f8b78686de9dceca8bb9c3f655a7fbbd1375a4"
  fuel-agent_sha: "2a65f11c10b0aeb5184247635a19740fc3edde21"
  astute_sha: "34e0493afa22999c4a07d3198ceb945116ab7932"
  fuel-library_sha: "0c29bbc15cf981a4d7d2c6ef7d9c25ba9f828873"
  fuel-ostf_sha: "9e32b35e9a774eec41b86cd5a6c63b71a3efa4bd"
  fuelmain_sha: "921918a3bd3d278431f35ad917989e46b0c24100"

[Oleksiy Molchanov (omolchanov)]

Seems that libvirtd failed w/o any reason. I am marking this as incomplete until not reproduced.

[Tatyanka (tatyana-leontovich)]

reproduced on bvt again
https://bugs.launchpad.net/fuel/+bug/1479238
http://jenkins-product.srt.mirantis.net:8080/view/7.0/job/7.0.ubuntu.bvt_2/104/

[Dennis Dmitriev (ddmitriev)]

Reproduced on swarm: http://jenkins-product.srt.mirantis.net:8080/job/7.0.system_test.ubuntu.thread_4/43/testReport/%28root%29/ceph_rados_gw/

[Vasyl Saienko (vsaienko)]

It looks like bug in debian package, that were previously fixed, but reproduced again on Ubuntu system

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=704012

root@node-6:/etc/libvirt# strace /usr/sbin/libvirtd returns https://paste.mirantis.net/show/836/

but if export LC_ALL=C, no error exist https://paste.mirantis.net/show/837/

[Aleksander Mogylchenko (amogylchenko)]

Severity of the bug you mentioned is minor. Why this is critical?

[Aleksander Mogylchenko (amogylchenko)]

And why this bug was only found now, if libvirt was merged a long time ago?

[Vasyl Saienko (vsaienko)]

it breaks deployment so it should be critical. I think that we faced with it only now, because we didn't have CI for a long time for 7.0 with Kilo packages

[Aleksander Mogylchenko (amogylchenko)]

It has clear workaround, so no critical.

[Aleksander Mogylchenko (amogylchenko)]

Please explain how it can block the deployment, if it was reproduced on ubuntu bvt tests (build #104), and the same tests passed in next build (build #106)?

[Alexei Sheplyakov (asheplyakov)]

open("/usr/lib/locale/uk_UA.UTF-8/LC_IDENTIFICATION", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/uk_UA.utf8/LC_IDENTIFICATION", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/uk_UA/LC_IDENTIFICATION", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/uk.UTF-8/LC_IDENTIFICATION", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/uk.utf8/LC_IDENTIFICATION", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

This is purely configuration problem: the user has specified a locale which hasn't been generated.
Please make sure the environment variables don't get polluted or generate the missing locale.

[Alexei Sheplyakov (asheplyakov)]

The dependency on running libvirt daemon is missing in

https://github.com/stackforge/fuel-library/blob/master/deployment/puppet/ceph/manifests/nova_compute.pp#L14-L18

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/208409

[Vasyl Saienko (vsaienko)]

rising to Critical, it is failed on stagging mirror: http://jenkins-product.srt.mirantis.net:8080/job/7.0.staging.ubuntu.bvt_2/111

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/208409
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=12f37b0ddcfdee6509860f80cc39e8e4ef2eba51
Submitter: Jenkins
Branch: master

commit 12f37b0ddcfdee6509860f80cc39e8e4ef2eba51
Author: vsaienko <email address hidden>
Date: Mon Aug 3 13:25:35 2015 +0300

    add dependency for libvirtd service in 'ceph::nova_compute'

     * 'Set Ceph RBD secret for Nova' should be
       called only if libvirtd is running

    Closes-Bug: #1478901

    Change-Id: Iad71a20abc77fe29d5d3d4b4caf6886212168b60

[Tatyanka (tatyana-leontovich)]

verified on 144 iso