Ceph::Nova_compute: missing dependency on libvirt service => broken deployment (undeterministic)

Bug #1478901 reported by Tatyanka
32
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
Critical
Vasyl Saienko

Bug Description

Steps:
Build verification test for Ubuntu

Case: ha_neutron_ceph
1. Create cluster
2. Add 3 nodes with controller and ceph OSD roles
3. Add 1 node with ceph OSD roles
4. Add 2 nodes with compute and ceph OSD roles
5. Deploy the cluster
6. Check ceph status

Actual result:
Deployment fail
with http://paste.openstack.org/show/406047/

VERSION:
  feature_groups:
    - experimental
  production: "docker"
  release: "7.0"
  openstack_version: "2015.1.0-7.0"
  api: "1.0"
  build_number: "97"
  build_id: "2015-07-28_06-50-15"
  nailgun_sha: "fd7392bb8d9d640631d9805bf1fbe3124e98db04"
  python-fuelclient_sha: "c2f8b78686de9dceca8bb9c3f655a7fbbd1375a4"
  fuel-agent_sha: "2a65f11c10b0aeb5184247635a19740fc3edde21"
  astute_sha: "34e0493afa22999c4a07d3198ceb945116ab7932"
  fuel-library_sha: "0c29bbc15cf981a4d7d2c6ef7d9c25ba9f828873"
  fuel-ostf_sha: "9e32b35e9a774eec41b86cd5a6c63b71a3efa4bd"
  fuelmain_sha: "921918a3bd3d278431f35ad917989e46b0c24100"

Revision history for this message
Tatyanka (tatyana-leontovich) wrote :
Changed in fuel:
status: New → Confirmed
assignee: Fuel Library Team (fuel-library) → Oleksiy Molchanov (omolchanov)
Revision history for this message
Oleksiy Molchanov (omolchanov) wrote :

Seems that libvirtd failed w/o any reason. I am marking this as incomplete until not reproduced.

Changed in fuel:
status: Confirmed → Incomplete
Revision history for this message
Tatyanka (tatyana-leontovich) wrote :
Changed in fuel:
status: Incomplete → Confirmed
Revision history for this message
Dennis Dmitriev (ddmitriev) wrote :
Revision history for this message
Vasyl Saienko (vsaienko) wrote :

It looks like bug in debian package, that were previously fixed, but reproduced again on Ubuntu system

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=704012

root@node-6:/etc/libvirt# strace /usr/sbin/libvirtd returns https://paste.mirantis.net/show/836/

but if export LC_ALL=C, no error exist https://paste.mirantis.net/show/837/

Changed in fuel:
assignee: Oleksiy Molchanov (omolchanov) → MOS Linux (mos-linux)
Revision history for this message
Aleksander Mogylchenko (amogylchenko) wrote :

Severity of the bug you mentioned is minor. Why this is critical?

Revision history for this message
Aleksander Mogylchenko (amogylchenko) wrote :

And why this bug was only found now, if libvirt was merged a long time ago?

Revision history for this message
Vasyl Saienko (vsaienko) wrote :

it breaks deployment so it should be critical. I think that we faced with it only now, because we didn't have CI for a long time for 7.0 with Kilo packages

Revision history for this message
Aleksander Mogylchenko (amogylchenko) wrote :

It has clear workaround, so no critical.

Changed in fuel:
importance: Critical → High
Revision history for this message
Aleksander Mogylchenko (amogylchenko) wrote :

Please explain how it can block the deployment, if it was reproduced on ubuntu bvt tests (build #104), and the same tests passed in next build (build #106)?

Changed in fuel:
status: Confirmed → Incomplete
assignee: MOS Linux (mos-linux) → Vasyl Saienko (vsaienko)
importance: High → Critical
Revision history for this message
Alexei Sheplyakov (asheplyakov) wrote :

open("/usr/lib/locale/uk_UA.UTF-8/LC_IDENTIFICATION", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/uk_UA.utf8/LC_IDENTIFICATION", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/uk_UA/LC_IDENTIFICATION", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/uk.UTF-8/LC_IDENTIFICATION", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/uk.utf8/LC_IDENTIFICATION", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

This is purely configuration problem: the user has specified a locale which hasn't been generated.
Please make sure the environment variables don't get polluted or generate the missing locale.

Changed in fuel:
status: Incomplete → Invalid
Vasyl Saienko (vsaienko)
summary: - [stage bvt] Deployment failed with
- /Stage[main]/Ceph::Nova_compute/Exec[Set Ceph RBD secret for
- Nova]/returns) change from notrun to 0 failed
+ ceph is trying to access to virsh when libvirtd is not runnig
Changed in fuel:
status: Invalid → Confirmed
Revision history for this message
Alexei Sheplyakov (asheplyakov) wrote : Re: ceph is trying to access to virsh when libvirtd is not runnig
summary: - ceph is trying to access to virsh when libvirtd is not runnig
+ Ceph::Nova_compute: missing dependency on libvirt service => broken
+ deployment (undeterministic)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/208409

Changed in fuel:
status: Confirmed → In Progress
Vasyl Saienko (vsaienko)
Changed in fuel:
importance: Critical → High
Revision history for this message
Vasyl Saienko (vsaienko) wrote :

rising to Critical, it is failed on stagging mirror: http://jenkins-product.srt.mirantis.net:8080/job/7.0.staging.ubuntu.bvt_2/111

Changed in fuel:
importance: High → Critical
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/208409
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=12f37b0ddcfdee6509860f80cc39e8e4ef2eba51
Submitter: Jenkins
Branch: master

commit 12f37b0ddcfdee6509860f80cc39e8e4ef2eba51
Author: vsaienko <email address hidden>
Date: Mon Aug 3 13:25:35 2015 +0300

    add dependency for libvirtd service in 'ceph::nova_compute'

     * 'Set Ceph RBD secret for Nova' should be
       called only if libvirtd is running

    Closes-Bug: #1478901

    Change-Id: Iad71a20abc77fe29d5d3d4b4caf6886212168b60

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
Tatyanka (tatyana-leontovich) wrote :

verified on 144 iso

Changed in fuel:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.