[10.0-community] Failed in deployment of ceph_rados_gw

[Vyacheslav Vakhlyuev]

Detailed bug description:
The ceph_rados_gw deployment failed.
Excerpt from sys_test.log:
"2016-07-28 20:07:13,174 - DEBUG __init__.py:62 -- Done: get_task with result: {u'status': u'running', u'name': u'deploy', u'cluster': 1, u'parent_id': None, u'result': {}, u'progress': 86, u'message': None, u'id': 5, u'uuid': u'964ac46a-165d-4326-be48-170d48811a17'}
2016-07-28 20:07:43,182 - DEBUG __init__.py:56 -- Calling: get_task with args: ([<class 'fuelweb_test.models.nailgun_client.NailgunClient'>(0x7f5095af07d0), url:None], 5) {}
2016-07-28 20:07:43,269 - DEBUG __init__.py:62 -- Done: get_task with result: {u'status': u'error', u'name': u'deploy', u'cluster': 1, u'parent_id': None, u'result': {}, u'progress': 100, u'message': u'Deployment has failed. All nodes are finished. Failed tasks: Task[ceph-radosgw/1], Task[ceph-radosgw/4], Task[ceph-radosgw/5] Stopping the deployment process!', u'id': 5, u'uuid': u'964ac46a-165d-4326-be48-170d48811a17'}
2016-07-28 20:07:43,270 - DEBUG __init__.py:56 -- Calling: get_task with args: ([<class 'fuelweb_test.models.nailgun_client.NailgunClient'>(0x7f5095af07d0), url:None], 5) {}
2016-07-28 20:07:43,287 - DEBUG __init__.py:62 -- Done: get_task with result: {u'status': u'error', u'name': u'deploy', u'cluster': 1, u'parent_id': None, u'result': {}, u'progress': 100, u'message': u'Deployment has failed. All nodes are finished. Failed tasks: Task[ceph-radosgw/1], Task[ceph-radosgw/4], Task[ceph-radosgw/5] Stopping the deployment process!', u'id': 5, u'uuid': u'964ac46a-165d-4326-be48-170d48811a17'}
2016-07-28 20:07:43,291 - INFO fuel_web_client.py:1381 -- Task finished. Took 3213.53384519 seconds.
 status error
 name deploy
 cluster 1
 parent_id None
 result {}
 progress 100
 message Deployment has failed. All nodes are finished. Failed tasks: Task[ceph-radosgw/1], Task[ceph-radosgw/4], Task[ceph-radosgw/5] Stopping the deployment process!
 id 5
 uuid 964ac46a-165d-4326-be48-170d48811a17
2016-07-28 20:07:43,293 - DEBUG __init__.py:62 -- Done: task_wait with result: {u'status': u'error', u'name': u'deploy', u'cluster': 1, u'parent_id': None, u'result': {}, u'progress': 100, u'message': u'Deployment has failed. All nodes are finished. Failed tasks: Task[ceph-radosgw/1], Task[ceph-radosgw/4], Task[ceph-radosgw/5] Stopping the deployment process!', u'id': 5, u'uuid': u'964ac46a-165d-4326-be48-170d48811a17'}
2016-07-28 20:07:43,308 - ERROR __init__.py:67 -- assert_task_success raised: AssertionError("Task 'deploy' has incorrect status. error != ready, 'Deployment has failed. All nodes are finished. Failed tasks: Task[ceph-radosgw/1], Task[ceph-radosgw/4], Task[ceph-radosgw/5] Stopping the deployment process!'",)
Traceback: Traceback (most recent call last):
  File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/fuelweb_test/__init__.py", line 60, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/fuelweb_test/models/fuel_web_client.py", line 335, in assert_task_success
    task["name"], task['status'], 'ready', _message(task)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/asserts.py", line 55, in assert_equal
    raise ASSERTION_ERROR(message)
AssertionError: Task 'deploy' has incorrect status. error != ready, 'Deployment has failed. All nodes are finished. Failed tasks: Task[ceph-radosgw/1], Task[ceph-radosgw/4], Task[ceph-radosgw/5] Stopping the deployment process!'
"

Reproducibility:
The issue was reproduced in 3 consecutive builds (Jenkins jobs)

Jenkins jobs:
https://ci.fuel-infra.org/job/10.0-community.main.ubuntu.bvt_2/420/
https://ci.fuel-infra.org/job/10.0-community.main.ubuntu.bvt_2/419/
https://ci.fuel-infra.org/job/10.0-community.main.ubuntu.bvt_2/418/

[Vyacheslav Vakhlyuev (vvakhlyuev)]

Seems that builds #147,#148,#149 are also affected by this issue.

MOS 10.

https://product-ci.infra.mirantis.net/job/10.0-mitaka.main.ubuntu.bvt_2/147/
https://product-ci.infra.mirantis.net/job/10.0-mitaka.main.ubuntu.bvt_2/148/
https://product-ci.infra.mirantis.net/job/10.0-mitaka.main.ubuntu.bvt_2/149/

Please find snapshots attached

[Alex Schultz (alex-schultz)]

10.109.31.4 2016-07-29T08:57:31.433470 err: (/Stage[main]/Osnailyfacter::Ceph::Radosgw/Ceph::Key[client.radosgw.gateway]/File[/etc/ceph/client.radosgw.gateway]/owner) change from root to ceph failed: Could not find user ceph
10.109.31.4 2016-07-29T08:57:31.436247 err: Could not find group ceph

[Alex Schultz (alex-schultz)]

caused by https://review.openstack.org/#/c/348221/

[Dmitry Burmistrov (dmburmistrov)]

I'm working on it here - https://review.openstack.org/#/c/348931/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/348931
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=822a55230c207ad9cb914905fc213f0fa4b6a698
Submitter: Jenkins
Branch: master

commit 822a55230c207ad9cb914905fc213f0fa4b6a698
Author: dmburmistrov <email address hidden>
Date: Fri Jul 29 18:32:37 2016 +0300

    ensure ceph user/group

    blueprint: mos-xenial

    Closes-bug: #1607304
    Closes-bug: #1607757

    Change-Id: Ieb4879f33235d19d63ee11ad1491f79c56837152

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-library 10.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-library 10.0.0 release.