HA deployment fails with swift ring builder rebalance timeout

Bug #1260206 reported by Vladimir Kuklin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
Critical
Andrey Korolyov

Bug Description

Looking into logs we see that HA deployment fails with timeout of swift rebalance, which has never happened earlier.

Revision history for this message
Vladimir Kuklin (vkuklin) wrote :
Revision history for this message
Andrey Korolyov (xdeller) wrote :

rebalance container failed *immediately* w/ expiration:

2013-12-12T02:42:55.037912+00:00 debug: (Exec[rebalance_container](provider=posix)) Executing 'swift-ring-builder /etc/swift/container.builder rebalance'
2013-12-12T02:42:55.038569+00:00 debug: Executing 'swift-ring-builder /etc/swift/container.builder rebalance'
2013-12-12T02:47:55.187447+00:00 err: (/Stage[main]/Swift::Ringbuilder/Swift::Ringbuilder::Rebalance[container]/Exec[rebalance_container]/returns) change from notrun to 0 1 failed: Command exceeded timeout at /etc/puppet/modules/swift/manifests/ringbuilder/rebalance.pp:32

buider rebalance took exactly the timeout interval to fail:
2013-12-12T02:48:04.508318+00:00 debug: Executing 'swift-ring-builder /etc/swift/account.builder rebalance'
2013-12-12T02:53:04.662572+00:00 err: (/Stage[main]/Swift::Ringbuilder/Swift::Ringbuilder::Rebalance[account]/Exec[rebalance_account]/returns) change from notrun to 0 1 failed: Command exceeded timeout at /etc/puppet/modules/swift/manifests/ringbuilder/rebalance.pp:32

Looks like swift issue but logs are not enough to determine the cause. Please reproduce and give me an env.

Revision history for this message
Andrey Korolyov (xdeller) wrote :

Sorry, was blind above for first one. Looks like swift issue or very huge regression in the puppet.

Changed in fuel:
importance: High → Medium
importance: Medium → Critical
status: New → Triaged
assignee: nobody → Andrey Korolyov (xdeller)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/61996

Changed in fuel:
assignee: Andrey Korolyov (xdeller) → Vladimir Kuklin (vkuklin)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/61996
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=6d3f0f29d2ae0e6adc4e2bc9e42cb06235d2f499
Submitter: Jenkins
Branch: master

commit 6d3f0f29d2ae0e6adc4e2bc9e42cb06235d2f499
Author: Vladimir Kuklin <email address hidden>
Date: Fri Dec 13 17:52:57 2013 +0400

    Wrap rebalance commands into shell provider

    Change-Id: I5f7761e1e8efaf956d4291de2974d631f33a3bc7
    Closes-bug: #1260206

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
Igor Shishkin (teran) wrote :

This fix does not work.

Changed in fuel:
status: Fix Committed → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/62393

Changed in fuel:
assignee: Vladimir Kuklin (vkuklin) → Andrey Korolyov (xdeller)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/62393
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=f397efe26e0751ae8e959f09d8c2687a3f3fa11d
Submitter: Jenkins
Branch: master

commit f397efe26e0751ae8e959f09d8c2687a3f3fa11d
Author: Andrey Korolyov <email address hidden>
Date: Mon Dec 16 11:56:24 2013 +0400

    increase rebalance timeout just for tests

    Closes-Bug: #1260206

    Depends-on: https://review.openstack.org/#/c/62387/

    Change-Id: Idf2b7b05db53bd149a5a881d0dde259732eea9b7

Changed in fuel:
status: In Progress → Fix Committed
Dmitry Pyzhov (dpyzhov)
Changed in fuel:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.