[9.2] improve ceph update procedure

Bug #1661260 reported by Denis Meltsaykin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Alexey Stupnikov

Bug Description

Since we obtain updated ceph packages earlier than do the real ceph update, there is a small chance that a service restart occurs before the correct update but having new packages. This can lead to a huge rebalance of the ceph cluster and might cause data loss. We need to improve this procedure in order to decrease the data loss probability.

Changed in fuel:
importance: Critical → High
milestone: 9.x-updates → 9.2-mu-2
status: New → Confirmed
Changed in fuel:
assignee: MOS Maintenance (mos-maintenance) → Alexey Stupnikov (astupnikov)
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to mos/mos-docs (stable/9.2)

Fix proposed to branch: stable/9.2
Change author: Alexey Stupnikov <email address hidden>
Review: https://review.fuel-infra.org/35628

Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to mos/mos-docs (stable/9.2)

Reviewed: https://review.fuel-infra.org/35628
Submitter: Olena Logvinova <email address hidden>
Branch: stable/9.2

Commit: 8be814a372e1fc589cda57ebba3fa725b2cd0d99
Author: Alexey Stupnikov <email address hidden>
Date: Wed Jun 14 17:28:19 2017

Improve 9.2 update process

Our documentation contain suboptimal update step sequence:
specific ceph packages are updated after all installed packages
were updated on slave nodes.

Another issue is late ceph restart that could lead to a broken
ceph cluster if customer will accidently restart node to solve
other update issues.

I have moved update ceph step earlier and changed the command to
restart ceph services right after package update.

Change-Id: Ib44e10256c41a2b182b80c8a3d8709af8454c47f
Closes-bug: #1661260

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
Alexey Stupnikov (astupnikov) wrote :

QA team: this fix doesn't contain code to be released, so please test the instruction on Fuel9.1 deployment with ceph and remove 9.2-mu2 nomination if everything will work as it should.

Revision history for this message
Ilya Bumarskov (ibumarskov) wrote :

New update procedure doesn't work because new ubuntu repositories will not added before step "Update the environment: fuel2 update --env <ENV_ID> install --repos mos9.2-updates"

Changed in fuel:
status: Fix Committed → Confirmed
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/mos-updates (9.0/mitaka)

Fix proposed to branch: 9.0/mitaka
Change author: Alexey Stupnikov <email address hidden>
Review: https://review.fuel-infra.org/35727

Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to mos/mos-docs (stable/9.2)

Fix proposed to branch: stable/9.2
Change author: Alexey Stupnikov <email address hidden>
Review: https://review.fuel-infra.org/35728

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to openstack/mos-updates (9.0/mitaka)

Reviewed: https://review.fuel-infra.org/35727
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0/mitaka

Commit: a50834082edabfba7bda9b6c3a00cdfd3ade84bc
Author: Alexey Stupnikov <email address hidden>
Date: Wed Jun 21 18:13:31 2017

Improve mos9.2-updates repo integration

Changed ceph update procedure to improve support of provided repo
with new packages. Here is a list of changes:

- Configured default repo with updated packages (mos9.2-updates).
- Added a separate play to remove added repo (if it was configured
  and added).

Change-Id: Id65c2bc7661058a80ec9be020109ce5bd2293344
Partial-bug: #1661260

Revision history for this message
Alexey Stupnikov (astupnikov) wrote :

QA team: please test mos-updates package with https://review.fuel-infra.org/#/c/35728/ mos-docs fix

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
Ilya Bumarskov (ibumarskov) wrote :

Done. New procedure works.

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to mos/mos-docs (stable/9.2)

Reviewed: https://review.fuel-infra.org/35728
Submitter: Olena Logvinova <email address hidden>
Branch: stable/9.2

Commit: 125111dd41277b925c27a04dacb57f0a80d408c1
Author: Alexey Stupnikov <email address hidden>
Date: Wed Jun 21 18:19:06 2017

Use mos9.2-updates repo during ceph update

We have to use additional repo during ceph update process, since
we haven't got correct sources configuration at this step.

Change-Id: I3a98461b6907654a1d57d0205a28d32637f67ec7
Closes-bug: #1661260

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-qa (stable/mitaka)

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/482084

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-qa (stable/mitaka)

Reviewed: https://review.openstack.org/482084
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=463b97563e2e5e28b1500eb3c80ec4d0477dfea7
Submitter: Jenkins
Branch: stable/mitaka

commit 463b97563e2e5e28b1500eb3c80ec4d0477dfea7
Author: ibumarskov <email address hidden>
Date: Mon Jul 10 14:01:00 2017 +0400

    Update procedure was changed (9.x)

    Sequence of update procedure was changed. Step with update ceph was
    moved before step 'update cluster'.
    Change-Id: Ie26b913d3a5a5eba2c8ae0ae6e9143de28fadb5b
    Related-bug: #1661260

tags: added: in-stable-mitaka
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.