Rollback is impossible due to package conflicts

Bug #1364068 reported by Kirill Omelchenko on 2014-09-01
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Critical
Vladimir Kuklin
6.0.x
Critical
Dmitry Ilyin

Bug Description

Scenario:
1. Setup fuel maste node from 5.0-26 iso
2. Create a cluster of next conf:
- HA, CentOS, NovaVlan
- 1 Controller
- 1 Compute
- 1 Cinder
3. Setup (is successful)
4. upgrade master node with the help of the tar ball
(in my case it was fuel-master-upgrade-491)
5. Update the env from 2014.1 to 2014.1.1-5.0.2

Expected:
Update finishes successfully.

Actual:
Update has failed. Check these nodes:
'Untitled (xx:xx)'
During update the next error appeared in puppet log on the controller:
http://paste.openstack.org/show/104276/

More logs are in the attached snapshot.

Kirill Omelchenko (komelchenko) wrote :
Ihor Kalnytskyi (ikalnytskyi) wrote :

This bug should be resolved as a part of this one:

    https://bugs.launchpad.net/mos/+bug/1359705

Please make sure that python-route is know to be updated in the bug above.

Changed in fuel:
milestone: none → 5.1
status: New → Confirmed
Dmitry Mescheryakov (dmitrymex) wrote :

Igor, the bug you are referencing to is already fixed and, to my understanding, the fix is present in ISO #491

Changed in fuel:
importance: High → Critical
Dmitry Mescheryakov (dmitrymex) wrote :

Also, since patching is completely broken, setting importance to 'critical'

Changed in fuel:
assignee: Igor Kalnitsky (ikalnitsky) → Fuel Library Team (fuel-library)
Vladimir Kuklin (vkuklin) wrote :

Kirill, please comment, is this related to rollback? As I can see, puppet is trying to downgrade python-routes, but what is really weird is that it tries to install python routes 1.10.3-2.el6 which I cannot find in our repo.

Tomasz 'Zen' Napierala (tzn) wrote :

Is anyone working on this bug? If so, please assign it accordingly and set appriopriate status.

Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Stanislaw Bogatkin (sbogatkin)
Kirill Omelchenko (komelchenko) wrote :

Vladimir it's just an update of the env, no downgrade took place.

summary: - puppet: Couldn't update python-routes package during cluster update
+ puppet: Couldn't update python-routes package during cluster rollback

In order to mitigate this issue we need to implement pre and post hooks and trigger them depending on the current version and future version of the environment.

Vladimir Kuklin (vkuklin) wrote :

Specifically for this bug we should run

`yum downgrade python-nova openstack-nova-common openstack-nova-conductor openstack-nova-api openstack-nova-console openstack-nova-cert openstack-nova-objectstore openstack-nova-scheduler openstack-nova-novncproxy python-keystone openstack-keystone python-routes`

Fix proposed to branch: master
Review: https://review.openstack.org/118432

Changed in fuel:
assignee: Stanislaw Bogatkin (sbogatkin) → Vladimir Kuklin (vkuklin)
status: Confirmed → In Progress
summary: - puppet: Couldn't update python-routes package during cluster rollback
+ Rollback is impossible due to package conflicts
Changed in fuel:
assignee: Vladimir Kuklin (vkuklin) → Vladimir Sharshov (vsharshov)

Reviewed: https://review.openstack.org/118324
Committed: https://git.openstack.org/cgit/stackforge/fuel-astute/commit/?id=3cb89755ab7d246b103aa3dc613c47df13cf4e80
Submitter: Jenkins
Branch: master

commit 3cb89755ab7d246b103aa3dc613c47df13cf4e80
Author: Vladimir Sharshov <email address hidden>
Date: Tue Sep 2 16:45:23 2014 +0400

    Pre deploy actions

    Change-Id: I2e6b5f33c68124422f49248c07893cabad4c4e4f
    Related-Bug: #1364068

Changed in fuel:
assignee: Vladimir Sharshov (vsharshov) → Vladimir Kuklin (vkuklin)
description: updated
Changed in fuel:
assignee: Vladimir Kuklin (vkuklin) → Vladimir Sharshov (vsharshov)
Changed in fuel:
assignee: Vladimir Sharshov (vsharshov) → Vladimir Kuklin (vkuklin)

Change abandoned by Vladimir Kuklin (<email address hidden>) on branch: master
Review: https://review.openstack.org/119221

Evgeniy L (rustyrobot) on 2014-09-10
tags: added: patching

Change abandoned by Vladimir Kuklin (<email address hidden>) on branch: master
Review: https://review.openstack.org/119105

Changed in fuel:
assignee: Vladimir Kuklin (vkuklin) → Dmitry Ilyin (idv1985)

Change abandoned by Vladimir Kuklin (<email address hidden>) on branch: master
Review: https://review.openstack.org/119209

Change abandoned by Vladimir Kuklin (<email address hidden>) on branch: master
Review: https://review.openstack.org/118432

Changed in fuel:
assignee: Dmitry Ilyin (idv1985) → Vladimir Kuklin (vkuklin)

Reviewed: https://review.openstack.org/120469
Committed: https://git.openstack.org/cgit/stackforge/fuel-astute/commit/?id=a3e5da62af91b99f958ab958161d3dcec09c657b
Submitter: Jenkins
Branch: master

commit a3e5da62af91b99f958ab958161d3dcec09c657b
Author: Dmitry Ilyin <email address hidden>
Date: Wed Sep 10 20:21:51 2014 +0400

    Add hooks to workaround patching bugs

    Add several pre and post patching hooks and tests
    for them.

    * Stop or migrate corosync services from controller node
      and return them after the deployment

    * Stop openstack services and kill pids that was
      not stopped together with their children

    * Remove openstack packages before deployment
      to prevent conflicts during the update and rollback

    Change-Id: Icfc17308d955e55b5e5b31a9c17e0b6d13d8ba10
    Closes-bug: #1364081
    Closes-bug: #1365635
    Closes-bug: #1364068
    Co-Authored-By: Vladimir Sharshov <email address hidden>
    Co-Authored-By: Bogdan Dobrelya <email address hidden>
    Co-Authored-By: Vladimir Kuklin <email address hidden>

Changed in fuel:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/120479
Committed: https://git.openstack.org/cgit/stackforge/fuel-astute/commit/?id=f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13
Submitter: Jenkins
Branch: stable/5.1

commit f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13
Author: Dmitry Ilyin <email address hidden>
Date: Wed Sep 10 20:21:51 2014 +0400

    Add hooks to workaround patching bugs

    Add several pre and post patching hooks and tests
    for them.

    * Stop or migrate corosync services from controller node
      and return them after the deployment

    * Stop openstack services and kill pids that was
      not stopped together with their children

    * Remove openstack packages before deployment
      to prevent conflicts during the update and rollback

    Change-Id: Icfc17308d955e55b5e5b31a9c17e0b6d13d8ba10
    Closes-bug: #1364081
    Closes-bug: #1365635
    Closes-bug: #1364068
    Co-Authored-By: Vladimir Sharshov <email address hidden>
    Co-Authored-By: Bogdan Dobrelya <email address hidden>
    Co-Authored-By: Vladimir Kuklin <email address hidden>

tags: added: release-notes
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers