Packages versions issue after env 5.0 is upgraded to 5.1

[Dennis Dmitriev]

Steps to reproduce:
      1. Deploy cluster 1 with release 2014.1 (5.0) and Ubuntu/HA
      2. Upload and unpack tarball with fuel-5.1 release (build #491) to /var
      3. run upgrade.sh with openstack
      4. Navigate to UI and update cluster 1 to release 2014.1.1-5.0.2
      5. Wait until update has finished and update cluster 1 back to release 2014.1 (5.0)

Expected result: all packages versions are returned to the same versions as it were before updating to 5.0.2

Actual result: after updating back to 5.0 several packages keep installed versions from 5.0.2 :

Package: python-oslo.messaging
Originally on the 5.0: 1.3.0~a9-0ubuntu1
After updating to 5.0.2: 1.3.0-fuel5.0.2~mira3
After updating back to 5.0: 1.3.0-fuel5.0.2~mira3

Package: python-sqlalchemy
Originally on the 5.0: 0.8.2-1~cloud1
After updating to 5.0.2: 0.8.2-ubuntu1
After updating back to 5.0: 0.8.2-ubuntu1

Package: python-sqlalchemy-ext
Originally on the 5.0: 0.8.2-1~cloud1
After updating to 5.0.2: 0.8.2-ubuntu1
After updating back to 5.0: 0.8.2-ubuntu1

Package: python-testtools
Originally on the 5.0: 0.9.32-0ubuntu1~cloud0
After updating to 5.0.2: 0.9.35-0ubuntu1
After updating back to 5.0: 0.9.35-0ubuntu1

May be connected with https://bugs.launchpad.net/mos/+bug/1359705

{"build_id": "2014-09-01_00-01-17", "ostf_sha": "4dcd99cc4bfa19f52d4b87ed321eb84ff03844da", "build_number": "491", "auth_required": true, "api": "1.0", "nailgun_sha": "d25ed02948a8be773e2bd87cfe583ef7be866bb2", "production": "docker", "fuelmain_sha": "109812be3425408dd7be192b5debf109cb1edd4c", "astute_sha": "bc60b7d027ab244039f48c505ac52ab8eb0a990c", "feature_groups": ["mirantis"], "release": "5.1", "fuellib_sha": "2cfa83119ae90b13a5bac6a844bdadfaf5aeb13f"}

[Vladimir Kuklin (vkuklin)]

Dennis, which steps did you follow to do this? It looks like, older manifests and repos were not configured for this deployment and puppet just ran a regular deployment with newer manifests and newer packages. Actually, we do rollback only if deployment fails and then Nailgun switches repositories and manifests to older ones. I am not aware of the test case where we allow user to rollback his environment if update was successful.

[Vladimir Kuklin (vkuklin)]

Also, Dennis, please, always add diagnostic snapshot to bug report

[Dennis Dmitriev (ddmitriev)]

Vladimir, I did all of it through Fuel UI. When the cluster upgraded to 5.0.2, I ran OSTF and then selected "Actions" -> "Update" to 2014.1
Most of packages were returned to their previous versions except these ones.

[Ihor Kalnytskyi (ikalnytskyi)]

I've just re-test and can confirm that old repos and puppets were successfully synced to nodes.
So there's an issue of library side.

[Dennis Dmitriev (ddmitriev)]

Vladimir, I did all of it through Fuel UI. When the cluster upgraded to 5.0.2, I ran OSTF and then selected "Actions" -> "Update" to 2014.1
Most of packages were returned to their previous versions except these ones.

[Vladimir Kuklin (vkuklin)]

Dennis, are you sure that rollback process ended succesfully? Because according to this bug patching rollback is broken now: https://bugs.launchpad.net/fuel/+bug/1364068

[Vladimir Kuklin (vkuklin)]

Also, please attach diagnostic snapshot

[Dennis Dmitriev (ddmitriev)]

As I see the bug https://bugs.launchpad.net/fuel/+bug/1364068 is about CentOS (I haven't finished with testing CentOS yet).
I will attach diagnostic snapshot after reproducing the bug on another environment.

[Dennis Dmitriev (ddmitriev)]

I have made update and rollback for CentOS and Ubuntu clusters.

Steps to reproduce:
        1. Create env with Fuel 5.0.26 (2014.1)
        2. Create CentOS cluster: (simple, neutron/vlan, ceilometer, ceph vol/ceph images, 1 controller+mongo, 4 compute+cinder+ceph)
        3. Store packages versions from cluster nodes to a file (5.0.0.26.orig.sorted.txt for centos and 5.0.0.26-before-update.sorted.txt for ubuntu)
        4. Upgrade env to 5.1 using http://jenkins-product.srt.mirantis.net:8080/job/fuel_master_iso/493/
        5. Update CentOS cluster to 5.0.2
        6. Store packages versions from cluster nodes to a file (5.0.2.versions.sorted.txt for centos and 5.0.2-updated.sorted.txt for ubuntu)
        7. Run OSTF
        8. Make diagnostic snapshot (centos-5.0.2-fuel-snapshot and ubuntu-5.0.2-fuel-snapshot was made)
        9. Rollback cluster to 5.0.0.26
        10. Store packages versions from cluster nodes to a file (5.0.0.26-reverted.versions.sorted.txt for centos and 5.0.0.26-reverted.sorted.txt for ubuntu)
        11. Make diagnostic snapshot (centos-reverted-to-5.0.0.26 and ubuntu-reverted-to-5.0.0.26)

The following are diagnostic snapshots:
centos-5.0.2-fuel-snapshot-2014-09-09_09-13-41.tgz
ubuntu-5.0.2-fuel-snapshot-2014-09-03_14-06-18.tgz
centos-reverted-to-5.0.0.26~ot-2014-09-09_11-43-48.tgz
ubuntu-reverted-to-5.0.0.26~ot-2014-09-03_14-06-18.tgz

The following are packages lists before updating cluster, after updating and after rollback (there is lists for controller and just one of compute nodes since they are the same):
centos-reverted-5.0.0.26.tar.gz
ubuntu-reverted-5.0.0.26.tar.gz

P.S. Updating of CentOS cluster to 5.0.2 failed, it is described in the https://bugs.launchpad.net/fuel/+bug/1367158 . Rollback for CentOS failed too. So on compute nodes some packages were not downgraded.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/120469

[Vladimir Kuklin (vkuklin)]

https://review.openstack.org/#/c/120469/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/120479

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/120469
Committed: https://git.openstack.org/cgit/stackforge/fuel-astute/commit/?id=a3e5da62af91b99f958ab958161d3dcec09c657b
Submitter: Jenkins
Branch: master

commit a3e5da62af91b99f958ab958161d3dcec09c657b
Author: Dmitry Ilyin <email address hidden>
Date: Wed Sep 10 20:21:51 2014 +0400

    Add hooks to workaround patching bugs

    Add several pre and post patching hooks and tests
    for them.

    * Stop or migrate corosync services from controller node
      and return them after the deployment

    * Stop openstack services and kill pids that was
      not stopped together with their children

    * Remove openstack packages before deployment
      to prevent conflicts during the update and rollback

    Change-Id: Icfc17308d955e55b5e5b31a9c17e0b6d13d8ba10
    Closes-bug: #1364081
    Closes-bug: #1365635
    Closes-bug: #1364068
    Co-Authored-By: Vladimir Sharshov <email address hidden>
    Co-Authored-By: Bogdan Dobrelya <email address hidden>
    Co-Authored-By: Vladimir Kuklin <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/120479
Committed: https://git.openstack.org/cgit/stackforge/fuel-astute/commit/?id=f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13
Submitter: Jenkins
Branch: stable/5.1

commit f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13
Author: Dmitry Ilyin <email address hidden>
Date: Wed Sep 10 20:21:51 2014 +0400

    Add hooks to workaround patching bugs

    Add several pre and post patching hooks and tests
    for them.

    * Stop or migrate corosync services from controller node
      and return them after the deployment

    * Stop openstack services and kill pids that was
      not stopped together with their children

    * Remove openstack packages before deployment
      to prevent conflicts during the update and rollback

    Change-Id: Icfc17308d955e55b5e5b31a9c17e0b6d13d8ba10
    Closes-bug: #1364081
    Closes-bug: #1365635
    Closes-bug: #1364068
    Co-Authored-By: Vladimir Sharshov <email address hidden>
    Co-Authored-By: Bogdan Dobrelya <email address hidden>
    Co-Authored-By: Vladimir Kuklin <email address hidden>

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/121194

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/5.0
Review: https://review.openstack.org/123140

[OpenStack Infra (hudson-openstack)]

Change abandoned by Dmitry Ilyin (<email address hidden>) on branch: master
Review: https://review.openstack.org/121194

[OpenStack Infra (hudson-openstack)]

Change abandoned by Dmitry Ilyin (<email address hidden>) on branch: stable/5.0
Review: https://review.openstack.org/123140