upgrades to pike: cinder-volumes fails to start

[Emilien Macchi]

When upgrading TripleO from Ocata to Pike, cinder-volume fails to start and retrieve volumes:

http://logs.openstack.org/00/461000/13/check/gate-tripleo-ci-centos-7-scenario002-multinode-upgrades-nv/3ba8e4e/logs/subnode-2/var/log/cinder/volume.txt.gz#_2017-05-18_13_56_30_480

[Gorka Eguileor (gorka)]

Looking at the log look like there are 3 issues, the first two are probably transient and won't be an issue, while the last one looks like a problem:

- Missing AMQP exchange 'cinder-scheduler_fanout' [1]
- No DB connection [2]
- DB error [3] probably caused by not running DB migrations prior to running the Cinder service because migration 99 seems not to be in place [4].

[1] http://logs.openstack.org/00/461000/13/check/gate-tripleo-ci-centos-7-scenario002-multinode-upgrades-nv/3ba8e4e/logs/subnode-2/var/log/cinder/volume.txt.gz#_2017-05-18_13_49_52_251
[2] http://logs.openstack.org/00/461000/13/check/gate-tripleo-ci-centos-7-scenario002-multinode-upgrades-nv/3ba8e4e/logs/subnode-2/var/log/cinder/volume.txt.gz#_2017-05-18_13_50_14_557
[3] http://logs.openstack.org/00/461000/13/check/gate-tripleo-ci-centos-7-scenario002-multinode-upgrades-nv/3ba8e4e/logs/subnode-2/var/log/cinder/volume.txt.gz#_2017-05-18_13_56_30_476
[4] https://github.com/openstack/cinder/blob/master/cinder/db/sqlalchemy/migrate_repo/versions/099_add_connection_info_to_attachment.py
~

[Alan Bishop (alan-bishop)]

I see cinder's db sync occurring, but looking at subnode's /var/log/messages [1] I don't see any signs of cinder-volume's upgrade tasks ever running. Also, the cinder-volume log also shows signs of the service starting just once, whereas cinder-api and cinder-scheduler both show signs of stopping, and then being started again after the upgrade.

I suspect the problem is related to cinder-volume being run under pacemaker, but I don't understand the details of how pacemaker upgrades are supposed to work.

[1] http://logs.openstack.org/67/462667/3/check/gate-tripleo-ci-centos-7-multinode-upgrades-nv/51759ea/logs/subnode-2/var/log/messages

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/467280
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=c4e3bbe039135f32f0e198365e704b3dbfd00290
Submitter: Jenkins
Branch: master

commit c4e3bbe039135f32f0e198365e704b3dbfd00290
Author: Alan Bishop <email address hidden>
Date: Tue May 23 10:42:24 2017 -0400

    Handle upgrading cinder-volume under pacemaker

    Add upgrade tasks for cinder-volume when it's controlled by pacemaker:

    o Stop the service before the entire pacemaker cluster is stopped.
      This ensures the service is stopped before infrastructure services
      (e.g. rabbitmq) go away.
    o Migrate the cinder DB prior to restarting the service. This covers
      the situation when puppet-cinder (who otherwise would handle the db
      sync) isn't managing the service.
    o Start the service after the rest of the pacemaker cluster has been
      started.

    Closes-Bug: #1691851
    Change-Id: I5874ab862964fadb68320d5c4de39b20f53dc25c

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/470354

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/470354
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=862585bbbcf5a8b9bd171ada44cda90d3600f2d5
Submitter: Jenkins
Branch: stable/ocata

commit 862585bbbcf5a8b9bd171ada44cda90d3600f2d5
Author: Alan Bishop <email address hidden>
Date: Tue May 23 10:42:24 2017 -0400

    Handle upgrading cinder-volume under pacemaker

    Add upgrade tasks for cinder-volume when it's controlled by pacemaker:

    o Stop the service before the entire pacemaker cluster is stopped.
      This ensures the service is stopped before infrastructure services
      (e.g. rabbitmq) go away.
    o Migrate the cinder DB prior to restarting the service. This covers
      the situation when puppet-cinder (who otherwise would handle the db
      sync) isn't managing the service.
    o Start the service after the rest of the pacemaker cluster has been
      started.

    Closes-Bug: #1691851
    Change-Id: I5874ab862964fadb68320d5c4de39b20f53dc25c
    (cherry picked from commit c4e3bbe039135f32f0e198365e704b3dbfd00290)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 7.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 6.2.0 release.