nova-scheduler upgrade from newton to ocata doesn't work

[Emilien Macchi]

When upgrading TripleO from Newton to Ocata, nova-scheduler fails to stop and start at step 2.

Looking at the logs:
http://logs.openstack.org/64/437364/2/check/gate-tripleo-ci-centos-7-multinode-upgrades-nv/1823021/logs/subnode-2/syslog.txt.gz#_Feb_24_11_24_56

It seems like nova-scheduler wasn't stopped by the Ansible task at step 2, though packages were updated.
Since RabbitMQ is stopped at step2, there is no way Nova services can work at the end of step 2, therefore, we need to make sure Nova is actually stopped, which doesn't seem the case.

We should upgrade Nova packages at step 3 like other services.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/437958

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/438058

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/437958
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=5593877817b332bd7a650f051cff5f5988ab1e54
Submitter: Jenkins
Branch: master

commit 5593877817b332bd7a650f051cff5f5988ab1e54
Author: Sofer Athlan-Guyot <email address hidden>
Date: Fri Feb 24 17:01:35 2017 +0100

    Upgrade nova-api/scheduler/conductor packages at step3 not step2.

    The nova-api, nova-scheduler nova-conductor packages are updated during
    step2. The package upgrade trigger a restart of the service which fails
    and is constantly retried by systemd:

        Feb 24 12:34:24 centos-7-2-node-rax-iad-7463943-440549 systemd[1]: Failed to start OpenStack Nova Scheduler Server.
        Feb 24 12:34:24 centos-7-2-node-rax-iad-7463943-440549 systemd[1]: Unit openstack-nova-scheduler.service entered failed state.
        Feb 24 12:34:24 centos-7-2-node-rax-iad-7463943-440549 systemd[1]: openstack-nova-scheduler.service failed.
        Feb 24 12:34:24 centos-7-2-node-rax-iad-7463943-440549 systemd[1]: openstack-nova-scheduler.service holdoff time over, scheduling restart.

    We eventually reach timeout. We use
    https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/tripleo-packages.yaml#L44-L46
    to upgrade existing packages.

    Add a note to the README.rst to make people aware of the general upgrade
    done at step3 and limit its usage to new package for individual service.

    Change-Id: I13b51bcfe0c98034944613f7e1c3f0168cd4de76
    Closes-Bug: #1667728

[OpenStack Infra (hudson-openstack)]

Change abandoned by Athlan-Guyot sofer (<email address hidden>) on branch: stable/ocata
Review: https://review.openstack.org/438058
Reason: redo the cherry pick after the merge.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/438058
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=577ea9b01fb62c7f59f7e47b1f9959c52163ce08
Submitter: Jenkins
Branch: stable/ocata

commit 577ea9b01fb62c7f59f7e47b1f9959c52163ce08
Author: Sofer Athlan-Guyot <email address hidden>
Date: Fri Feb 24 17:01:35 2017 +0100

    Upgrade nova-api/scheduler/conductor packages at step3 not step2.

    The nova-api, nova-scheduler nova-conductor packages are updated during
    step2. The package upgrade trigger a restart of the service which fails
    and is constantly retried by systemd:

        Feb 24 12:34:24 centos-7-2-node-rax-iad-7463943-440549 systemd[1]: Failed to start OpenStack Nova Scheduler Server.
        Feb 24 12:34:24 centos-7-2-node-rax-iad-7463943-440549 systemd[1]: Unit openstack-nova-scheduler.service entered failed state.
        Feb 24 12:34:24 centos-7-2-node-rax-iad-7463943-440549 systemd[1]: openstack-nova-scheduler.service failed.
        Feb 24 12:34:24 centos-7-2-node-rax-iad-7463943-440549 systemd[1]: openstack-nova-scheduler.service holdoff time over, scheduling restart.

    We eventually reach timeout. We use
    https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/tripleo-packages.yaml#L44-L46
    to upgrade existing packages.

    Add a note to the README.rst to make people aware of the general upgrade
    done at step3 and limit its usage to new package for individual service.

    Change-Id: I13b51bcfe0c98034944613f7e1c3f0168cd4de76
    Closes-Bug: #1667728
    (cherry picked from commit 5593877817b332bd7a650f051cff5f5988ab1e54)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 6.0.0.0rc2 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 7.0.0.0b1 development milestone.