placement is not being properly restarted in grenade (pike to master)

[Chris Dent]

When the placement service is supposed to restart in grenade (pike to master) it doesn't actually restart:

http://logs.openstack.org/93/385693/84/check/legacy-grenade-dsvm-neutron-multinode-live-migration/9fa93e0/logs/grenade.sh.txt.gz#_2017-12-05_00_08_01_111

This leads to issues with new microversions not being available:

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Unacceptable%20version%20header%3A%201.14%5C%22

This is a latent bug that was revealed, at least in part, by efried's (correct) changes in https://review.openstack.org/#/c/524263/

It looks like a bad assumption is being made somewhere in the handling of the systemd unit files: a 'start' when it is already started is success, but does not restart (thus new code is not loaded).

We can probably fix this by using the 'restart' command instead of 'start':

 restart PATTERN...
           Restart one or more units specified on the command line. If the units are not running yet, they will be started.

Adding grenade and devstack as relate projects as the fix is presumably in devstack itself.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/525533

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/525534

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/525605

[Sylvain Bauza (sylvain-bauza)]

Not sure it's something needing to modify Nova...

[Chris Dent (cdent)]

Yeah, thanks bauzas, it's definitely a problem to be fixed in grenade, but I wanted to be sure to report the problem in nova because that's where the gate failures are showing up.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Chris Dent (<email address hidden>) on branch: master
Review: https://review.openstack.org/525533
Reason: wrong fix, the right fix is in grenade but having some issues there: I7f2158aeaef82a47e11c6e29675e542023fff4be

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/525789

[OpenStack Infra (hudson-openstack)]

Change abandoned by Chris Dent (<email address hidden>) on branch: master
Review: https://review.openstack.org/525534
Reason: wrong fix

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/526035

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/525789
Committed: https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=1489b9e7101fef6270eea008917b08f3da91a771
Submitter: Zuul
Branch: master

commit 1489b9e7101fef6270eea008917b08f3da91a771
Author: Chris Dent <email address hidden>
Date: Tue Dec 5 23:46:58 2017 +0000

    Move remove_uwsgi_config to cleanup_placement

    It had been in stop_placement, but we don't want it there: the old
    side of grenade needs to call that but should not remove the uwsgi
    configuration when doing so. It is configuration, after all.

    Change-Id: Iee763adf7895145d97b184924896db3f1f48a015
    Partial-Bug: #1736385

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/526035
Committed: https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=4a85d5d6e040b08e085f9f5752de90a2346df6d1
Submitter: Zuul
Branch: stable/pike

commit 4a85d5d6e040b08e085f9f5752de90a2346df6d1
Author: Chris Dent <email address hidden>
Date: Tue Dec 5 23:46:58 2017 +0000

    Move remove_uwsgi_config to cleanup_placement

    It had been in stop_placement, but we don't want it there: the old
    side of grenade needs to call that but should not remove the uwsgi
    configuration when doing so. It is configuration, after all.

    Change-Id: Iee763adf7895145d97b184924896db3f1f48a015
    Partial-Bug: #1736385

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/525605
Committed: https://git.openstack.org/cgit/openstack-dev/grenade/commit/?id=b7dbc632b6a0fe2b625096b3a53576e746150b0a
Submitter: Zuul
Branch: master

commit b7dbc632b6a0fe2b625096b3a53576e746150b0a
Author: Chris Dent <email address hidden>
Date: Tue Dec 5 13:21:19 2017 +0000

    Shutdown placement at the end of the base run

    Placement API was not being shutdown at the end of the OLD target,
    thus when it was being started at the beginning of the NEW target it
    was not actually getting new code. A call to stop_placement is added.

    In addition, make sure placement is installed in upgrade.sh so that
    any new stuff is in place.

    The depends-on is to a devstack change which makes sure that
    lib/placement does not call remove_uwsgi_config when stop_placement is
    called. Without this, there's no config file for the
    devstack@placement-api systemd unit to run, and there is an immediate
    exit.

    Change-Id: I7f2158aeaef82a47e11c6e29675e542023fff4be
    Depends-On: Iee763adf7895145d97b184924896db3f1f48a015
    Closes-Bug: #1736385

[Dr. Jens Harbott (j-harbott)]

Looks like the fix in devstack has been abandoned and the issue resolved via a patch in grenade, please update if this is incorrect.