neutron-openvswitch-agent fails to talk to rabbit after upgrade completes successfully

[Marios Andreou]

After a successful overcloud upgrade:

openstack overcloud deploy --templates /usr/share/openstack-tripleo-heat-templates -e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e network_env.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-pacemaker-converge.yaml

and including the (at time of writing still in review) https://review.openstack.org/#/c/298685/ and https://review.openstack.org/#/c/298695/

the overcloud upgrade completes successfully with the services being restarted fine on controllers. However on the compute node I see

Mar 29 15:34:04 overcloud-compute-0.localdomain neutron-openvswitch-agent[1666]: 2016-03-29 15:34:04.832 1666 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server 172.16.2.6:5672 closed the connection. Check login credentials: Socket closed
Mar 29 15:34:04 overcloud-compute-0.localdomain neutron-openvswitch-agent[1666]: 2016-03-29 15:34:04.833 1666 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server 172.16.2.6:5672 closed the connection. Check login credentials: Socket closed
Mar 29 15:34:06 overcloud-compute-0.localdomain neutron-openvswitch-agent[1666]: 2016-03-29 15:34:06.721 1666 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server 172.16.2.6:5672 closed the connection. Check login credentials: Socket closed
Mar 29 15:34:07 overcloud-compute-0.localdomain neutron-openvswitch-agent[1666]: 2016-03-29 15:34:07.196 1666 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server 172.16.2.6:5672 closed the connection. Check login credentials: Socket closed

It is related to https://review.openstack.org/#/c/298695 in the sense that it is to do with the rabbit password being changed on upgrade. It only goes away after I restart the neutron-openvswitch-agent service. As a fix for now I've just tested adding the neutron-openvswitch restart as a post-puppet config for the compute node, i'll submit as review momentarily

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/298865

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/298946

[OpenStack Infra (hudson-openstack)]

Change abandoned by Marios Andreou (<email address hidden>) on branch: master
Review: https://review.openstack.org/298865

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/298946
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=bc00cf7e99e68b61450234de1ecfb4aa96aec174
Submitter: Jenkins
Branch: master

commit bc00cf7e99e68b61450234de1ecfb4aa96aec174
Author: Emilien Macchi <email address hidden>
Date: Tue Mar 29 15:25:20 2016 -0400

    compute: restart OVS agent on neutron.conf changes

    Some options in neutron.conf are used bu OVS agent, like logging &
    messaging.
    During the upgrade process, you need to restart the agent if these
    options change.

    We could patch puppet-neutron to add a notify, but the community won't
    like it because Neutron OVS agent is not able to restart gracefully
    until [1] got merged. Until that, we can fix it in TripleO, where we
    suppose Puppet runs happenning during bootstraps and upgrades.

    Later, we'll drop this code from here and move it in puppet-neutron.

    [1] https://review.openstack.org/#/c/297211

    Change-Id: I02b17b66e93331ddfb1a7abd8adff672bc7a32d6
    Closes-Bug: #1563437

[Ihar Hrachyshka (ihar-hrachyshka)]

Note that the neutron patch mentioned in the commit message landed Newton and will be backported to Liberty included.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/299303

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/299305

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/299305
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=03f1ce70779c87742addfee573845e7188400310
Submitter: Jenkins
Branch: stable/mitaka

commit 03f1ce70779c87742addfee573845e7188400310
Author: Emilien Macchi <email address hidden>
Date: Tue Mar 29 15:25:20 2016 -0400

    compute: restart OVS agent on neutron.conf changes

    Some options in neutron.conf are used bu OVS agent, like logging &
    messaging.
    During the upgrade process, you need to restart the agent if these
    options change.

    We could patch puppet-neutron to add a notify, but the community won't
    like it because Neutron OVS agent is not able to restart gracefully
    until [1] got merged. Until that, we can fix it in TripleO, where we
    suppose Puppet runs happenning during bootstraps and upgrades.

    Later, we'll drop this code from here and move it in puppet-neutron.

    [1] https://review.openstack.org/#/c/297211

    Change-Id: I02b17b66e93331ddfb1a7abd8adff672bc7a32d6
    Closes-Bug: #1563437
    (cherry picked from commit bc00cf7e99e68b61450234de1ecfb4aa96aec174)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/299303
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=537aaab152125498f550a48c76c8d2984bc70df8
Submitter: Jenkins
Branch: stable/liberty

commit 537aaab152125498f550a48c76c8d2984bc70df8
Author: Emilien Macchi <email address hidden>
Date: Tue Mar 29 15:25:20 2016 -0400

    compute: restart OVS agent on neutron.conf changes

    Some options in neutron.conf are used bu OVS agent, like logging &
    messaging.
    During the upgrade process, you need to restart the agent if these
    options change.

    We could patch puppet-neutron to add a notify, but the community won't
    like it because Neutron OVS agent is not able to restart gracefully
    until [1] got merged. Until that, we can fix it in TripleO, where we
    suppose Puppet runs happenning during bootstraps and upgrades.

    Later, we'll drop this code from here and move it in puppet-neutron.

    [1] https://review.openstack.org/#/c/297211

    Change-Id: I02b17b66e93331ddfb1a7abd8adff672bc7a32d6
    Closes-Bug: #1563437
    (cherry picked from commit bc00cf7e99e68b61450234de1ecfb4aa96aec174)
    Conflicts:
     puppet/manifests/overcloud_compute.pp

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/tripleo-heat-templates 5.0.0.0b1 development milestone.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/tripleo-heat-templates 2.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 2.1.0 release.