neutron-openvswitch-agent fails to talk to rabbit after upgrade completes successfully

Bug #1563437 reported by Marios Andreou
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
puppet-neutron
Fix Released
High
Emilien Macchi
tripleo
Fix Released
High
Emilien Macchi

Bug Description

After a successful overcloud upgrade:

openstack overcloud deploy --templates /usr/share/openstack-tripleo-heat-templates -e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e network_env.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-pacemaker-converge.yaml

and including the (at time of writing still in review) https://review.openstack.org/#/c/298685/ and https://review.openstack.org/#/c/298695/

the overcloud upgrade completes successfully with the services being restarted fine on controllers. However on the compute node I see

Mar 29 15:34:04 overcloud-compute-0.localdomain neutron-openvswitch-agent[1666]: 2016-03-29 15:34:04.832 1666 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server 172.16.2.6:5672 closed the connection. Check login credentials: Socket closed
Mar 29 15:34:04 overcloud-compute-0.localdomain neutron-openvswitch-agent[1666]: 2016-03-29 15:34:04.833 1666 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server 172.16.2.6:5672 closed the connection. Check login credentials: Socket closed
Mar 29 15:34:06 overcloud-compute-0.localdomain neutron-openvswitch-agent[1666]: 2016-03-29 15:34:06.721 1666 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server 172.16.2.6:5672 closed the connection. Check login credentials: Socket closed
Mar 29 15:34:07 overcloud-compute-0.localdomain neutron-openvswitch-agent[1666]: 2016-03-29 15:34:07.196 1666 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server 172.16.2.6:5672 closed the connection. Check login credentials: Socket closed

It is related to https://review.openstack.org/#/c/298695 in the sense that it is to do with the rabbit password being changed on upgrade. It only goes away after I restart the neutron-openvswitch-agent service. As a fix for now I've just tested adding the neutron-openvswitch restart as a post-puppet config for the compute node, i'll submit as review momentarily

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.openstack.org/298865

Changed in tripleo:
status: Triaged → In Progress
Changed in puppet-neutron:
assignee: nobody → Emilien Macchi (emilienm)
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/298946

Changed in tripleo:
assignee: Marios Andreou (marios-b) → Emilien Macchi (emilienm)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-heat-templates (master)

Change abandoned by Marios Andreou (<email address hidden>) on branch: master
Review: https://review.openstack.org/298865

Steven Hardy (shardy)
Changed in tripleo:
importance: Undecided → High
tags: added: mitaka-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.openstack.org/298946
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=bc00cf7e99e68b61450234de1ecfb4aa96aec174
Submitter: Jenkins
Branch: master

commit bc00cf7e99e68b61450234de1ecfb4aa96aec174
Author: Emilien Macchi <email address hidden>
Date: Tue Mar 29 15:25:20 2016 -0400

    compute: restart OVS agent on neutron.conf changes

    Some options in neutron.conf are used bu OVS agent, like logging &
    messaging.
    During the upgrade process, you need to restart the agent if these
    options change.

    We could patch puppet-neutron to add a notify, but the community won't
    like it because Neutron OVS agent is not able to restart gracefully
    until [1] got merged. Until that, we can fix it in TripleO, where we
    suppose Puppet runs happenning during bootstraps and upgrades.

    Later, we'll drop this code from here and move it in puppet-neutron.

    [1] https://review.openstack.org/#/c/297211

    Change-Id: I02b17b66e93331ddfb1a7abd8adff672bc7a32d6
    Closes-Bug: #1563437

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
Ihar Hrachyshka (ihar-hrachyshka) wrote :

Note that the neutron patch mentioned in the commit message landed Newton and will be backported to Liberty included.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/liberty)

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/299303

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/299305

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/mitaka)

Reviewed: https://review.openstack.org/299305
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=03f1ce70779c87742addfee573845e7188400310
Submitter: Jenkins
Branch: stable/mitaka

commit 03f1ce70779c87742addfee573845e7188400310
Author: Emilien Macchi <email address hidden>
Date: Tue Mar 29 15:25:20 2016 -0400

    compute: restart OVS agent on neutron.conf changes

    Some options in neutron.conf are used bu OVS agent, like logging &
    messaging.
    During the upgrade process, you need to restart the agent if these
    options change.

    We could patch puppet-neutron to add a notify, but the community won't
    like it because Neutron OVS agent is not able to restart gracefully
    until [1] got merged. Until that, we can fix it in TripleO, where we
    suppose Puppet runs happenning during bootstraps and upgrades.

    Later, we'll drop this code from here and move it in puppet-neutron.

    [1] https://review.openstack.org/#/c/297211

    Change-Id: I02b17b66e93331ddfb1a7abd8adff672bc7a32d6
    Closes-Bug: #1563437
    (cherry picked from commit bc00cf7e99e68b61450234de1ecfb4aa96aec174)

tags: added: in-stable-mitaka
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/liberty)

Reviewed: https://review.openstack.org/299303
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=537aaab152125498f550a48c76c8d2984bc70df8
Submitter: Jenkins
Branch: stable/liberty

commit 537aaab152125498f550a48c76c8d2984bc70df8
Author: Emilien Macchi <email address hidden>
Date: Tue Mar 29 15:25:20 2016 -0400

    compute: restart OVS agent on neutron.conf changes

    Some options in neutron.conf are used bu OVS agent, like logging &
    messaging.
    During the upgrade process, you need to restart the agent if these
    options change.

    We could patch puppet-neutron to add a notify, but the community won't
    like it because Neutron OVS agent is not able to restart gracefully
    until [1] got merged. Until that, we can fix it in TripleO, where we
    suppose Puppet runs happenning during bootstraps and upgrades.

    Later, we'll drop this code from here and move it in puppet-neutron.

    [1] https://review.openstack.org/#/c/297211

    Change-Id: I02b17b66e93331ddfb1a7abd8adff672bc7a32d6
    Closes-Bug: #1563437
    (cherry picked from commit bc00cf7e99e68b61450234de1ecfb4aa96aec174)
    Conflicts:
     puppet/manifests/overcloud_compute.pp

tags: added: in-stable-liberty
Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/tripleo-heat-templates 5.0.0.0b1

This issue was fixed in the openstack/tripleo-heat-templates 5.0.0.0b1 development milestone.

Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/tripleo-heat-templates 2.1.0

This issue was fixed in the openstack/tripleo-heat-templates 2.1.0 release.

Matt Fischer (mfisch)
Changed in puppet-neutron:
status: New → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

This issue was fixed in the openstack/tripleo-heat-templates 2.1.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.