ovs plugin agent terminates due to non-existent br-ex in some cases

Bug #1190117 reported by Michael Chapman
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
puppet-neutron
Fix Released
Undecided
Michael Chapman

Bug Description

Currently, the external bridge is brought up after the quantum agent service via these lines in puppet-quantum / manifests / agents / ovs.pp :

    quantum::plugins::ovs::bridge{ $bridge_mappings:
      require => Service['quantum-plugin-ovs-service'],
    }
    quantum::plugins::ovs::port{ $bridge_uplinks:
      require => Service['quantum-plugin-ovs-service'],
    }

While testing this, I see the following order:

notice: /Stage[main]/Quantum::Agents::Ovs/Service[quantum-plugin-ovs-service]/ensure: ensure changed 'stopped' to 'running'
notice: /Stage[main]/Quantum::Agents::Ovs/Service[quantum-plugin-ovs-service]: Triggered 'refresh' from 30 events
notice: /Stage[main]/Quantum::Agents::Ovs/Vs_bridge[br-int]/ensure: created
notice: /Stage[main]/Quantum::Agents::L3/Service[quantum-l3]: Triggered 'refresh' from 25 events

notice: /Stage[main]/Quantum::Agents::Ovs/Quantum::Plugins::Ovs::Bridge[default:br-ex]/Vs_bridge[br-ex]/ensure: created

notice: /Stage[main]/Quantum::Agents::Ovs/Quantum::Plugins::Ovs::Port[br-ex:eth2]/Vs_port[eth2]/ensure: created
notice: /Stage[main]/Quantum::Agents::Ovs/Vs_bridge[br-tun]/ensure: created

Depending on how quickly the intermediate resources are resolved, br-ex may or may not be active when the plugin agent service looks for it, and will fail and exit depending on whether it finds it or not.

The 'require' should actually be 'before' - the bridge needs to be active before the agent starts running.

Changed in puppet-openstack:
assignee: nobody → Michael Chapman (michaeltchapman)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to puppet-quantum (master)

Fix proposed to branch: master
Review: https://review.openstack.org/32820

Changed in puppet-openstack:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to puppet-quantum (master)

Reviewed: https://review.openstack.org/32820
Committed: http://github.com/stackforge/puppet-quantum/commit/f237b6895767728d731a630d06cb61d5b47c8a5f
Submitter: Jenkins
Branch: master

commit f237b6895767728d731a630d06cb61d5b47c8a5f
Author: Michael Chapman <email address hidden>
Date: Thu Jun 13 10:06:56 2013 +1000

    Creates external bridge before starting ovs agent

    The agent is currently involved in a race condition
    where br-ex is created after it, but will still be
    created before the agent searches for it in cases
    where the intermediate resources are resolved quickly.

    The agent will terminate immediately if the bridge
    is not found.

    Change-Id: Ie93289ce156f5c1d76bae7616b573107d16dd6aa
    Fixes: bug #1190117

Changed in puppet-openstack:
status: In Progress → Fix Committed
Mathieu Gagné (mgagne)
Changed in puppet-openstack:
status: Fix Committed → Fix Released
Mathieu Gagné (mgagne)
affects: puppet-openstack → puppet-neutron
Changed in puppet-neutron:
milestone: none → 2.1.0
Mathieu Gagné (mgagne)
Changed in puppet-neutron:
milestone: 2.1.0 → 2.1.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.