br-tun lost ports if openvswitch restart when l2pop enabled

Bug #1332450 reported by Chengli Xu
46
This bug affects 7 people
Affects Status Importance Assigned to Milestone
neutron
Invalid
High
ZongKai LI

Bug Description

When openvswitch restart, ovs agent will reset br-tun, lose all tunnel network related ports/flows, and break all tunnel networks.
If l2 population used, We could maintain all l2 population fdb entries locally and recreate ports/flows . if not, set tunnel_sync = True works.

Tags: ovs
Revision history for this message
Eugene Nikanorov (enikanorov) wrote :

are you talking about openvswitch restart or ovs agent restart?

tags: added: ovs
Changed in neutron:
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/101581

Changed in neutron:
assignee: nobody → Chengli Xu (superekcah)
status: New → In Progress
Revision history for this message
Chengli Xu (superekcah) wrote : Re: br-tun lost ports/flows if openvswitch restart

openvswitch restarts and ovs agent not. There is a patch try to fix those problem (bug 1290486), but I think it's not complete yet.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/102532

Revision history for this message
Chengli Xu (superekcah) wrote : Re: br-tun lost ports/flows if openvswitch restart

As some guys suggested, I split it to two commits, one is for l2-pop used and the other for l2-pop not used

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (master)

Change abandoned by Chengli Xu (<email address hidden>) on branch: master
Review: https://review.openstack.org/102532
Reason: covered by other change

Revision history for this message
Chengli Xu (superekcah) wrote : Re: br-tun lost ports/flows if openvswitch restart

Well, the problem would be partially resolved if l2 pop is not used, by bug 1292105

Revision history for this message
Eugene Nikanorov (enikanorov) wrote :

Reopening as proposed fix was abandoned

Changed in neutron:
status: In Progress → Confirmed
Revision history for this message
Kyle Mestery (mestery) wrote :

I can triage this one a bit more. Also, when you say OVS restart, do you mean "ovs-vswitchd" or OVS DB?

Revision history for this message
Aman Kumar (amank) wrote :

"service openvswitch-switch restart"

It restarts both "ovs-vswitchd" as well as "OVSDB"

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (master)

Change abandoned by Kyle Mestery (<email address hidden>) on branch: master
Review: https://review.openstack.org/101581
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Romil Gupta (romilg)
Changed in neutron:
assignee: Chengli Xu (superekcah) → Romil Gupta (romilg)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/159775

Changed in neutron:
assignee: Romil Gupta (romilg) → Aman Kumar (amank)
status: Confirmed → In Progress
Kyle Mestery (mestery)
Changed in neutron:
milestone: none → liberty-1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (master)

Change abandoned by Kyle Mestery (<email address hidden>) on branch: master
Review: https://review.openstack.org/159775
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Thierry Carrez (ttx)
Changed in neutron:
milestone: liberty-1 → liberty-2
Changed in neutron:
assignee: Aman Kumar (amank) → Romil Gupta (romilg)
Thierry Carrez (ttx)
Changed in neutron:
milestone: liberty-2 → liberty-3
Revision history for this message
Armando Migliaccio (armando-migliaccio) wrote : Re: br-tun lost ports/flows if openvswitch restart

Can't see how this can merge in the next few days.

Changed in neutron:
milestone: liberty-3 → none
Revision history for this message
Armando Migliaccio (armando-migliaccio) wrote :

@Romil: are you actively working on this?

Revision history for this message
Romil Gupta (romilg) wrote :

No, hence unassigned to myself.

Changed in neutron:
assignee: Romil Gupta (romilg) → nobody
Revision history for this message
ZongKai LI (zongkai) wrote :

Hi guys, is anybody still working this?

Revision history for this message
ZongKai LI (zongkai) wrote :

I did a little test, yes, ports on br-tun will disappear after restart openvswitch, but not flows on br-tun.

For losing ports on br-tun, I consider the root reason is here:
https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/l2pop/rpc_manager/l2population_rpc.py#L231
If we only restart openvswitch, neutron ovs-agent will keep ofport cache for ports on br-tun. And that will stop neutron ovs-agent to create tunnel to remove IP, so ports on br-tun will not be re-created.

As we have methods like get_port_name_list and get_port_ofport in https://github.com/openstack/neutron/blob/master/neutron/agent/common/ovs_lib.py, so I think we can add some logic to check whether ofport is in br-tun current existing ofports.

This way is not the best way, but will cause less coding changing.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/219807

Changed in neutron:
assignee: nobody → ZongKai LI (lzklibj)
ZongKai LI (zongkai)
summary: - br-tun lost ports/flows if openvswitch restart
+ br-tun lost ports/flows if openvswitch restart when l2pop enabled
ZongKai LI (zongkai)
summary: - br-tun lost ports/flows if openvswitch restart when l2pop enabled
+ br-tun lost ports if openvswitch restart when l2pop enabled
Revision history for this message
Rossella Sblendido (rossella-o) wrote :

It this bug still an issue after https://review.openstack.org/#/c/182920 ?

Revision history for this message
ZongKai LI (zongkai) wrote :

@Rossella Thanks, the patch you mentioned handles flows on tun_br, so I changed this bug summary from "br-tun lost ports/flows" to "br-tun lost ports".

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (master)

Change abandoned by ZongKai LI (<email address hidden>) on branch: master
Review: https://review.openstack.org/219807
Reason: In https://review.openstack.org/#/c/182920/58 , tun_br.reset_bridge is deprecated, that makes tunnel ports no longer be deleted. So issue #1332450 no more exists.

Revision history for this message
ZongKai LI (zongkai) wrote :

Per https://review.openstack.org/#/c/182920/58 , to keep tunnel connectivity, reset_bridge
isn't used anymore. And by that, tunnel ports will not be deleted anymore. This issue is no longer exist, mark it as invalid.

Changed in neutron:
status: In Progress → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers