Port information (binding:host_id) not updated for network:router_gateway after qRouter failover

Bug #1694337 reported by Felipe Reyes on 2017-05-29
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu Cloud Archive
Undecided
Unassigned
Mitaka
Undecided
Unassigned
Newton
Undecided
Unassigned
Ocata
Undecided
Unassigned
neutron
Undecided
Felipe Reyes
neutron (Ubuntu)
Undecided
Unassigned
Xenial
Undecided
Felipe Reyes
Yakkety
Undecided
Felipe Reyes
Zesty
Undecided
Felipe Reyes

Bug Description

[Impact]

When using l3 ha and a router agent fails over, the interface holding the network:router_gateway interface does not get its property binding:host_id updated to reflect where the keepalived moved the router.

[Steps to reproduce]

0) Deploy a cloud with l3ha enabled
  - If familiar with juju, it's possible to use this bundle http://paste.ubuntu.com/24707730/ , but the deployment tool is not relevant

1) Once it's deployed, configure it and create a router see https://docs.openstack.org/mitaka/networking-guide/deploy-lb-ha-vrrp.html )
  - This is the script used during the troubleshooting
-----8<------
#!/bin/bash -x

source novarc # admin

neutron net-create ext-net --router:external True --provider:physical_network physnet1 --provider:network_type flat

neutron subnet-create ext-net 10.5.0.0/16 --name ext-subnet --allocation-pool start=10.5.254.100,end=10.5.254.199 --disable-dhcp --gateway 10.5.0.1 --dns-nameserver 10.5.0.3

keystone tenant-create --name demo 2>/dev/null
keystone user-role-add --user admin --tenant demo --role Admin 2>/dev/null

export TENANT_ID_DEMO=$(keystone tenant-list | grep demo | awk -F'|' '{print $2}' | tr -d ' ' 2>/dev/null )

neutron net-create demo-net --tenant-id ${TENANT_ID_DEMO} --provider:network_type vxlan

env OS_TENANT_NAME=demo neutron subnet-create demo-net 192.168.1.0/24 --name demo-subnet --gateway 192.168.1.1
env OS_TENANT_NAME=demo neutron router-create demo-router
env OS_TENANT_NAME=demo neutron router-interface-add demo-router demo-subnet
env OS_TENANT_NAME=demo neutron router-gateway-set demo-router ext-net

# verification
neutron net-list
neutron l3-agent-list-hosting-router demo-router
neutron router-port-list demo-router
----- 8< -------

2) Kill the associated master keepalived process for the router
    ps aux | grep keepalived | grep $ROUTER_ID
    kill $PID

3) Wait until "neutron l3-agent-list-hosting-router demo-router" shows the other host as active
4) Check the binding:host_id property for the interfaces of the router
    for ID in `neutron port-list --device-id $ROUTER_ID | tail -n +4 | head -n -1| awk -F' ' '{print $2}' `; do neutron port-show $ID ; done

Expected results:

The interface where the device_owner is network:router_gateway has its property binding:host_id set to where the keepalived process is master

Actual result:

The binding:host_id is never updated, it stays set with the value obtainer during the creation of the port.

[Regression Potential]
- This patch changes the UPDATE query to the port bindings in the database, a possible regression will express as failures in the query or binding:host_id property outdated.

[Other Info]

The patches for zesty and yakkety are a direct backport from stable/ocata and stable/newton respectively. The patch for xenial is NOT merged in stable/xenial because it's already EOL.

Felipe Reyes (freyes) on 2017-05-29
tags: added: l3-ha sts
Changed in neutron:
assignee: nobody → Felipe Reyes (freyes)
status: New → In Progress

Reviewed: https://review.openstack.org/466414
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d8334b41d2c5bcd4916347d20008b1538d48b0ef
Submitter: Jenkins
Branch: master

commit d8334b41d2c5bcd4916347d20008b1538d48b0ef
Author: Felipe Reyes <email address hidden>
Date: Fri May 19 17:13:52 2017 -0400

    Update the host_id for network:router_gateway interfaces

    The ports owned by a router_gateway need to get its host_id property
    updated during the failover of a router. Otherwise the port connected
    to the external network will always have its host_id set to the value
    obtained during creation.

    Change-Id: I5eca20e3cc64d7a9e52b0556a3cadd29eb4c821d
    Closes-Bug: 1694337

Changed in neutron:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/470045
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f6b3d25c6ef335ff891030b8e34c1d27f45b896c
Submitter: Jenkins
Branch: stable/newton

commit f6b3d25c6ef335ff891030b8e34c1d27f45b896c
Author: Felipe Reyes <email address hidden>
Date: Fri May 19 17:13:52 2017 -0400

    Update the host_id for network:router_gateway interfaces

    The ports owned by a router_gateway need to get its host_id property
    updated during the failover of a router. Otherwise the port connected
    to the external network will always have its host_id set to the value
    obtained during creation.

    Change-Id: I5eca20e3cc64d7a9e52b0556a3cadd29eb4c821d
    Closes-Bug: 1694337
    (cherry picked from commit d8334b41d2c5bcd4916347d20008b1538d48b0ef)

tags: added: in-stable-newton

Reviewed: https://review.openstack.org/470044
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=6964c903175994c25b4ffb44204e31672ee2ea8b
Submitter: Jenkins
Branch: stable/ocata

commit 6964c903175994c25b4ffb44204e31672ee2ea8b
Author: Felipe Reyes <email address hidden>
Date: Fri May 19 17:13:52 2017 -0400

    Update the host_id for network:router_gateway interfaces

    The ports owned by a router_gateway need to get its host_id property
    updated during the failover of a router. Otherwise the port connected
    to the external network will always have its host_id set to the value
    obtained during creation.

    Change-Id: I5eca20e3cc64d7a9e52b0556a3cadd29eb4c821d
    Closes-Bug: 1694337
    (cherry picked from commit d8334b41d2c5bcd4916347d20008b1538d48b0ef)

tags: added: in-stable-ocata

This issue was fixed in the openstack/neutron 11.0.0.0b2 development milestone.

tags: added: neutron-proactive-backport-potential
Brian Haley (brian-haley) wrote :

Already in stable/ocata and stable/newton

tags: removed: neutron-proactive-backport-potential
Felipe Reyes (freyes) on 2017-06-21
Changed in neutron (Ubuntu):
assignee: nobody → Felipe Reyes (freyes)
Eric Desrochers (slashd) on 2017-06-21
Changed in neutron (Ubuntu Xenial):
assignee: nobody → Felipe Reyes (freyes)
Changed in neutron (Ubuntu Yakkety):
assignee: nobody → Felipe Reyes (freyes)
Changed in neutron (Ubuntu Zesty):
assignee: nobody → Felipe Reyes (freyes)
Felipe Reyes (freyes) on 2017-06-22
Changed in neutron (Ubuntu):
status: New → Fix Released
assignee: Felipe Reyes (freyes) → nobody
Felipe Reyes (freyes) wrote :

Artful already has the fix for this, so marking it as 'fix released'

Changed in neutron (Ubuntu Xenial):
status: New → In Progress
Changed in neutron (Ubuntu Yakkety):
status: New → In Progress
Changed in neutron (Ubuntu Zesty):
status: New → In Progress
Felipe Reyes (freyes) wrote :
description: updated
Felipe Reyes (freyes) wrote :
description: updated
Felipe Reyes (freyes) on 2017-06-22
tags: added: sts-sru-needed
James Page (james-page) on 2017-07-05
Changed in neutron (Ubuntu Yakkety):
status: In Progress → Won't Fix
Felipe Reyes (freyes) wrote :

I will rebase my patch on top of neutron 9.4.0

Felipe Reyes (freyes) wrote :

yakkety debdiff rebased on top of neutron 9.4.0

James Page (james-page) wrote :

Uploaded to newton-staging

Hello Felipe, or anyone else affected,

Accepted neutron into newton-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:newton-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-newton-needed to verification-newton-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-newton-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-newton-needed
James Page (james-page) wrote :

Uploaded for zesty-proposed

James Page (james-page) wrote :

Uploaded for xenial-proposed

Brian Murray (brian-murray) wrote :

Hello Felipe, or anyone else affected,

Accepted neutron into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:10.0.2-0ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-zesty to verification-done-zesty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-zesty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in neutron (Ubuntu Zesty):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-zesty
Changed in neutron (Ubuntu Xenial):
status: In Progress → Fix Committed
tags: added: verification-needed-xenial
Brian Murray (brian-murray) wrote :

Hello Felipe, or anyone else affected,

Accepted neutron into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:8.4.0-0ubuntu4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

James Page (james-page) wrote :

Hello Felipe, or anyone else affected,

Accepted neutron into ocata-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ocata-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ocata-needed to verification-ocata-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ocata-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-ocata-needed
James Page (james-page) wrote :

Hello Felipe, or anyone else affected,

Accepted neutron into mitaka-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:mitaka-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-mitaka-needed to verification-mitaka-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-mitaka-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-mitaka-needed
Felipe Reyes (freyes) wrote :

I verified the packages available in -proposed (xenial-mitaka, xenial-newton, xenial-ocata, zesty-ocata) and they do fix the issue following the test case describe in the SRU, and there were no regressions detected (launching/destroying VMs showed no issues, also networks manipulation works as expected).

I'm here attaching the evidence of my verification for each setup.

Felipe Reyes (freyes) wrote :
Felipe Reyes (freyes) wrote :
Felipe Reyes (freyes) wrote :
Felipe Reyes (freyes) wrote :
tags: added: verification-done verification-done-xenial verification-done-zesty verification-mitaka-done verification-newton-done verification-ocata-done
removed: sts-sru-needed verification-mitaka-needed verification-needed verification-needed-xenial verification-needed-zesty verification-newton-needed verification-ocata-needed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package neutron - 2:10.0.2-0ubuntu1.1

---------------
neutron (2:10.0.2-0ubuntu1.1) zesty; urgency=medium

  * d/p/Update-the-host_id-for-network-router_gateway-interf.patch:
    keep the router's gateway interface updated when keepalived fails
    over (LP: #1694337).

 -- Felipe Reyes <email address hidden> Wed, 21 Jun 2017 18:01:36 -0400

Changed in neutron (Ubuntu Zesty):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package neutron - 2:8.4.0-0ubuntu4

---------------
neutron (2:8.4.0-0ubuntu4) xenial; urgency=medium

  * d/p/Update-the-host_id-for-network-router_gateway-interf.patch:
    keep the router's gateway interface updated when keepalived
    fails over (LP: #1694337).

 -- Felipe Reyes <email address hidden> Tue, 25 Jul 2017 17:50:16 +0100

Changed in neutron (Ubuntu Xenial):
status: Fix Committed → Fix Released
James Page (james-page) wrote :

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

James Page (james-page) wrote :

This bug was fixed in the package neutron - 2:8.4.0-0ubuntu4~cloud0
---------------

 neutron (2:8.4.0-0ubuntu4~cloud0) trusty-mitaka; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 neutron (2:8.4.0-0ubuntu4) xenial; urgency=medium
 .
   * d/p/Update-the-host_id-for-network-router_gateway-interf.patch:
     keep the router's gateway interface updated when keepalived
     fails over (LP: #1694337).

James Page (james-page) wrote :

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

James Page (james-page) wrote :

This bug was fixed in the package neutron - 2:9.4.0-0ubuntu1.1~cloud0
---------------

 neutron (2:9.4.0-0ubuntu1.1~cloud0) xenial-newton; urgency=medium
 .
   * d/p/Update-the-host_id-for-network-router_gateway-interf.patch:
     keep the router's gateway interface updated when keepalived fails over
     (LP: #1694337).

James Page (james-page) wrote :

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

James Page (james-page) wrote :

This bug was fixed in the package neutron - 2:10.0.2-0ubuntu1.1~cloud0
---------------

 neutron (2:10.0.2-0ubuntu1.1~cloud0) xenial-ocata; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 neutron (2:10.0.2-0ubuntu1.1) zesty; urgency=medium
 .
   * d/p/Update-the-host_id-for-network-router_gateway-interf.patch:
     keep the router's gateway interface updated when keepalived fails
     over (LP: #1694337).

This issue was fixed in the openstack/neutron 9.4.1 release.

This issue was fixed in the openstack/neutron 10.0.3 release.

Changed in cloud-archive:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers