Port information (binding:host_id) not updated for network:router_gateway after qRouter failover

[Felipe Reyes]

[Impact]

When using l3 ha and a router agent fails over, the interface holding the network:router_gateway interface does not get its property binding:host_id updated to reflect where the keepalived moved the router.

[Steps to reproduce]

0) Deploy a cloud with l3ha enabled
  - If familiar with juju, it's possible to use this bundle http://paste.ubuntu.com/24707730/ , but the deployment tool is not relevant

1) Once it's deployed, configure it and create a router see https://docs.openstack.org/mitaka/networking-guide/deploy-lb-ha-vrrp.html )
  - This is the script used during the troubleshooting
-----8<------
#!/bin/bash -x

source novarc # admin

neutron net-create ext-net --router:external True --provider:physical_network physnet1 --provider:network_type flat

neutron subnet-create ext-net 10.5.0.0/16 --name ext-subnet --allocation-pool start=10.5.254.100,end=10.5.254.199 --disable-dhcp --gateway 10.5.0.1 --dns-nameserver 10.5.0.3

keystone tenant-create --name demo 2>/dev/null
keystone user-role-add --user admin --tenant demo --role Admin 2>/dev/null

export TENANT_ID_DEMO=$(keystone tenant-list | grep demo | awk -F'|' '{print $2}' | tr -d ' ' 2>/dev/null )

neutron net-create demo-net --tenant-id ${TENANT_ID_DEMO} --provider:network_type vxlan

env OS_TENANT_NAME=demo neutron subnet-create demo-net 192.168.1.0/24 --name demo-subnet --gateway 192.168.1.1
env OS_TENANT_NAME=demo neutron router-create demo-router
env OS_TENANT_NAME=demo neutron router-interface-add demo-router demo-subnet
env OS_TENANT_NAME=demo neutron router-gateway-set demo-router ext-net

# verification
neutron net-list
neutron l3-agent-list-hosting-router demo-router
neutron router-port-list demo-router
----- 8< -------

2) Kill the associated master keepalived process for the router
    ps aux | grep keepalived | grep $ROUTER_ID
    kill $PID

3) Wait until "neutron l3-agent-list-hosting-router demo-router" shows the other host as active
4) Check the binding:host_id property for the interfaces of the router
    for ID in `neutron port-list --device-id $ROUTER_ID | tail -n +4 | head -n -1| awk -F' ' '{print $2}' `; do neutron port-show $ID ; done

Expected results:

The interface where the device_owner is network:router_gateway has its property binding:host_id set to where the keepalived process is master

Actual result:

The binding:host_id is never updated, it stays set with the value obtainer during the creation of the port.

[Regression Potential]
- This patch changes the UPDATE query to the port bindings in the database, a possible regression will express as failures in the query or binding:host_id property outdated.

[Other Info]

The patches for zesty and yakkety are a direct backport from stable/ocata and stable/newton respectively. The patch for xenial is NOT merged in stable/xenial because it's already EOL.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/466414
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d8334b41d2c5bcd4916347d20008b1538d48b0ef
Submitter: Jenkins
Branch: master

commit d8334b41d2c5bcd4916347d20008b1538d48b0ef
Author: Felipe Reyes <email address hidden>
Date: Fri May 19 17:13:52 2017 -0400

    Update the host_id for network:router_gateway interfaces

    The ports owned by a router_gateway need to get its host_id property
    updated during the failover of a router. Otherwise the port connected
    to the external network will always have its host_id set to the value
    obtained during creation.

    Change-Id: I5eca20e3cc64d7a9e52b0556a3cadd29eb4c821d
    Closes-Bug: 1694337

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/470044

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/470045

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/470045
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f6b3d25c6ef335ff891030b8e34c1d27f45b896c
Submitter: Jenkins
Branch: stable/newton

commit f6b3d25c6ef335ff891030b8e34c1d27f45b896c
Author: Felipe Reyes <email address hidden>
Date: Fri May 19 17:13:52 2017 -0400

    Update the host_id for network:router_gateway interfaces

    The ports owned by a router_gateway need to get its host_id property
    updated during the failover of a router. Otherwise the port connected
    to the external network will always have its host_id set to the value
    obtained during creation.

    Change-Id: I5eca20e3cc64d7a9e52b0556a3cadd29eb4c821d
    Closes-Bug: 1694337
    (cherry picked from commit d8334b41d2c5bcd4916347d20008b1538d48b0ef)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/470044
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=6964c903175994c25b4ffb44204e31672ee2ea8b
Submitter: Jenkins
Branch: stable/ocata

commit 6964c903175994c25b4ffb44204e31672ee2ea8b
Author: Felipe Reyes <email address hidden>
Date: Fri May 19 17:13:52 2017 -0400

    Update the host_id for network:router_gateway interfaces

    The ports owned by a router_gateway need to get its host_id property
    updated during the failover of a router. Otherwise the port connected
    to the external network will always have its host_id set to the value
    obtained during creation.

    Change-Id: I5eca20e3cc64d7a9e52b0556a3cadd29eb4c821d
    Closes-Bug: 1694337
    (cherry picked from commit d8334b41d2c5bcd4916347d20008b1538d48b0ef)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 11.0.0.0b2 development milestone.

[Brian Haley (brian-haley)]

Already in stable/ocata and stable/newton

[Felipe Reyes (freyes)]

Artful already has the fix for this, so marking it as 'fix released'

[Felipe Reyes (freyes)]

I will rebase my patch on top of neutron 9.4.0

[Felipe Reyes (freyes)]

yakkety debdiff rebased on top of neutron 9.4.0

[James Page (james-page)]

Uploaded to newton-staging

[James Page (james-page)]

Hello Felipe, or anyone else affected,

Accepted neutron into newton-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:newton-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-newton-needed to verification-newton-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-newton-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[James Page (james-page)]

Uploaded for zesty-proposed

[James Page (james-page)]

Uploaded for xenial-proposed

[Brian Murray (brian-murray)]

Hello Felipe, or anyone else affected,

Accepted neutron into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:10.0.2-0ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-zesty to verification-done-zesty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-zesty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Felipe, or anyone else affected,

Accepted neutron into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:8.4.0-0ubuntu4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[James Page (james-page)]

Hello Felipe, or anyone else affected,

Accepted neutron into ocata-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ocata-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ocata-needed to verification-ocata-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ocata-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[James Page (james-page)]

Hello Felipe, or anyone else affected,

Accepted neutron into mitaka-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:mitaka-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-mitaka-needed to verification-mitaka-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-mitaka-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Felipe Reyes (freyes)]

I verified the packages available in -proposed (xenial-mitaka, xenial-newton, xenial-ocata, zesty-ocata) and they do fix the issue following the test case describe in the SRU, and there were no regressions detected (launching/destroying VMs showed no issues, also networks manipulation works as expected).

I'm here attaching the evidence of my verification for each setup.

[Launchpad Janitor (janitor)]

This bug was fixed in the package neutron - 2:10.0.2-0ubuntu1.1

---------------
neutron (2:10.0.2-0ubuntu1.1) zesty; urgency=medium

  * d/p/Update-the-host_id-for-network-router_gateway-interf.patch:
    keep the router's gateway interface updated when keepalived fails
    over (LP: #1694337).

 -- Felipe Reyes <email address hidden> Wed, 21 Jun 2017 18:01:36 -0400

[Robie Basak (racb)]

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package neutron - 2:8.4.0-0ubuntu4

---------------
neutron (2:8.4.0-0ubuntu4) xenial; urgency=medium

  * d/p/Update-the-host_id-for-network-router_gateway-interf.patch:
    keep the router's gateway interface updated when keepalived
    fails over (LP: #1694337).

 -- Felipe Reyes <email address hidden> Tue, 25 Jul 2017 17:50:16 +0100

[James Page (james-page)]

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[James Page (james-page)]

This bug was fixed in the package neutron - 2:8.4.0-0ubuntu4~cloud0
---------------

 neutron (2:8.4.0-0ubuntu4~cloud0) trusty-mitaka; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 neutron (2:8.4.0-0ubuntu4) xenial; urgency=medium
 .
   * d/p/Update-the-host_id-for-network-router_gateway-interf.patch:
     keep the router's gateway interface updated when keepalived
     fails over (LP: #1694337).

[James Page (james-page)]

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[James Page (james-page)]

This bug was fixed in the package neutron - 2:9.4.0-0ubuntu1.1~cloud0
---------------

 neutron (2:9.4.0-0ubuntu1.1~cloud0) xenial-newton; urgency=medium
 .
   * d/p/Update-the-host_id-for-network-router_gateway-interf.patch:
     keep the router's gateway interface updated when keepalived fails over
     (LP: #1694337).

[James Page (james-page)]

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[James Page (james-page)]

This bug was fixed in the package neutron - 2:10.0.2-0ubuntu1.1~cloud0
---------------

 neutron (2:10.0.2-0ubuntu1.1~cloud0) xenial-ocata; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 neutron (2:10.0.2-0ubuntu1.1) zesty; urgency=medium
 .
   * d/p/Update-the-host_id-for-network-router_gateway-interf.patch:
     keep the router's gateway interface updated when keepalived fails
     over (LP: #1694337).

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 9.4.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 10.0.3 release.