Neutron remote security group does not work in UCA Rocky and Stein - fixed upstream

Bug #1877797 reported by James Troup on 2020-05-09
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu Cloud Archive
Critical
Unassigned
Rocky
Critical
Unassigned
Stein
Critical
Unassigned
Train
Critical
Unassigned
Ussuri
Critical
Unassigned
neutron (Ubuntu)
Status tracked in Groovy
Eoan
Critical
Unassigned
Focal
Critical
Unassigned
Groovy
Critical
James Page

Bug Description

[Impact]
OpenStack deployments using the OVS firewall driver are broken when remote security groups are used due to a regression caused by bug 1854131.

[Test Case]
Deploy OpenStack (using charms)
Follow reproduction steps as detailed in bug 1862703
# create bastion-sec-grp to allow ssh from anywhere
openstack security group create bastion-sec-grp
openstack security group rule create --ethertype=IPv4 --protocol tcp --remote-ip 0.0.0.0/0 --ingress --dst-port=22 bastion-sec-grp

# create application-sec-grp
openstack security group create application-sec-grp

# Allow ssh to egress from the bastion group to the application group
openstack security group rule create --ethertype=IPv4 --protocol tcp --remote-group application-sec-grp --egress --dst-port=22 bastion-sec-grp

# Allow ssh to ingress to the application group from the bastion group
openstack security group rule create --ethertype=IPv4 --protocol tcp --remote-group bastion-sec-grp --ingress --dst-port=22 application-sec-grp

# create servers and associate with security groups
openstack server create --wait --image rhel7 --flavor small --security-group bastion-sec-grp bastion-server
openstack server create --wait --image rhel7 --flavor small --security-group application-sec-grp application-server

After boot, bastion-server and application-server are landed on different HVs and we can ssh to bastion-server but cannot ssh to application-server from there. Neutron debug log from application-server's HV shows:

2020-02-05 22:57:05,825 DEBUG [neutron.agent.linux.openvswitch_firewall.firewall] /opt/openstack/venv/neutron/lib/python2.7/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py:_build_addr_conj_id_map:297 No member for SG <BASTION_SEC_GRP_ID>

[Regression Potential]
Low - the fix is upstream across multiple releases and resolves a previous regression in functionality.

[Original Bug Report]
Remote security groups are broken in the UCA Rocky and Stein versions of Neutron.

The broken patch was introduced in LP #1854131 and fixed in LP #1862703.

The relevant fixed has landed in Neutron 13.0.7 for Rocky¹.

The relevant fixed landed in Neutron 14.1.0-37 for Stein², alternatively the specific fix is available here:

  https://github.com/openstack/neutron/commit/4193c6ca0e0165a2bcc7a11eee775df15019e755

The Queens version of Neutron currently in UCA (12.1.0) doesn't appear to have the bad patch from #1854131 in it.

We ran into this while upgrading a customer cloud and it caused several hours of VM connectivity downtime while we diagnosed it. Please upgrade Neutron in the Ubuntu Cloud Archive to have this fix available for at least Rocky and Stein.

I realise Rocky is no longer supported, but given that the supported upgrade path from Queens is via Rocky, I think it needs fixed there too.

¹ https://docs.openstack.org/releasenotes/neutron/rocky.html
² https://docs.openstack.org/releasenotes/neutron/stein.html

James Troup (elmo) wrote :

Subscribed ~field-high due to network outages on upgrade.

summary: - rocky neutron-openvswitch-agent has a bug which causes VM connectivity
- problems during Rocky upgrade (to get to Stein)
+ UCA rocky neutron-openvswitch-agent has a bug which causes VM
+ connectivity problems during Rocky upgrade (to get to Stein)

OK, so it turns out this isn't even fixed in the Stein version of neutron in UCA. Upgrading to field-critical.

The Stein patch is here:

 https://github.com/openstack/neutron/commit/4193c6ca0e0165a2bcc7a11eee775df15019e755

James Troup (elmo) on 2020-05-10
summary: - UCA rocky neutron-openvswitch-agent has a bug which causes VM
- connectivity problems during Rocky upgrade (to get to Stein)
+ Neutron remote security group does not work in UCA Rocky and Stein -
+ fixed upstream
James Troup (elmo) on 2020-05-10
description: updated
James Troup (elmo) on 2020-05-10
description: updated
James Troup (elmo) wrote :

For reference this is the debdiff I used to build fixed packages for Bionic/Stein:

 https://paste.ubuntu.com/p/bHVZFz29dN/

Those packages are available in a PPA here:

 https://launchpad.net/~elmo/+archive/ubuntu/neutron-lp-1862703-public

James Page (james-page) wrote :

Rocky/13.0.7 is in rocky-proposed under bug 1875462 - looking at stein now.

James Page (james-page) on 2020-05-11
Changed in neutron (Ubuntu):
assignee: nobody → James Page (james-page)
importance: Undecided → Critical
James Page (james-page) wrote :

Ussuri has the required fix - rocky,stein,train are all impacted as they don't container the followup fix under bug 1862703. Queens does not have the regression (bug 1854131) or the fix (bug 1862703) but needs to be addressed as part of the next set of SRU's

James Page (james-page) on 2020-05-11
no longer affects: cloud-archive/queens
Changed in neutron (Ubuntu Groovy):
status: New → Fix Released
Changed in neutron (Ubuntu Focal):
status: New → Fix Released
Changed in neutron (Ubuntu Eoan):
status: New → Triaged
importance: Undecided → Critical
Changed in neutron (Ubuntu Focal):
importance: Undecided → Critical
description: updated
James Page (james-page) wrote :

Uploaded to eoan for SRU team review.

description: updated
description: updated

Hello James, or anyone else affected,

Accepted neutron into eoan-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:15.0.2-0ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-eoan to verification-done-eoan. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-eoan. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in neutron (Ubuntu Eoan):
status: Triaged → Fix Committed
tags: added: verification-needed verification-needed-eoan
Łukasz Zemczak (sil2100) wrote :

Could we maybe get the reproduction steps from bug LP: #1862703 copied over to the bug description for readability? Anyway, accepted, I like that this fix comes with an unit test as well.

James Page (james-page) wrote :

Hello James, or anyone else affected,

Accepted neutron into train-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:train-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-train-needed to verification-train-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-train-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-train-needed
James Page (james-page) wrote :

Hello James, or anyone else affected,

Accepted neutron into stein-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:stein-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-stein-needed to verification-stein-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-stein-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-stein-needed
description: updated
James Page (james-page) wrote :

eoan/proposed

# apt-cache policy python3-neutron
python3-neutron:
  Installed: 2:15.0.2-0ubuntu1.1
  Candidate: 2:15.0.2-0ubuntu1.1
  Version table:
 *** 2:15.0.2-0ubuntu1.1 500
        500 http://archive.ubuntu.com/ubuntu eoan-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     2:15.0.2-0ubuntu1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu eoan-updates/main amd64 Packages
     2:15.0.0-0ubuntu1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu eoan/main amd64 Packages

No log errors seem, able to SSH from bastion to application server.

Fix verified. Also ran tempest smoke to ensure no other regressions.

tags: added: verification-done verification-done-eoan
removed: verification-needed verification-needed-eoan
James Page (james-page) wrote :

UCA bionic-train/proposed

$ apt-cache policy python3-neutron
python3-neutron:
  Installed: 2:15.0.2-0ubuntu1.1~cloud0
  Candidate: 2:15.0.2-0ubuntu1.1~cloud0
  Version table:
 *** 2:15.0.2-0ubuntu1.1~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-proposed/train/main amd64 Packages
        100 /var/lib/dpkg/status

No log errors seem, able to SSH from bastion to application server.

Fix verified. Also ran tempest smoke to ensure no other regressions.

tags: added: verification-train-done
removed: verification-train-needed
James Page (james-page) wrote :

UCA bionic-stein/proposed

$ apt-cache policy python3-neutron
python3-neutron:
  Installed: 2:14.1.0-0ubuntu1~cloud1
  Candidate: 2:14.1.0-0ubuntu1~cloud1
  Version table:
 *** 2:14.1.0-0ubuntu1~cloud1 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-proposed/stein/main amd64 Packages
        100 /var/lib/dpkg/status

No log errors seem, able to SSH from bastion to application server.

Fix verified. Also ran tempest smoke to ensure no other regressions.

tags: added: verification-stein-done
removed: verification-stein-needed
James Page (james-page) wrote :

UCA bionic-rocky/proposed

$ apt-cache policy python3-neutron
python3-neutron:
  Installed: 2:13.0.7-0ubuntu1~cloud1
  Candidate: 2:13.0.7-0ubuntu1~cloud1
  Version table:
 *** 2:13.0.7-0ubuntu1~cloud1 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-proposed/rocky/main amd64 Packages
        100 /var/lib/dpkg/status

No log errors seem, able to SSH from bastion to application server.

Fix verified. Also ran tempest smoke to ensure no other regressions.

tags: added: verification-rocky-done
James Page (james-page) wrote :

Verification completed across all UCA and Ubuntu series.

It would be good to get this released today - waiting on SRU team for eoan release before releasing to the UCA pockets.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package neutron - 2:15.0.2-0ubuntu1.1

---------------
neutron (2:15.0.2-0ubuntu1.1) eoan; urgency=medium

  * d/p/lp1877797.patch: Cherry pick fix to resolve issues with
    remote security groups when used with the OVS firewall driver
    (LP: #1877797).

 -- James Page <email address hidden> Mon, 11 May 2020 08:24:20 +0100

Changed in neutron (Ubuntu Eoan):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for neutron has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

James Page (james-page) wrote :

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

James Page (james-page) wrote :

This bug was fixed in the package neutron - 2:15.0.2-0ubuntu1.1~cloud0
---------------

 neutron (2:15.0.2-0ubuntu1.1~cloud0) bionic-train; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 neutron (2:15.0.2-0ubuntu1.1) eoan; urgency=medium
 .
   * d/p/lp1877797.patch: Cherry pick fix to resolve issues with
     remote security groups when used with the OVS firewall driver
     (LP: #1877797).

James Page (james-page) wrote :

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

James Page (james-page) wrote :

This bug was fixed in the package neutron - 2:14.1.0-0ubuntu1~cloud1
---------------

 neutron (2:14.1.0-0ubuntu1~cloud1) bionic-stein; urgency=medium
 .
   * d/p/lp1877797.patch: Cherry pick fix to resolve issues with
     remote security groups when used with the OVS firewall driver
     (LP: #1877797).

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers