Neutron remote security group does not work

[Hang Yang]

Steps to reproduce the issue using Neutron Rocky with OVS:

# create bastion-sec-grp to allow ssh from anywhere
openstack security group create bastion-sec-grp
openstack security group rule create --ethertype=IPv4 --protocol tcp --remote-ip 0.0.0.0/0 --ingress --dst-port=22 bastion-sec-grp

# create application-sec-grp
openstack security group create application-sec-grp

# Allow ssh to egress from the bastion group to the application group
openstack security group rule create --ethertype=IPv4 --protocol tcp --remote-group application-sec-grp --egress --dst-port=22 bastion-sec-grp

# Allow ssh to ingress to the application group from the bastion group
openstack security group rule create --ethertype=IPv4 --protocol tcp --remote-group bastion-sec-grp --ingress --dst-port=22 application-sec-grp

# create servers and associate with security groups
openstack server create --wait --image rhel7 --flavor small --security-group bastion-sec-grp bastion-server
openstack server create --wait --image rhel7 --flavor small --security-group application-sec-grp application-server

After boot, bastion-server and application-server are landed on different HVs and we can ssh to bastion-server but cannot ssh to application-server from there. Neutron debug log from application-server's HV shows:

2020-02-05 22:57:05,825 DEBUG [neutron.agent.linux.openvswitch_firewall.firewall] /opt/openstack/venv/neutron/lib/python2.7/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py:_build_addr_conj_id_map:297 No member for SG <BASTION_SEC_GRP_ID>

Suspect this is related to: https://bugs.launchpad.net/neutron/+bug/1854131 and after reverting the change https://review.opendev.org/#/c/696976/ locally then remote security group started to work again.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/707248

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/708484

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/708488

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/708490

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/708491

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/707248
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=6dbba8d5ce18805a6f104782510c055017267435
Submitter: Zuul
Branch: master

commit 6dbba8d5ce18805a6f104782510c055017267435
Author: Hang Yang <email address hidden>
Date: Tue Feb 11 12:38:25 2020 -0800

    Check SG members instead of ports to skip flow update

    Security group can have a state of empty ports but non-empty members. So
    we need skip the flow update only when members dict is empty.

    Change-Id: I429edb3d2dea5fa97441909b4d2c776f97f0516f
    Closes-Bug: #1862703
    Related-Bug: #1854131

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/708484
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=fb06c497a651369682d453a283234f2414d7ace4
Submitter: Zuul
Branch: stable/train

commit fb06c497a651369682d453a283234f2414d7ace4
Author: Hang Yang <email address hidden>
Date: Tue Feb 11 12:38:25 2020 -0800

    Check SG members instead of ports to skip flow update

    Security group can have a state of empty ports but non-empty members. So
    we need skip the flow update only when members dict is empty.

    Change-Id: I429edb3d2dea5fa97441909b4d2c776f97f0516f
    Closes-Bug: #1862703
    Related-Bug: #1854131
    (cherry picked from commit 6dbba8d5ce18805a6f104782510c055017267435)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/708491
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=51eb5b2f6098b9404169b5856bdb72604f30fc0c
Submitter: Zuul
Branch: stable/queens

commit 51eb5b2f6098b9404169b5856bdb72604f30fc0c
Author: Hang Yang <email address hidden>
Date: Tue Feb 11 12:38:25 2020 -0800

    Check SG members instead of ports to skip flow update

    Security group can have a state of empty ports but non-empty members. So
    we need skip the flow update only when members dict is empty.

    Change-Id: I429edb3d2dea5fa97441909b4d2c776f97f0516f
    Closes-Bug: #1862703
    Related-Bug: #1854131
    (cherry picked from commit 6dbba8d5ce18805a6f104782510c055017267435)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/708490
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1adda631c431245234615cbcd8e8b7b27ad657f5
Submitter: Zuul
Branch: stable/rocky

commit 1adda631c431245234615cbcd8e8b7b27ad657f5
Author: Hang Yang <email address hidden>
Date: Tue Feb 11 12:38:25 2020 -0800

    Check SG members instead of ports to skip flow update

    Security group can have a state of empty ports but non-empty members. So
    we need skip the flow update only when members dict is empty.

    Change-Id: I429edb3d2dea5fa97441909b4d2c776f97f0516f
    Closes-Bug: #1862703
    Related-Bug: #1854131
    (cherry picked from commit 6dbba8d5ce18805a6f104782510c055017267435)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 16.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/708488
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=4193c6ca0e0165a2bcc7a11eee775df15019e755
Submitter: Zuul
Branch: stable/stein

commit 4193c6ca0e0165a2bcc7a11eee775df15019e755
Author: Hang Yang <email address hidden>
Date: Tue Feb 11 12:38:25 2020 -0800

    Check SG members instead of ports to skip flow update

    Security group can have a state of empty ports but non-empty members. So
    we need skip the flow update only when members dict is empty.

    Change-Id: I429edb3d2dea5fa97441909b4d2c776f97f0516f
    Closes-Bug: #1862703
    Related-Bug: #1854131
    (cherry picked from commit 6dbba8d5ce18805a6f104782510c055017267435)

[Yang Li (yang-li)]

I tested this patch, and it's true this bug can be fixed, but the bug https://bugs.launchpad.net/neutron/+bug/1854131 will be reproduced again :(

I think we should find a best way to solve the two bugs.

[Slawek Kaplonski (slaweq)]

So, do You think we should revert this patch now? Or will You work on better fix?

[Hang Yang (hangyang)]

Reverting this patch will compromise the basic functionality of remote security group again, unless we revert the patch for https://bugs.launchpad.net/neutron/+bug/1854131 as well. @yang-li Can you elaborate how this patch reopens the bug/1854131? Does that mean after moving a vm from sg1 to sg2, sg1 still somehow holds the vm's ip address in its member dictionary? If so, then I think we need a different approach to fix the issue.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 13.0.7 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron queens-eol release.