FWaaS: Invalid port error on associating ports (distributed router) to firewall group

[Matthias Bastian]

This bug is probably very similar to #1759773.

Creating a firewall group fails on CentOS 7.4. and OS Ocata with fwaas_v2 when using a port of a distributed router. The issue is also still present in Queens.
The validation only accepts "network:router_interface" as "device_owner", but not "network:router_interface_distributed".

The creation of the firewall group itself works, setting a port does not:

# openstack firewall group set --port ff2c03f4-22d9-4d7a-bc7a-9632ba6cd9d8 oh_noes
Failed to set firewall group 'oh_noes': Firewall Group Port ff2c03f4-22d9-4d7a-bc7a-9632ba6cd9d8 is invalid
Neutron server returns request_ids: ['req-8a8a320b-659e-4364-9604-d41e0b04d6ea']

The port in question:

# openstack port show ff2c03f4-22d9-4d7a-bc7a-9632ba6cd9d8 -f json
{
  "allowed_address_pairs": "",
  "extra_dhcp_opts": "",
  "updated_at": "2018-04-09T15:15:07Z",
  "device_owner": "network:router_interface_distributed",
  "revision_number": 9,
  "port_security_enabled": false,
  "fixed_ips": "ip_address='192.168.133.1', subnet_id='4d0e4235-a1e8-44c8-9297-e226a65beda6'",
  "id": "ff2c03f4-22d9-4d7a-bc7a-9632ba6cd9d8",
  "security_groups": "",
  "option_value": null,
  "binding_vnic_type": "normal",
  "option_name": null,
  "description": "",
  "qos_policy_id": null,
  "mac_address": "fa:16:3e:75:c8:06",
  "project_id": "4c7effe5f22b4d11ade21982746d650c",
  "status": "ACTIVE",
  "binding_profile": "",
  "binding_vif_type": "distributed",
  "binding_vif_details": "",
  "dns_assignment": "fqdn='host-192-168-133-1.vm.environment.uf0.de.', hostname='host-192-168-133-1', ip_address='192.168.133.1'",
  "ip_address": null,
  "device_id": "f305a116-5d6d-4539-883b-117de552d291",
  "name": "",
  "admin_state_up": "UP",
  "network_id": "25b641fb-b104-480c-b347-4b5f66e9bd2b",
  "dns_name": "",
  "created_at": "2018-04-09T15:15:00Z",
  "subnet_id": null,
  "binding_host_id": ""
}

[Pawel Suder (pasuder)]

Thank you for opening that bug. I would like to ask few more questions:

- would you like to provide logs from neutron server, please? it would be great to have logs related to req-8a8a320b-659e-4364-9604-d41e0b04d6ea
- could you provide information about firewall group 'oh_noes', please?
- could you provide example for working scenario, with "network:router_interface", please? logs would be welcome

Thank you!

[Matthias Bastian (piepmatz)]

Thank you for looking into the bug!

Not sure how this helps as we already know that the port validation does not accept "network:router_interface_distributed" as a port's "device_owner" but anyway, here are the logs.

OSC logs of failing scenario due to distributed router:
https://pastebin.com/KnyKRkPh

Neutron-API logs of failing scenario due to distributed router:
https://pastebin.com/svnbA8qT

OSC logs of succeeding scenario due to centralized router:
https://pastebin.com/KVS61eDg

Neutron-API logs of succeeding scenario due to centralized router:
https://pastebin.com/1xe4HvBB

(BTW, did I do anything wrong when http://paste.openstack.org/ cuts my logs?)

If you need anything else, please let me know.

[Pawel Suder (pasuder)]

Thank you for your update.

Non-working for network:router_interface_distributed

REQ: curl -g -i -X PUT http://10.0.6.174:9696/v2.0/fwaas/firewall_groups/c2cff5a9-22dd-463d-a074-ea1551bcc116.json -H "Accept: application/json" -H "Content-Type: application/json" -H "User-Agent: python-neutronclient" -H "X-Auth-Token: {SHA1}342c0419cc48c8159a1f54b772f2871226728dc3" -d '{"firewall_group": {"ports": ["04222c40-dde1-4887-9fc2-fcf42a035fd4"]}}'
http://10.0.6.174:9696 "PUT /v2.0/fwaas/firewall_groups/c2cff5a9-22dd-463d-a074-ea1551bcc116.json HTTP/1.1" 409 150
RESP: [409] Connection: keep-alive Content-Length: 150 Content-Type: application/json Date: Fri, 27 Apr 2018 12:23:39 GMT X-Openstack-Request-Id: req-8a8a320b-659e-4364-9604-d41e0b04d6ea
RESP BODY: {"NeutronError": {"message": "Firewall Group Port 04222c40-dde1-4887-9fc2-fcf42a035fd4 is invalid", "type": "FirewallGroupPortInvalid", "detail": ""}}
PUT call to network for http://10.0.6.174:9696/v2.0/fwaas/firewall_groups/c2cff5a9-22dd-463d-a074-ea1551bcc116.json used request id req-8a8a320b-659e-4364-9604-d41e0b04d6ea
Error message: {"NeutronError": {"message": "Firewall Group Port 04222c40-dde1-4887-9fc2-fcf42a035fd4 is invalid", "type": "FirewallGroupPortInvalid", "detail": ""}}
PUT call to neutron for http://10.0.6.174:9696/v2.0/fwaas/firewall_groups/c2cff5a9-22dd-463d-a074-ea1551bcc116.json used request id req-8a8a320b-659e-4364-9604-d41e0b04d6ea
Failed to set firewall group 'oh_noes': Firewall Group Port 04222c40-dde1-4887-9fc2-fcf42a035fd4 is invalid
Neutron server returns request_ids: ['req-8a8a320b-659e-4364-9604-d41e0b04d6ea']
clean_up SetFirewallGroup: Failed to set firewall group 'oh_noes': Firewall Group Port 04222c40-dde1-4887-9fc2-fcf42a035fd4 is invalid
Neutron server returns request_ids: ['req-8a8a320b-659e-4364-9604-d41e0b04d6ea']
END return value: 1

Working for network:router_interface

REQ: curl -g -i -X PUT http://10.0.6.174:9696/v2.0/fwaas/firewall_groups/c2cff5a9-22dd-463d-a074-ea1551bcc116.json -H "Accept: application/json" -H "Content-Type: application/json" -H "User-Agent: python-neutronclient" -H "X-Auth-Token: {SHA1}6f70229308bbf77338f1d789acfaf5c887873874" -d '{"firewall_group": {"ports": ["769ef119-7f98-4bfc-8572-20501b12d727"]}}'
http://10.0.6.174:9696 "PUT /v2.0/fwaas/firewall_groups/c2cff5a9-22dd-463d-a074-ea1551bcc116.json HTTP/1.1" 200 388
RESP: [200] Connection: keep-alive Content-Length: 388 Content-Type: application/json Date: Fri, 27 Apr 2018 12:39:01 GMT X-Openstack-Request-Id: req-dc077abe-9c0b-4c10-8734-754f2c28c5e8
RESP BODY: {"firewall_group": {"status": "INACTIVE", "description": "", "ingress_firewall_policy_id": null, "id": "c2cff5a9-22dd-463d-a074-ea1551bcc116", "name": "oh_noes", "admin_state_up": true, "tenant_id": "2ca9201df90c4367a178d72426231509", "public": false, "project_id": "2ca9201df90c4367a178d72426231509", "ports": ["769ef119-7f98-4bfc-8572-20501b12d727"], "egress_firewall_policy_id": null}}
PUT call to network for http://10.0.6.174:9696/v2.0/fwaas/firewall_groups/c2cff5a9-22dd-463d-a074-ea1551bcc116.json used request id req-dc077abe-9c0b-4c10-8734-754f2c28c5e8
PUT call to neutron for http://10.0.6.174:9696/v2.0/fwaas/firewall_groups/c2c...

[Matthias Bastian (piepmatz)]

Sure, here are the DEBUG logs from Neutron API for the non-working case with a distributed router.
It's just the moment when running
openstack -vv firewall group set --port 04222c40-dde1-4887-9fc2-fcf42a035fd4 oh_noes

2018-05-03 15:25:01.414 28360 DEBUG neutron.wsgi [-] (28360) accepted ('10.0.6.171', 42152) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:867
2018-05-03 15:25:01.419 28360 WARNING keystonemiddleware.auth_token [-] Using the in-process token cache is deprecated as of the 4.2.0 release and may be removed in the 5.0.0 release or the 'O' development cycle. The in-process cache causes inconsistent results and high memory usage. When the feature is removed the auth_token middleware will not cache tokens by default which may result in performance issues. It is recommended to use memcache for the auth_token token cache by setting the memcached_servers option.
2018-05-03 15:25:01.841 28360 DEBUG oslo_policy._cache_handler [req-cbd43c29-9637-443e-92cd-865fd2efcf64 ceeebe28f43b415a8c4059c414a1574e 2ca9201df90c4367a178d72426231509 - - -] Reloading cached file /etc/neutron/policy.json read_cached_file /usr/lib/python2.7/site-packages/oslo_policy/_cache_handler.py:40
2018-05-03 15:25:01.865 28360 DEBUG oslo_policy.policy [req-cbd43c29-9637-443e-92cd-865fd2efcf64 ceeebe28f43b415a8c4059c414a1574e 2ca9201df90c4367a178d72426231509 - - -] Reloaded policy file: /etc/neutron/policy.json _load_policy_file /usr/lib/python2.7/site-packages/oslo_policy/policy.py:666
2018-05-03 15:25:02.078 28360 DEBUG neutron_fwaas.db.firewall.v2.firewall_db_v2 [req-cbd43c29-9637-443e-92cd-865fd2efcf64 ceeebe28f43b415a8c4059c414a1574e 2ca9201df90c4367a178d72426231509 - - -] get_firewall_groups() called get_firewall_groups /usr/lib/python2.7/site-packages/neutron_fwaas/db/firewall/v2/firewall_db_v2.py:812
2018-05-03 15:25:02.111 28360 INFO neutron.wsgi [req-cbd43c29-9637-443e-92cd-865fd2efcf64 ceeebe28f43b415a8c4059c414a1574e 2ca9201df90c4367a178d72426231509 - - -] 10.0.6.171 "GET /v2.0/fwaas/firewall_groups.json?name=oh_noes HTTP/1.1" status: 200 len: 552 time: 0.6945391
2018-05-03 15:25:02.221 28360 INFO neutron.wsgi [req-fa1c064f-3706-473d-8f4f-d57eb334bf7f ceeebe28f43b415a8c4059c414a1574e 2ca9201df90c4367a178d72426231509 - - -] 10.0.6.171 "GET /v2.0/ports.json?id=04222c40-dde1-4887-9fc2-fcf42a035fd4 HTTP/1.1" status: 200 len: 1238 time: 0.1052740
2018-05-03 15:25:02.239 28360 DEBUG neutron_fwaas.db.firewall.v2.firewall_db_v2 [req-41c572e3-6b53-4be9-9218-a67ff5d54d96 ceeebe28f43b415a8c4059c414a1574e 2ca9201df90c4367a178d72426231509 - - -] get_firewall_groups() called get_firewall_groups /usr/lib/python2.7/site-packages/neutron_fwaas/db/firewall/v2/firewall_db_v2.py:812
2018-05-03 15:25:02.247 28360 INFO neutron.wsgi [req-41c572e3-6b53-4be9-9218-a67ff5d54d96 ceeebe28f43b415a8c4059c414a1574e 2ca9201df90c4367a178d72426231509 - - -] 10.0.6.171 "GET /v2.0/fwaas/firewall_groups.json?name=oh_noes HTTP/1.1" status: 200 len: 552 time: 0.0208750
2018-05-03 15:25:02.276 28360 DEBUG oslo_messaging._drivers.amqpdriver [req-8a8a320b-659e-4364-9604-d41e0b04d6ea ceeebe28f43b415a8c4059c414a1574e 2ca9201df90c4367a178d72426231509 - - -] CAST unique_id: a99da93ddac747dc...

[Sridar Kandaswamy (skandasw)]

Many Thanks Matthias, I am evaluating something and will get a patch out.

[Swaminathan Vasudevan (swaminathan-vasudevan)]

I discussed about this bug with Sridar while I was in Vancouver.
Why don't we just apply to the ports that have a device_id of the router, rather than checking on the device_owner, that would be helpful here.

Again for the DVR routers applying it on the router_ports will not be optimal, since we are stateless on the router interface.

[Sridar Kandaswamy (skandasw)]

Thanks Swami for updating. Matthias, I am continuing the conversation with Swami to see if we can take this approach. While fixing the validation is straightforward - we want to make sure the data path is not doing something weird.

[Matthias Bastian (piepmatz)]

Thanks for the update, guys!

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/580552

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/580552
Committed: https://git.openstack.org/cgit/openstack/neutron-fwaas/commit/?id=f8e4a193e7930c2e9ef169c6e3be53a3e2a39dbe
Submitter: Zuul
Branch: master

commit f8e4a193e7930c2e9ef169c6e3be53a3e2a39dbe
Author: Yushiro FURUKAWA <email address hidden>
Date: Fri Jul 6 13:16:40 2018 +0900

    Fix associating firewall group with DVR/L3HA port

    This commit enables to specify DVR/L3HA port for firewall group. We can
    select a port with following device_owner in creating/updating firewall
    group.

        * DVR: 'network:router_interface_distributed'
        * L3HA: 'network:ha_router_replicated_interface'

    Co-Authored-By: Nguyen Phuong An <email address hidden>
    Change-Id: I05f0f652f3e43d5c1ce5ae7933991cf92a418920
    Closes-Bug: #1762454

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/613657

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/613669

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/613691

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/613657
Committed: https://git.openstack.org/cgit/openstack/neutron-fwaas/commit/?id=eafcbce1977a541e3530398cfe362e3474a3de6a
Submitter: Zuul
Branch: stable/rocky

commit eafcbce1977a541e3530398cfe362e3474a3de6a
Author: Yushiro FURUKAWA <email address hidden>
Date: Fri Jul 6 13:16:40 2018 +0900

    Fix associating firewall group with DVR/L3HA port

    This commit enables to specify DVR/L3HA port for firewall group. We can
    select a port with following device_owner in creating/updating firewall
    group.

        * DVR: 'network:router_interface_distributed'
        * L3HA: 'network:ha_router_replicated_interface'

    Co-Authored-By: Nguyen Phuong An <email address hidden>
    Change-Id: I05f0f652f3e43d5c1ce5ae7933991cf92a418920
    Closes-Bug: #1762454
    (cherry picked from commit f8e4a193e7930c2e9ef169c6e3be53a3e2a39dbe)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron-fwaas 14.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron-fwaas 13.0.2 release.

[Slawek Kaplonski (slaweq)]

This bug has had a related patch abandoned and has been automatically un-assigned due to inactivity. Please re-assign yourself if you are continuing work or adjust the state as appropriate if it is no longer valid.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: stable/queens
Review: https://review.opendev.org/613669
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[Slawek Kaplonski (slaweq)]

This bug has had a related patch abandoned and has been automatically un-assigned due to inactivity. Please re-assign yourself if you are continuing work or adjust the state as appropriate if it is no longer valid.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: stable/pike
Review: https://review.opendev.org/613691
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.