Firewall group stuck in PENDING_UPDATE

[Giuseppe Petralia]

neutron-common 2:14.0.2-0ubuntu1~cloud0
neutron-fwaas-common 1:14.0.0-0ubuntu1~cloud0
neutron-plugin-ml2 2:14.0.2-0ubuntu1~cloud0
neutron-server 2:14.0.2-0ubuntu1~cloud0
python3-neutron 2:14.0.2-0ubuntu1~cloud0
python3-neutron-dynamic-routing 2:14.0.0-0ubuntu1~cloud0
python3-neutron-fwaas 1:14.0.0-0ubuntu1~cloud0
python3-neutron-lbaas 2:14.0.0-0ubuntu1~cloud0
python3-neutron-lib 1.25.0-0ubuntu1~cloud0

When adding or removing a port to a firewall group it remains stuck in pending_update state and any update operation fails with:

ERROR neutron_lib.callbacks.manager [req-3acdfb35-f2d6-428d-a367-0a84d6df126a d090c19794dd4f27b08deab6713bd4ac b7b614bf32a64c7d8dfc0994f9c1dc7d - a1effaa626284677ade0fbe3e85c59bd a1effaa626284677ade0fbe3e85c59bd] Error during notification for neutron_fwaas.services.firewall.fwaas_plugin_v2.FirewallPluginV2.handle_update_port--9223372036854603287 port, after_update: neutron_lib.exceptions.firewall_v2.FirewallGroupInPendingState: Operation cannot be performed since associated firewall group 41f281cb-5ffd-4c0b-998f-86804825c2f6 is in PENDING_UPDATE.

Steps to reproduce:

openstack firewall group set --ingress-firewall-policy 036a0d73-f34e-43f7-87a5-c264b918af41 --egress-firewall-policy eb09e58c-683d-4a9d-8aca-c765b94f8d69 2f3f2dc5-2903-4151-af30-219065ee664e

openstack firewall group show 2f3f2dc5-2903-4151-af30-219065ee664e
+-------------------+--------------------------------------+
| Field | Value |
+-------------------+--------------------------------------+
| Description | |
| Egress Policy ID | eb09e58c-683d-4a9d-8aca-c765b94f8d69 |
| ID | 2f3f2dc5-2903-4151-af30-219065ee664e |
| Ingress Policy ID | 036a0d73-f34e-43f7-87a5-c264b918af41 |
| Name | test-fw1 |
| Ports | [] |
| Project | 8ca4fc0104ba4b72aeaf3e2a70f43519 |
| Shared | False |
| State | UP |
| Status | INACTIVE |
| project_id | 8ca4fc0104ba4b72aeaf3e2a70f43519 |
+-------------------+--------------------------------------+

openstack port show 524f3c08-ce81-4d18-b5c8-508b7762ca1d

+-----------------------+-------------------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+-------------------------------------------------------------------------------------------+
| admin_state_up | UP |
| allowed_address_pairs | |
| binding_host_id | vcd41021 |
| binding_profile | |
| binding_vif_details | bridge_name='br-int', datapath_type='system', ovs_hybrid_plug='False', port_filter='True' |
| binding_vif_type | ovs |
| binding_vnic_type | normal |
| created_at | 2019-08-08T12:49:49Z |
| data_plane_status | None |
| description | |
| device_id | 1a2d060c-5860-4cc8-b294-c30cdc4a9489 |
| device_owner | compute:AZ3 |
| dns_assignment | fqdn='test2.openstack.voith.eu1.lan.', hostname='test2', ip_address='192.168.1.21' |
| dns_domain | |
| dns_name | test2 |
| extra_dhcp_opts | |
| fixed_ips | ip_address='192.168.1.21', subnet_id='b783270c-6e5b-462d-a501-078b1a152bc6' |
| id | 524f3c08-ce81-4d18-b5c8-508b7762ca1d |
| mac_address | fa:16:3e:66:98:49 |
| name | |
| network_id | cd2a6db6-a1b7-492c-9f30-fc8d3cec9c90 |
| port_security_enabled | True |
| project_id | 8ca4fc0104ba4b72aeaf3e2a70f43519 |
| qos_policy_id | None |
| revision_number | 4 |
| security_group_ids | 695e60b0-5877-481d-aa35-5ca06b9ce528 |
| status | ACTIVE |
| tags | |
| trunk_details | None |
| updated_at | 2019-08-08T12:49:56Z |
+-----------------------+-------------------------------------------------------------------------------------------+

openstack firewall group set --port 524f3c08-ce81-4d18-b5c8-508b7762ca1d 2f3f2dc5-2903-4151-af30-219065ee664e

openstack firewall group show 2f3f2dc5-2903-4151-af30-219065ee664e
+-------------------+------------------------------------------+
| Field | Value |
+-------------------+------------------------------------------+
| Description | |
| Egress Policy ID | eb09e58c-683d-4a9d-8aca-c765b94f8d69 |
| ID | 2f3f2dc5-2903-4151-af30-219065ee664e |
| Ingress Policy ID | 036a0d73-f34e-43f7-87a5-c264b918af41 |
| Name | test-fw1 |
| Ports | ['524f3c08-ce81-4d18-b5c8-508b7762ca1d'] |
| Project | 8ca4fc0104ba4b72aeaf3e2a70f43519 |
| Shared | False |
| State | UP |
| Status | PENDING_UPDATE |
| project_id | 8ca4fc0104ba4b72aeaf3e2a70f43519 |
+-------------------+------------------------------------------+

From a functional perspective the firewall rules are not working either and we can see traffic allowed on 192.168.1.21:22 i.e.

We can't update the firewall either:

openstack firewall group set --port bbce83fa-d03f-433c-9dfe-2b72e4d1151c 2f3f2dc5-2903-4151-af30-219065ee664e
Failed to set firewall group '2f3f2dc5-2903-4151-af30-219065ee664e': Operation cannot be performed since associated firewall group 2f3f2dc5-2903-4151-af30-219065ee664e is in PENDING_UPDATE.
Neutron server returns request_ids: ['req-8cfe982a-8b15-47da-b290-079c4cad9c30']

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Edward Hope-Morley (hopem)]

I see these errors in all Ussuri deployments so lets get this fixed. If you need information I can provide it.

[Edward Hope-Morley (hopem)]

To close the loop somewhat, since fwaas is deprecated in Neutron it has been removed entirely for Victoria onwards in Ubuntu and the charms now also have an option to disable it for earlier releases [1].

[1] https://github.com/openstack/charm-neutron-api/blob/f7d248e6e6dddc24d503c5cd18888ab035fecb2a/config.yaml#L25