When Firewall is created in conjunction with a distributed router, the firewall may or may not fail to reach the ACTIVE state as observed in [1]. The reason for this faulty behavior is because, when the firewall create request comes in, the firewall object's status is set PENDING_CREATE (see [2]), then the request is send to the L3 agent, which will work to make the state transition to ACTIVE (see [3]). This state transition is predicated on the following statements being true:
1) The tenant has firewalls
2) The tenant has routers
3) The routers' namespaces have been created on the L3 agent
Now, in the DVR case, 3) might be true or not depending on the state of the router or the cloud (see [4] for details). For instance, if the router had an external gateway set, condition 3) would be True, and the firewall state would transition to ACTIVE. This may lead the user to believe that everything is correct when it is actually not. What makes the matter worse is the fact that in the DVR case, the firewall needs itself to be distributed, which means that if we kept the same logic as outlined in [2], [3], the last L3 agent to update the state of the firewall will overwrite any other (last write wins), leading to potential inconsistency.
To start addressing this issue, it would be appropriate to tweak the logic as follow:
a) When DVR is present, firewall should be created directly in CREATED state, we'll keep the logic for the centralized case as is, where the firewall is created in PENDING_CREATE state
b) When L3 agents can install the right firewall rules, the server will need to collect all the acknowledgments from the L3 agents
c) Only after all acknowledgments have been collected and they are positive, the firewall state will transition from CREATED to ACTIVE, ERROR otherwise.
In theory, step a) could be simplified by renaming PENDING_CREATE to CREATED and leave it at that, however this would be a non-backward compatible API change which would affect the legacy case, and should be discouraged.
[1] - http://logs.openstack.org/91/114691/2/experimental/check-tempest-dsvm-neutron-dvr/93b2ff0/logs/testr_results.html.gz
[2] - https://github.com/openstack/neutron/blob/master/neutron/services/firewall/fwaas_plugin.py#L227
[3] - https://github.com/openstack/neutron/blob/master/neutron/services/firewall/agents/l3reference/firewall_l3_agent.py#L194
[4] - https://review.openstack.org/#/c/116100/
If this is a sensible approach, we'd need to tweak the Tempest FWaaS API test to accept both CREATED and PENDING_CREATE as acceptable states for the test_create_