neutron-fwaas tempest v2 job on stable/newton fails with "extension could not be found"

[Nate Johnston]

The gate-neutron-fwaas-v2-dsvm-tempest is failing for stable/newton jobs in neutron-fwaas. The errors look like:

ft1.3: neutron_fwaas.tests.tempest_plugin.tests.api.test_fwaas_extensions.FWaaSExtensionTestJSON.test_create_update_delete_firewall_rule[id-563564f7-7077-4f5e-8cdc-51f37ae5a2b9]_StringException: Empty attachments:
  stderr
  stdout

pythonlogging:'': {{{
2017-02-02 21:56:08,309 22180 INFO [tempest.lib.common.rest_client] Request (FWaaSExtensionTestJSON:setUp): 404 POST http://198.61.190.237:9696/v2.0/fw/firewall_rules 0.025s
2017-02-02 21:56:08,310 22180 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>', 'Accept': 'application/json', 'Content-Type': 'application/json'}
        Body: {"firewall_rule": {"name": "fw-rule-1600127867", "protocol": "tcp", "action": "allow"}}
    Response - Headers: {u'x-openstack-request-id': 'req-267c7949-c777-4f3f-a63e-55ecf98338aa', u'content-type': 'application/json; charset=UTF-8', u'date': 'Thu, 02 Feb 2017 21:56:08 GMT', 'content-location': 'http://198.61.190.237:9696/v2.0/fw/firewall_rules', 'status': '404', u'connection': 'close', u'content-length': '112'}
        Body: {"message": "The resource could not be found.<br /><br />\n\n\n", "code": "404 Not Found", "title": "Not Found"}
}}}

Traceback (most recent call last):
  File "/opt/stack/new/neutron-fwaas/neutron_fwaas/tests/tempest_plugin/tests/api/test_fwaas_extensions.py", line 65, in setUp
    protocol="tcp")
  File "/opt/stack/new/neutron-fwaas/neutron_fwaas/tests/tempest_plugin/tests/fwaas_client.py", line 65, in create_firewall_rule
    **kwargs)
  File "/opt/stack/new/neutron-fwaas/neutron_fwaas/tests/tempest_plugin/services/client.py", line 62, in create_firewall_rule
    return self.create_resource(uri, post_data)
  File "tempest/lib/services/network/base.py", line 60, in create_resource
    resp, body = self.post(req_uri, req_post_data)
  File "tempest/lib/common/rest_client.py", line 276, in post
    return self.request('POST', url, extra_headers, headers, body, chunked)
  File "tempest/lib/common/rest_client.py", line 664, in request
    self._error_checker(resp, resp_body)
  File "tempest/lib/common/rest_client.py", line 761, in _error_checker
    raise exceptions.NotFound(resp_body, resp=resp)
tempest.lib.exceptions.NotFound: Object not found
Details: {u'title': u'Not Found', u'code': u'404 Not Found', u'message': u'The resource could not be found.<br /><br />\n\n\n'}

Example: http://logs.openstack.org/08/427508/1/check/gate-neutron-fwaas-v2-dsvm-tempest/7ca2bc0/testr_results.html.gz

[Nate Johnston (nate-johnston)]

Looks like the plugin is not getting loaded: http://logs.openstack.org/08/427508/1/check/gate-neutron-fwaas-v2-dsvm-tempest/7ca2bc0/logs/screen-q-svc.txt.gz#_2017-02-02_21_50_21_332

2017-02-02 21:50:21.331 12113 INFO neutron.manager [-] Loading Plugin: neutron_fwaas.services.firewall.fwaas_plugin_v2.FirewallPluginV2
2017-02-02 21:50:21.332 12113 WARNING stevedore.named [-] Could not load neutron_fwaas.services.firewall.fwaas_plugin_v2.FirewallPluginV2
2017-02-02 21:50:21.405 12113 DEBUG neutron.manager [-] Successfully loaded FIREWALL_V2 plugin. Description: Firewall Service v2 Plugin _load_service_plugins /opt/stack/new/neutron/neutron/manager.py:218

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/429052

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/429052
Committed: https://git.openstack.org/cgit/openstack/neutron-fwaas/commit/?id=1918b9731f43d8fdaee771904ec88944685bdf18
Submitter: Jenkins
Branch: stable/newton

commit 1918b9731f43d8fdaee771904ec88944685bdf18
Author: YAMAMOTO Takashi <email address hidden>
Date: Fri Sep 23 13:34:43 2016 +0900

    Fix functional/tempest v2 failures

    gate_hook: Fix INSTALL_MYSQL_ONLY != True case

    (cherry picked from commit 2a4cffb77a4337b9f0ae1c03131c7adb74ea8965)

    Add a variant of gate hook script

    I plan to use this for gate-neutron-fwaas-{fwaas-version}-dsvm-tempest
    to populate NETWORK_API_EXTENSIONS.

    (cherry picked from commit 8af9d55670bbf9571b870915ceda2808330736e8)

    Closes-bug: 1661420
    Closes-bug: 1661419

    Add Tempest API tests for FWaaS v2.
    Partially-Implements: blueprint fwaas-api-2.0

    (cherry picked from commit 7880f27d15b104d88e8a017aef56100dea3aff28)

    Fix tempest gate issue on FirewallNotFound exception

    In case of failed tests the following error appears in
    logs/screen-q-svc.txt.gz

      File "/opt/stack/new/neutron-fwaas/neutron_fwaas/db/firewall/v2/firewall_db_v2.py", line 805, in delete_firewall_group
        raise fw_ext.FirewallNotFound(firewall_id=id)
    AttributeError: 'module' object has no attribute 'FirewallNotFound'

    This exception doesn't exists anymore.
    It has been replaced by FirewallGroupNotFound exception.

    Closes-Bug: #1649703
    Closes-Bug: #1657299
    (cherry picked from commit 8a86782c229537ab430cac87b5df63a9ddd5424c)

    Change-Id: I625146043c7589f0935fcc9a5043b36d5eb763d5

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron-fwaas 9.0.1 release.