ExternalNetworksTest[JSON,XML].test_delete_external_networks_with_floating_ip failures

Bug #1395368 reported by Joshua Harlow
60
This bug affects 12 people
Affects Status Importance Assigned to Milestone
Glance Client
Invalid
Undecided
Unassigned
neutron
Fix Released
Undecided
Unassigned
tempest
Invalid
Undecided
Unassigned

Bug Description

I'm unsure as to the root cause (looking through the basic neutron, nova api logs didn't show much usefulness around the times these failed)...

Here is the full log and link though:

http://logs.openstack.org/51/136551/2/check/gate-tempest-dsvm-neutron-src-taskflow-icehouse/8e0bb9f/

2014-11-22 18:50:53.315 | {1} tempest.api.network.admin.test_external_network_extension.ExternalNetworksTestJSON.test_delete_external_networks_with_floating_ip [0.984493s] ... FAILED

ft333.2: tempest.api.network.admin.test_external_network_extension.ExternalNetworksTestXML.test_delete_external_networks_with_floating_ip_StringException: pythonlogging:'': {{{
2014-11-22 18:54:54,689 2113 DEBUG [tempest.common.rest_client] Request (ExternalNetworksTestXML:test_delete_external_networks_with_floating_ip): 201 POST http://127.0.0.1:9696/v2.0/networks 0.080s
    Request - Headers: {'Content-Type': 'application/xml', 'Accept': 'application/xml', 'X-Auth-Token': '<omitted>'}
        Body: <?xml version="1.0" encoding="UTF-8"?>
<network xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:router="http://docs.openstack.org/ext/neutron/router/api/v1.0"><router:external >True</router:external></network>
    Response - Headers: {'status': '201', 'content-length': '805', 'connection': 'close', 'date': 'Sat, 22 Nov 2014 18:54:54 GMT', 'content-type': 'application/xml; charset=UTF-8', 'x-openstack-request-id': 'req-9d70cf16-d0ae-46e6-9f40-9c68cc50a2f0'}
        Body: <?xml version='1.0' encoding='UTF-8'?>
<network xmlns="http://openstack.org/quantum/api/v2.0" xmlns:provider="http://docs.openstack.org/ext/provider/api/v1.0" xmlns:quantum="http://openstack.org/quantum/api/v2.0" xmlns:router="http://docs.openstack.org/ext/neutron/router/api/v1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><status>ACTIVE</status><subnets quantum:type="list" /><name /><provider:physical_network xsi:nil="true" /><admin_state_up quantum:type="bool">True</admin_state_up><tenant_id>a3451f42641c4a3ab319253134e5a036</tenant_id><provider:network_type>local</provider:network_type><router:external quantum:type="bool">True</router:external><shared quantum:type="bool">False</shared><id>0b8004ad-7e9d-4b8c-9de1-efa8177f6dfc</id><provider:segmentation_id xsi:nil="true" /></network>
2014-11-22 18:54:54,789 2113 DEBUG [tempest.common.rest_client] Request (ExternalNetworksTestXML:test_delete_external_networks_with_floating_ip): 201 POST http://127.0.0.1:9696/v2.0/subnets 0.098s
    Request - Headers: {'Content-Type': 'application/xml', 'Accept': 'application/xml', 'X-Auth-Token': '<omitted>'}
        Body: <?xml version="1.0" encoding="UTF-8"?>
<subnet xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><ip_version >4</ip_version><network_id >0b8004ad-7e9d-4b8c-9de1-efa8177f6dfc</network_id><cidr >10.100.0.0/28</cidr><gateway_ip >10.100.0.1</gateway_ip></subnet>
    Response - Headers: {'status': '201', 'content-length': '729', 'connection': 'close', 'date': 'Sat, 22 Nov 2014 18:54:54 GMT', 'content-type': 'application/xml; charset=UTF-8', 'x-openstack-request-id': 'req-4f27ac74-c53f-4dd7-bd90-7bb217c82626'}
        Body: <?xml version='1.0' encoding='UTF-8'?>
<subnet xmlns="http://openstack.org/quantum/api/v2.0" xmlns:quantum="http://openstack.org/quantum/api/v2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><name /><enable_dhcp quantum:type="bool">True</enable_dhcp><network_id>0b8004ad-7e9d-4b8c-9de1-efa8177f6dfc</network_id><tenant_id>a3451f42641c4a3ab319253134e5a036</tenant_id><dns_nameservers quantum:type="list" /><allocation_pools><allocation_pool><start>10.100.0.2</start><end>10.100.0.14</end></allocation_pool></allocation_pools><host_routes quantum:type="list" /><ip_version quantum:type="int">4</ip_version><gateway_ip>10.100.0.1</gateway_ip><cidr>10.100.0.0/28</cidr><id>49d84272-4d7e-49fd-bf72-0dcf7c923db6</id></subnet>
2014-11-22 18:54:54,877 2113 DEBUG [tempest.common.rest_client] Request (ExternalNetworksTestXML:test_delete_external_networks_with_floating_ip): 201 POST http://127.0.0.1:9696/v2.0/floatingips 0.087s
    Request - Headers: {'Content-Type': 'application/xml', 'Accept': 'application/xml', 'X-Auth-Token': '<omitted>'}
        Body: <?xml version="1.0" encoding="UTF-8"?>
<floatingip xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><floating_network_id >0b8004ad-7e9d-4b8c-9de1-efa8177f6dfc</floating_network_id></floatingip>
    Response - Headers: {'status': '201', 'content-length': '560', 'connection': 'close', 'date': 'Sat, 22 Nov 2014 18:54:54 GMT', 'content-type': 'application/xml; charset=UTF-8', 'x-openstack-request-id': 'req-f287a723-e173-4c69-b063-c0574860abfc'}
        Body: <?xml version='1.0' encoding='UTF-8'?>
<floatingip xmlns="http://openstack.org/quantum/api/v2.0" xmlns:quantum="http://openstack.org/quantum/api/v2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><floating_network_id>0b8004ad-7e9d-4b8c-9de1-efa8177f6dfc</floating_network_id><router_id xsi:nil="true" /><fixed_ip_address xsi:nil="true" /><floating_ip_address>10.100.0.2</floating_ip_address><tenant_id>a3451f42641c4a3ab319253134e5a036</tenant_id><status>DOWN</status><port_id xsi:nil="true" /><id>ded008db-5868-45c9-9186-0b700211f0ef</id></floatingip>
2014-11-22 18:54:54,888 2113 DEBUG [tempest.common.rest_client] Request (ExternalNetworksTestXML:test_delete_external_networks_with_floating_ip): 200 GET http://127.0.0.1:9696/v2.0/floatingips?network=0b8004ad-7e9d-4b8c-9de1-efa8177f6dfc 0.009s
    Request - Headers: {'Content-Type': 'application/xml', 'Accept': 'application/xml', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'status': '200', 'content-length': '587', 'content-location': 'http://127.0.0.1:9696/v2.0/floatingips?network=0b8004ad-7e9d-4b8c-9de1-efa8177f6dfc', 'connection': 'close', 'date': 'Sat, 22 Nov 2014 18:54:54 GMT', 'content-type': 'application/xml; charset=UTF-8', 'x-openstack-request-id': 'req-745d0e19-d3e0-4166-9959-0a260f5877e4'}
        Body: <?xml version='1.0' encoding='UTF-8'?>
<floatingips xmlns="http://openstack.org/quantum/api/v2.0" xmlns:quantum="http://openstack.org/quantum/api/v2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><floatingip><floating_network_id>0b8004ad-7e9d-4b8c-9de1-efa8177f6dfc</floating_network_id><router_id xsi:nil="true" /><fixed_ip_address xsi:nil="true" /><floating_ip_address>10.100.0.2</floating_ip_address><tenant_id>a3451f42641c4a3ab319253134e5a036</tenant_id><status>DOWN</status><port_id xsi:nil="true" /><id>ded008db-5868-45c9-9186-0b700211f0ef</id></floatingip></floatingips>
2014-11-22 18:54:54,918 2113 DEBUG [tempest.common.rest_client] Request (ExternalNetworksTestXML:test_delete_external_networks_with_floating_ip): 409 DELETE http://127.0.0.1:9696/v2.0/networks/0b8004ad-7e9d-4b8c-9de1-efa8177f6dfc 0.029s
    Request - Headers: {'Content-Type': 'application/xml', 'Accept': 'application/xml', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'status': '409', 'content-length': '410', 'connection': 'close', 'date': 'Sat, 22 Nov 2014 18:54:54 GMT', 'content-type': 'application/xml; charset=UTF-8', 'x-openstack-request-id': 'req-9db8577f-771e-412f-97a7-15281818cea5'}
        Body: <?xml version='1.0' encoding='UTF-8'?>
<NeutronError xmlns="http://openstack.org/quantum/api/v2.0" xmlns:quantum="http://openstack.org/quantum/api/v2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><message>Unable to complete operation on network 0b8004ad-7e9d-4b8c-9de1-efa8177f6dfc. There are one or more ports still in use on the network.</message><type>NetworkInUse</type><detail /></NeutronError>
2014-11-22 18:54:54,981 2113 DEBUG [tempest.common.rest_client] Request (ExternalNetworksTestXML:_run_cleanups): 204 DELETE http://127.0.0.1:9696/v2.0/floatingips/ded008db-5868-45c9-9186-0b700211f0ef 0.061s
    Request - Headers: {'Content-Type': 'application/xml', 'Accept': 'application/xml', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'date': 'Sat, 22 Nov 2014 18:54:54 GMT', 'status': '204', 'connection': 'close', 'content-length': '0', 'x-openstack-request-id': 'req-c9a14df8-a1e2-4476-8ae1-59485bdc5ff0'}
        Body:
2014-11-22 18:54:55,107 2113 DEBUG [tempest.common.rest_client] Request (ExternalNetworksTestXML:_run_cleanups): 204 DELETE http://127.0.0.1:9696/v2.0/networks/0b8004ad-7e9d-4b8c-9de1-efa8177f6dfc 0.125s
    Request - Headers: {'Content-Type': 'application/xml', 'Accept': 'application/xml', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'date': 'Sat, 22 Nov 2014 18:54:55 GMT', 'status': '204', 'connection': 'close', 'content-length': '0', 'x-openstack-request-id': 'req-04569985-946b-405d-b29e-5e2224f8952f'}
        Body:
}}}

Traceback (most recent call last):
  File "tempest/api/network/admin/test_external_network_extension.py", line 110, in test_delete_external_networks_with_floating_ip
    client.delete_network(external_network['id'])
  File "tempest/services/network/network_client_base.py", line 124, in _delete
    resp, body = self.delete(uri)
  File "tempest/services/network/network_client_base.py", line 83, in delete
    return self.rest_client.delete(uri, headers)
  File "tempest/common/rest_client.py", line 240, in delete
    return self.request('DELETE', url, extra_headers, headers, body)
  File "tempest/common/rest_client.py", line 454, in request
    resp, resp_body)
  File "tempest/common/rest_client.py", line 508, in _error_checker
    raise exceptions.Conflict(resp_body)
Conflict: An object with that identifier already exists
Details: {'message': 'Unable to complete operation on network 0b8004ad-7e9d-4b8c-9de1-efa8177f6dfc. There are one or more ports still in use on the network.', 'type': 'NetworkInUse', 'detail': {}}
Traceback (most recent call last):
_StringException: Empty attachments:
  stderr
  stdout

pythonlogging:'': {{{
2014-11-22 18:54:54,689 2113 DEBUG [tempest.common.rest_client] Request (ExternalNetworksTestXML:test_delete_external_networks_with_floating_ip): 201 POST http://127.0.0.1:9696/v2.0/networks 0.080s
    Request - Headers: {'Content-Type': 'application/xml', 'Accept': 'application/xml', 'X-Auth-Token': '<omitted>'}
        Body: <?xml version="1.0" encoding="UTF-8"?>
<network xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:router="http://docs.openstack.org/ext/neutron/router/api/v1.0"><router:external >True</router:external></network>
    Response - Headers: {'status': '201', 'content-length': '805', 'connection': 'close', 'date': 'Sat, 22 Nov 2014 18:54:54 GMT', 'content-type': 'application/xml; charset=UTF-8', 'x-openstack-request-id': 'req-9d70cf16-d0ae-46e6-9f40-9c68cc50a2f0'}
        Body: <?xml version='1.0' encoding='UTF-8'?>
<network xmlns="http://openstack.org/quantum/api/v2.0" xmlns:provider="http://docs.openstack.org/ext/provider/api/v1.0" xmlns:quantum="http://openstack.org/quantum/api/v2.0" xmlns:router="http://docs.openstack.org/ext/neutron/router/api/v1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><status>ACTIVE</status><subnets quantum:type="list" /><name /><provider:physical_network xsi:nil="true" /><admin_state_up quantum:type="bool">True</admin_state_up><tenant_id>a3451f42641c4a3ab319253134e5a036</tenant_id><provider:network_type>local</provider:network_type><router:external quantum:type="bool">True</router:external><shared quantum:type="bool">False</shared><id>0b8004ad-7e9d-4b8c-9de1-efa8177f6dfc</id><provider:segmentation_id xsi:nil="true" /></network>
2014-11-22 18:54:54,789 2113 DEBUG [tempest.common.rest_client] Request (ExternalNetworksTestXML:test_delete_external_networks_with_floating_ip): 201 POST http://127.0.0.1:9696/v2.0/subnets 0.098s
    Request - Headers: {'Content-Type': 'application/xml', 'Accept': 'application/xml', 'X-Auth-Token': '<omitted>'}
        Body: <?xml version="1.0" encoding="UTF-8"?>
<subnet xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><ip_version >4</ip_version><network_id >0b8004ad-7e9d-4b8c-9de1-efa8177f6dfc</network_id><cidr >10.100.0.0/28</cidr><gateway_ip >10.100.0.1</gateway_ip></subnet>
    Response - Headers: {'status': '201', 'content-length': '729', 'connection': 'close', 'date': 'Sat, 22 Nov 2014 18:54:54 GMT', 'content-type': 'application/xml; charset=UTF-8', 'x-openstack-request-id': 'req-4f27ac74-c53f-4dd7-bd90-7bb217c82626'}
        Body: <?xml version='1.0' encoding='UTF-8'?>
<subnet xmlns="http://openstack.org/quantum/api/v2.0" xmlns:quantum="http://openstack.org/quantum/api/v2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><name /><enable_dhcp quantum:type="bool">True</enable_dhcp><network_id>0b8004ad-7e9d-4b8c-9de1-efa8177f6dfc</network_id><tenant_id>a3451f42641c4a3ab319253134e5a036</tenant_id><dns_nameservers quantum:type="list" /><allocation_pools><allocation_pool><start>10.100.0.2</start><end>10.100.0.14</end></allocation_pool></allocation_pools><host_routes quantum:type="list" /><ip_version quantum:type="int">4</ip_version><gateway_ip>10.100.0.1</gateway_ip><cidr>10.100.0.0/28</cidr><id>49d84272-4d7e-49fd-bf72-0dcf7c923db6</id></subnet>
2014-11-22 18:54:54,877 2113 DEBUG [tempest.common.rest_client] Request (ExternalNetworksTestXML:test_delete_external_networks_with_floating_ip): 201 POST http://127.0.0.1:9696/v2.0/floatingips 0.087s
    Request - Headers: {'Content-Type': 'application/xml', 'Accept': 'application/xml', 'X-Auth-Token': '<omitted>'}
        Body: <?xml version="1.0" encoding="UTF-8"?>
<floatingip xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><floating_network_id >0b8004ad-7e9d-4b8c-9de1-efa8177f6dfc</floating_network_id></floatingip>
    Response - Headers: {'status': '201', 'content-length': '560', 'connection': 'close', 'date': 'Sat, 22 Nov 2014 18:54:54 GMT', 'content-type': 'application/xml; charset=UTF-8', 'x-openstack-request-id': 'req-f287a723-e173-4c69-b063-c0574860abfc'}
        Body: <?xml version='1.0' encoding='UTF-8'?>
<floatingip xmlns="http://openstack.org/quantum/api/v2.0" xmlns:quantum="http://openstack.org/quantum/api/v2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><floating_network_id>0b8004ad-7e9d-4b8c-9de1-efa8177f6dfc</floating_network_id><router_id xsi:nil="true" /><fixed_ip_address xsi:nil="true" /><floating_ip_address>10.100.0.2</floating_ip_address><tenant_id>a3451f42641c4a3ab319253134e5a036</tenant_id><status>DOWN</status><port_id xsi:nil="true" /><id>ded008db-5868-45c9-9186-0b700211f0ef</id></floatingip>
2014-11-22 18:54:54,888 2113 DEBUG [tempest.common.rest_client] Request (ExternalNetworksTestXML:test_delete_external_networks_with_floating_ip): 200 GET http://127.0.0.1:9696/v2.0/floatingips?network=0b8004ad-7e9d-4b8c-9de1-efa8177f6dfc 0.009s
    Request - Headers: {'Content-Type': 'application/xml', 'Accept': 'application/xml', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'status': '200', 'content-length': '587', 'content-location': 'http://127.0.0.1:9696/v2.0/floatingips?network=0b8004ad-7e9d-4b8c-9de1-efa8177f6dfc', 'connection': 'close', 'date': 'Sat, 22 Nov 2014 18:54:54 GMT', 'content-type': 'application/xml; charset=UTF-8', 'x-openstack-request-id': 'req-745d0e19-d3e0-4166-9959-0a260f5877e4'}
        Body: <?xml version='1.0' encoding='UTF-8'?>
<floatingips xmlns="http://openstack.org/quantum/api/v2.0" xmlns:quantum="http://openstack.org/quantum/api/v2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><floatingip><floating_network_id>0b8004ad-7e9d-4b8c-9de1-efa8177f6dfc</floating_network_id><router_id xsi:nil="true" /><fixed_ip_address xsi:nil="true" /><floating_ip_address>10.100.0.2</floating_ip_address><tenant_id>a3451f42641c4a3ab319253134e5a036</tenant_id><status>DOWN</status><port_id xsi:nil="true" /><id>ded008db-5868-45c9-9186-0b700211f0ef</id></floatingip></floatingips>
2014-11-22 18:54:54,918 2113 DEBUG [tempest.common.rest_client] Request (ExternalNetworksTestXML:test_delete_external_networks_with_floating_ip): 409 DELETE http://127.0.0.1:9696/v2.0/networks/0b8004ad-7e9d-4b8c-9de1-efa8177f6dfc 0.029s
    Request - Headers: {'Content-Type': 'application/xml', 'Accept': 'application/xml', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'status': '409', 'content-length': '410', 'connection': 'close', 'date': 'Sat, 22 Nov 2014 18:54:54 GMT', 'content-type': 'application/xml; charset=UTF-8', 'x-openstack-request-id': 'req-9db8577f-771e-412f-97a7-15281818cea5'}
        Body: <?xml version='1.0' encoding='UTF-8'?>
<NeutronError xmlns="http://openstack.org/quantum/api/v2.0" xmlns:quantum="http://openstack.org/quantum/api/v2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><message>Unable to complete operation on network 0b8004ad-7e9d-4b8c-9de1-efa8177f6dfc. There are one or more ports still in use on the network.</message><type>NetworkInUse</type><detail /></NeutronError>
2014-11-22 18:54:54,981 2113 DEBUG [tempest.common.rest_client] Request (ExternalNetworksTestXML:_run_cleanups): 204 DELETE http://127.0.0.1:9696/v2.0/floatingips/ded008db-5868-45c9-9186-0b700211f0ef 0.061s
    Request - Headers: {'Content-Type': 'application/xml', 'Accept': 'application/xml', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'date': 'Sat, 22 Nov 2014 18:54:54 GMT', 'status': '204', 'connection': 'close', 'content-length': '0', 'x-openstack-request-id': 'req-c9a14df8-a1e2-4476-8ae1-59485bdc5ff0'}
        Body:
2014-11-22 18:54:55,107 2113 DEBUG [tempest.common.rest_client] Request (ExternalNetworksTestXML:_run_cleanups): 204 DELETE http://127.0.0.1:9696/v2.0/networks/0b8004ad-7e9d-4b8c-9de1-efa8177f6dfc 0.125s
    Request - Headers: {'Content-Type': 'application/xml', 'Accept': 'application/xml', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'date': 'Sat, 22 Nov 2014 18:54:55 GMT', 'status': '204', 'connection': 'close', 'content-length': '0', 'x-openstack-request-id': 'req-04569985-946b-405d-b29e-5e2224f8952f'}
        Body:
}}}

Traceback (most recent call last):
  File "tempest/api/network/admin/test_external_network_extension.py", line 110, in test_delete_external_networks_with_floating_ip
    client.delete_network(external_network['id'])
  File "tempest/services/network/network_client_base.py", line 124, in _delete
    resp, body = self.delete(uri)
  File "tempest/services/network/network_client_base.py", line 83, in delete
    return self.rest_client.delete(uri, headers)
  File "tempest/common/rest_client.py", line 240, in delete
    return self.request('DELETE', url, extra_headers, headers, body)
  File "tempest/common/rest_client.py", line 454, in request
    resp, resp_body)
  File "tempest/common/rest_client.py", line 508, in _error_checker
    raise exceptions.Conflict(resp_body)
Conflict: An object with that identifier already exists
Details: {'message': 'Unable to complete operation on network 0b8004ad-7e9d-4b8c-9de1-efa8177f6dfc. There are one or more ports still in use on the network.', 'type': 'NetworkInUse', 'detail': {}}

Traceback (most recent call last):
_StringException: Empty attachments:
  stderr
  stdout

pythonlogging:'': {{{
2014-11-22 18:54:54,689 2113 DEBUG [tempest.common.rest_client] Request (ExternalNetworksTestXML:test_delete_external_networks_with_floating_ip): 201 POST http://127.0.0.1:9696/v2.0/networks 0.080s
    Request - Headers: {'Content-Type': 'application/xml', 'Accept': 'application/xml', 'X-Auth-Token': '<omitted>'}
        Body: <?xml version="1.0" encoding="UTF-8"?>
<network xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:router="http://docs.openstack.org/ext/neutron/router/api/v1.0"><router:external >True</router:external></network>
    Response - Headers: {'status': '201', 'content-length': '805', 'connection': 'close', 'date': 'Sat, 22 Nov 2014 18:54:54 GMT', 'content-type': 'application/xml; charset=UTF-8', 'x-openstack-request-id': 'req-9d70cf16-d0ae-46e6-9f40-9c68cc50a2f0'}
        Body: <?xml version='1.0' encoding='UTF-8'?>
<network xmlns="http://openstack.org/quantum/api/v2.0" xmlns:provider="http://docs.openstack.org/ext/provider/api/v1.0" xmlns:quantum="http://openstack.org/quantum/api/v2.0" xmlns:router="http://docs.openstack.org/ext/neutron/router/api/v1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><status>ACTIVE</status><subnets quantum:type="list" /><name /><provider:physical_network xsi:nil="true" /><admin_state_up quantum:type="bool">True</admin_state_up><tenant_id>a3451f42641c4a3ab319253134e5a036</tenant_id><provider:network_type>local</provider:network_type><router:external quantum:type="bool">True</router:external><shared quantum:type="bool">False</shared><id>0b8004ad-7e9d-4b8c-9de1-efa8177f6dfc</id><provider:segmentation_id xsi:nil="true" /></network>
2014-11-22 18:54:54,789 2113 DEBUG [tempest.common.rest_client] Request (ExternalNetworksTestXML:test_delete_external_networks_with_floating_ip): 201 POST http://127.0.0.1:9696/v2.0/subnets 0.098s
    Request - Headers: {'Content-Type': 'application/xml', 'Accept': 'application/xml', 'X-Auth-Token': '<omitted>'}
        Body: <?xml version="1.0" encoding="UTF-8"?>
<subnet xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><ip_version >4</ip_version><network_id >0b8004ad-7e9d-4b8c-9de1-efa8177f6dfc</network_id><cidr >10.100.0.0/28</cidr><gateway_ip >10.100.0.1</gateway_ip></subnet>
    Response - Headers: {'status': '201', 'content-length': '729', 'connection': 'close', 'date': 'Sat, 22 Nov 2014 18:54:54 GMT', 'content-type': 'application/xml; charset=UTF-8', 'x-openstack-request-id': 'req-4f27ac74-c53f-4dd7-bd90-7bb217c82626'}
        Body: <?xml version='1.0' encoding='UTF-8'?>
<subnet xmlns="http://openstack.org/quantum/api/v2.0" xmlns:quantum="http://openstack.org/quantum/api/v2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><name /><enable_dhcp quantum:type="bool">True</enable_dhcp><network_id>0b8004ad-7e9d-4b8c-9de1-efa8177f6dfc</network_id><tenant_id>a3451f42641c4a3ab319253134e5a036</tenant_id><dns_nameservers quantum:type="list" /><allocation_pools><allocation_pool><start>10.100.0.2</start><end>10.100.0.14</end></allocation_pool></allocation_pools><host_routes quantum:type="list" /><ip_version quantum:type="int">4</ip_version><gateway_ip>10.100.0.1</gateway_ip><cidr>10.100.0.0/28</cidr><id>49d84272-4d7e-49fd-bf72-0dcf7c923db6</id></subnet>
2014-11-22 18:54:54,877 2113 DEBUG [tempest.common.rest_client] Request (ExternalNetworksTestXML:test_delete_external_networks_with_floating_ip): 201 POST http://127.0.0.1:9696/v2.0/floatingips 0.087s
    Request - Headers: {'Content-Type': 'application/xml', 'Accept': 'application/xml', 'X-Auth-Token': '<omitted>'}
        Body: <?xml version="1.0" encoding="UTF-8"?>
<floatingip xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><floating_network_id >0b8004ad-7e9d-4b8c-9de1-efa8177f6dfc</floating_network_id></floatingip>
    Response - Headers: {'status': '201', 'content-length': '560', 'connection': 'close', 'date': 'Sat, 22 Nov 2014 18:54:54 GMT', 'content-type': 'application/xml; charset=UTF-8', 'x-openstack-request-id': 'req-f287a723-e173-4c69-b063-c0574860abfc'}
        Body: <?xml version='1.0' encoding='UTF-8'?>
<floatingip xmlns="http://openstack.org/quantum/api/v2.0" xmlns:quantum="http://openstack.org/quantum/api/v2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><floating_network_id>0b8004ad-7e9d-4b8c-9de1-efa8177f6dfc</floating_network_id><router_id xsi:nil="true" /><fixed_ip_address xsi:nil="true" /><floating_ip_address>10.100.0.2</floating_ip_address><tenant_id>a3451f42641c4a3ab319253134e5a036</tenant_id><status>DOWN</status><port_id xsi:nil="true" /><id>ded008db-5868-45c9-9186-0b700211f0ef</id></floatingip>
2014-11-22 18:54:54,888 2113 DEBUG [tempest.common.rest_client] Request (ExternalNetworksTestXML:test_delete_external_networks_with_floating_ip): 200 GET http://127.0.0.1:9696/v2.0/floatingips?network=0b8004ad-7e9d-4b8c-9de1-efa8177f6dfc 0.009s
    Request - Headers: {'Content-Type': 'application/xml', 'Accept': 'application/xml', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'status': '200', 'content-length': '587', 'content-location': 'http://127.0.0.1:9696/v2.0/floatingips?network=0b8004ad-7e9d-4b8c-9de1-efa8177f6dfc', 'connection': 'close', 'date': 'Sat, 22 Nov 2014 18:54:54 GMT', 'content-type': 'application/xml; charset=UTF-8', 'x-openstack-request-id': 'req-745d0e19-d3e0-4166-9959-0a260f5877e4'}
        Body: <?xml version='1.0' encoding='UTF-8'?>
<floatingips xmlns="http://openstack.org/quantum/api/v2.0" xmlns:quantum="http://openstack.org/quantum/api/v2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><floatingip><floating_network_id>0b8004ad-7e9d-4b8c-9de1-efa8177f6dfc</floating_network_id><router_id xsi:nil="true" /><fixed_ip_address xsi:nil="true" /><floating_ip_address>10.100.0.2</floating_ip_address><tenant_id>a3451f42641c4a3ab319253134e5a036</tenant_id><status>DOWN</status><port_id xsi:nil="true" /><id>ded008db-5868-45c9-9186-0b700211f0ef</id></floatingip></floatingips>
2014-11-22 18:54:54,918 2113 DEBUG [tempest.common.rest_client] Request (ExternalNetworksTestXML:test_delete_external_networks_with_floating_ip): 409 DELETE http://127.0.0.1:9696/v2.0/networks/0b8004ad-7e9d-4b8c-9de1-efa8177f6dfc 0.029s
    Request - Headers: {'Content-Type': 'application/xml', 'Accept': 'application/xml', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'status': '409', 'content-length': '410', 'connection': 'close', 'date': 'Sat, 22 Nov 2014 18:54:54 GMT', 'content-type': 'application/xml; charset=UTF-8', 'x-openstack-request-id': 'req-9db8577f-771e-412f-97a7-15281818cea5'}
        Body: <?xml version='1.0' encoding='UTF-8'?>
<NeutronError xmlns="http://openstack.org/quantum/api/v2.0" xmlns:quantum="http://openstack.org/quantum/api/v2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><message>Unable to complete operation on network 0b8004ad-7e9d-4b8c-9de1-efa8177f6dfc. There are one or more ports still in use on the network.</message><type>NetworkInUse</type><detail /></NeutronError>
2014-11-22 18:54:54,981 2113 DEBUG [tempest.common.rest_client] Request (ExternalNetworksTestXML:_run_cleanups): 204 DELETE http://127.0.0.1:9696/v2.0/floatingips/ded008db-5868-45c9-9186-0b700211f0ef 0.061s
    Request - Headers: {'Content-Type': 'application/xml', 'Accept': 'application/xml', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'date': 'Sat, 22 Nov 2014 18:54:54 GMT', 'status': '204', 'connection': 'close', 'content-length': '0', 'x-openstack-request-id': 'req-c9a14df8-a1e2-4476-8ae1-59485bdc5ff0'}
        Body:
2014-11-22 18:54:55,107 2113 DEBUG [tempest.common.rest_client] Request (ExternalNetworksTestXML:_run_cleanups): 204 DELETE http://127.0.0.1:9696/v2.0/networks/0b8004ad-7e9d-4b8c-9de1-efa8177f6dfc 0.125s
    Request - Headers: {'Content-Type': 'application/xml', 'Accept': 'application/xml', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'date': 'Sat, 22 Nov 2014 18:54:55 GMT', 'status': '204', 'connection': 'close', 'content-length': '0', 'x-openstack-request-id': 'req-04569985-946b-405d-b29e-5e2224f8952f'}
        Body:
}}}

Traceback (most recent call last):
  File "tempest/api/network/admin/test_external_network_extension.py", line 110, in test_delete_external_networks_with_floating_ip
    client.delete_network(external_network['id'])
  File "tempest/services/network/network_client_base.py", line 124, in _delete
    resp, body = self.delete(uri)
  File "tempest/services/network/network_client_base.py", line 83, in delete
    return self.rest_client.delete(uri, headers)
  File "tempest/common/rest_client.py", line 240, in delete
    return self.request('DELETE', url, extra_headers, headers, body)
  File "tempest/common/rest_client.py", line 454, in request
    resp, resp_body)
  File "tempest/common/rest_client.py", line 508, in _error_checker
    raise exceptions.Conflict(resp_body)
Conflict: An object with that identifier already exists
Details: {'message': 'Unable to complete operation on network 0b8004ad-7e9d-4b8c-9de1-efa8177f6dfc. There are one or more ports still in use on the network.', 'type': 'NetworkInUse', 'detail': {}}

Revision history for this message
Joshua Harlow (harlowja) wrote :

Only piece of useful information I see is:

http://logs.openstack.org/51/136551/2/check/gate-tempest-dsvm-neutron-src-taskflow-icehouse/8e0bb9f/logs/screen-q-svc.txt.gz#_2014-11-22_18_54_54_916 (which is the above error from the neutron api server...)

Revision history for this message
Brant Knudson (blk-u) wrote :

The description says this is sporadic but all changes in python-keystoneclient and keystonemiddleware are failing.

Revision history for this message
Joshua Harlow (harlowja) wrote :

Agreed. I thought it was sporadic but I've yet to see it not happen (although I thought I did once, but maybe I was dreaming...)

summary: ExternalNetworksTestJSON.test_delete_external_networks_with_floating_ip
- (icehouse) sporadic failures
+ (icehouse) failures
Revision history for this message
Jamie Lennox (jamielennox) wrote : Re: ExternalNetworksTestJSON.test_delete_external_networks_with_floating_ip (icehouse) failures

I'm having the same trouble in neutronclient, seems to be all changes.

Revision history for this message
Nikhil Manchanda (slicknik) wrote :

All changes in python-troveclient are also failing due to this.

Changed in tempest:
status: New → Confirmed
Revision history for this message
Matthew Treinish (treinish) wrote :

So I'm failing to see how this is a tempest bug, looking at the test code it's testing exactly what it set out to. That deleting an external network deletes all the resources (subnets, ports, etc) associated with it. It looks like the behavior differs from what's expected on icehouse neutron which is why the test fails.

In fact that there is a backport up to fix this on neutron: https://review.openstack.org/#/c/135903/

Changed in tempest:
status: Confirmed → Incomplete
Revision history for this message
Joshua Harlow (harlowja) wrote :

If there is a better place to put the bug I'm all fine with moving it; maybe neutron?

Revision history for this message
Steven Hardy (shardy) wrote :

Same problem in python-heatclient, I'm marking bug #1394530 as a duplicate of this

Joshua Harlow (harlowja)
summary: ExternalNetworksTestJSON.test_delete_external_networks_with_floating_ip
- (icehouse) failures
+ failures
summary: - ExternalNetworksTestJSON.test_delete_external_networks_with_floating_ip
+ ExternalNetworksTest[JSON,XML].test_delete_external_networks_with_floating_ip
failures
Revision history for this message
David Stanek (dstanek) wrote :

This keeps failing for a few reviews including: https://review.openstack.org/#/c/134993/

Revision history for this message
Markus Zoeller (markus_z) (mzoeller) wrote :

Affects reviews in python-glanceclient as well: https://review.openstack.org/#/c/133632/

Revision history for this message
Brant Knudson (blk-u) wrote :

Here's my understanding of the situation.

A change is proposed to infra/project-config -- https://review.openstack.org/#/c/137419/ -- that will disable the failing test for the client jobs. Once https://review.openstack.org/#/c/137419/ merges then those with client changes should be able to recheck and see jenkins pass.

https://review.openstack.org/#/c/135903/ was proposed to neutron stable/icehouse to fix the problem, but it's failing Jenkins. I think the fix for that failure is https://review.openstack.org/#/c/137423/ . Once https://review.openstack.org/#/c/137423/ merges then we should be able to recheck https://review.openstack.org/#/c/135903/ and get that one in.

For those of us with client changes, https://review.openstack.org/#/c/137419/ is the only one that matters.

Revision history for this message
Brant Knudson (blk-u) wrote :

Since https://review.openstack.org/#/c/135903/ is merged things seem to be working again.

no longer affects: keystonemiddleware
Changed in neutron:
status: New → Fix Committed
Thierry Carrez (ttx)
Changed in neutron:
milestone: none → kilo-2
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in neutron:
milestone: kilo-2 → 2015.1.0
Revision history for this message
Yaroslav Lobankov (ylobankov) wrote :

It looks like this issue has nothing to do with Tempest and it is a Neutron bug. Moving the bug status to "Invalid".

Changed in tempest:
status: Incomplete → Invalid
Revision history for this message
Kairat Kushaev (kkushaev) wrote :

Looks like Neutron bug, not related to glanceclient.

Changed in python-glanceclient:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.