OSTF test "Check that VM is accessible via floating IP address" with error Floating IP cannot be removed

[Anastasia Palkina]

ISO #197
"build_id": "2013-12-24_11-59-38",
"ostf_sha": "b2adccf409e3c03237860b6a4a3e7d8199fc2c0a",
"build_number": "197",
"nailgun_sha": "7fda5f6ad4cd559a379946b68a5afa3c01c66b18",
"fuelmain_sha": "31adf1ead24b939847e3f95eef6a9ef7ca78554c",
"astute_sha": "9000b12aa0f3639bd4a18efeb82ef554fdbde9c0",
"release": "4.0",
"fuellib_sha": "8d4984453bc2d6bf503c58d6bc7183bf948a6ecf"

1. Create new environment (Ubuntu, simple mode)
2. Choose installing Savanna and Ceilometer
3. Add 1 controller+cinder and 1 compute
4. Start deployment. It was successful
5. Start OSTF test. Test "Check that VM is accessible via floating IP address" has failed on step 6 with error: Floating IP cannot be removed. Please refer to OpenStack logs for more details.

(nose_storage_plugin) fuel_health.tests.smoke.test_nova_create_instance_with_connectivity.TestNovaNetwork.test_005_check_public_network_connectivity
Traceback (most recent call last):
  File "/opt/fuel_plugins/ostf/lib/python2.6/site-packages/unittest2/case.py", line 340, in run
    testMethod()
  File "/opt/fuel_plugins/ostf/lib/python2.6/site-packages/fuel_health/tests/smoke/test_nova_create_instance_with_connectivity.py", line 221, in test_005_check_public_network_connectivity
    "removing floating IP", server, floating_ip)
  File "/opt/fuel_plugins/ostf/lib/python2.6/site-packages/fuel_health/common/test_mixins.py", line 186, in verify
    " Please refer to OpenStack logs for more details.")
  File "/opt/fuel_plugins/ostf/lib/python2.6/site-packages/unittest2/case.py", line 415, in fail
    raise self.failureException(msg)
AssertionError: Step 6 failed: Floating IP cannot be removed. Please refer to OpenStack logs for more details.

6. Manually I created security group, instance, floating IP. Associate floating IP and diassociate floating IP.

[Mike Scherbakov (mihgen)]

Please set confirmed state when you set priority.

[Tatyanka (tatyana-leontovich)]

Actualy failed with 500 from api
fuel_health.nmanager: DEBUG: Command execution successful.
requests.packages.urllib3.connectionpool: INFO: Starting new HTTP connection (1): 10.20.0.4
requests.packages.urllib3.connectionpool: DEBUG: "POST http://172.16.0.2:8774/v2/4475cfaae9ae4787b32140fbfa944fc8/servers/63bb14bb-2b68-479a-8d73-93e070ca9487/action HTTP/1.1" 500 128
fuel_health.common.test_mixins: DEBUG: Traceback (most recent call last):
  File "/opt/fuel_plugins/ostf/lib/python2.6/site-packages/fuel_health/common/test_mixins.py", line 180, in verify
    result = func(*args, **kwargs)
  File "/opt/fuel_plugins/ostf/lib/python2.6/site-packages/novaclient/v1_1/servers.py", line 452, in remove_floating_ip
    self._action('removeFloatingIp', server, {'address': address})
  File "/opt/fuel_plugins/ostf/lib/python2.6/site-packages/novaclient/v1_1/servers.py", line 951, in _action
    return self.api.client.post(url, body=body)
  File "/opt/fuel_plugins/ostf/lib/python2.6/site-packages/novaclient/client.py", line 232, in post
    return self._cs_request(url, 'POST', **kwargs)
  File "/opt/fuel_plugins/ostf/lib/python2.6/site-packages/novaclient/client.py", line 213, in _cs_request
    **kwargs)
  File "/opt/fuel_plugins/ostf/lib/python2.6/site-packages/novaclient/client.py", line 195, in _time_request
    resp, body = self.request(url, method, **kwargs)
  File "/opt/fuel_plugins/ostf/lib/python2.6/site-packages/novaclient/client.py", line 189, in request
    raise exceptions.from_response(resp, body, url, method)
ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-3fbe963c-4ed2-422e-a519-cf3078a1f520)

traceback in nova-nova.api.openstack.log:

ttributeError: 'LocalManager' object has no attribute 'l3driver'
2013-12-24T12:09:54.402633+00:00 info: INFO: http://172.16.0.2:8774/v2/4475cfaae9ae4787b32140fbfa944fc8/servers/63bb14bb-2b68-479a-8d73-93e070ca9487/action returned with HTTP 500
2013-12-24T12:32:13.623224+00:00 err: ERROR: Caught error: 'LocalManager' object has no attribute 'l3driver'
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 119, in __call__
    return req.get_response(self.application)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1296, in send
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1260, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 545, in __call__
    return self.app(env, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
    response = self.app(environ, start_response)
  File "/usr/lib/pyth...

[Tatyanka (tatyana-leontovich)]

Assign to Tatyana for reproducing and confirming

[Tatyanka (tatyana-leontovich)]

can not reproduce, If you see this again, please ping me

[Nastya Urlapova (aurlapova)]

Tanya, please take a look at smoke env on our server,
http://jenkins-product.srt.mirantis.net:8080/view/Product%20and%20Community/job/fuel_main_iso_smoke/606

[Tatyanka (tatyana-leontovich)]

ip was not deleted according next error
The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-909d11c4-afd4-480a-89eb-7c832b689cb0)

[Vladimir Kuklin (vkuklin)]

http://paste.openstack.org/show/67960/

[Tatyanka (tatyana-leontovich)]

remove ostf tag according to tests work fine

[Tatyanka (tatyana-leontovich)]

assign to Tatyana to reproduce

[Tatyanka (tatyana-leontovich)]

try to reproduce on 184, 193, 199 iso. Works great, ip removed successfully - so move it to invalid

[Meg McRoberts (dreidellhasa)]

Not documented in 5.0 Release Notes because it is marked as Invalid.

[Artem Panchenko (apanchenko-8)]

This bug was reproduced on ISO # 330 (fuel 4.1):

http://jenkins-product.srt.mirantis.net:8080/view/4.1_swarm/job/4.1_fuelmain.system_test.centos.thread_1/158/testReport/(root)/deploy_ceilometer_simple/deploy_ceilometer_simple/

Here you can find the part of logs:

http://paste.openstack.org/show/82021/

[Andrey Sledzinskiy (asledzinskiy)]

Bug was reproduced on 334 ISO - http://jenkins-product.srt.mirantis.net:8080/view/4.1_swarm/job/4.1_fuelmain.system_test.centos.thread_1/162/testReport/junit/%28root%29/ceph_rados_gw/ceph_rados_gw/
OSTF log:
ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500)

Logs are attached

[Dmitry Borodaenko (angdraug)]

Nova stacktrace: http://paste.openstack.org/show/82703/

[Mike Scherbakov (mihgen)]

Is it a blocker for releasing 4.1.1? Is it possible to disassociate IP manually, or in a second retry? If so, I tend to move it to 4.1.2, if we create that.

[Ryan Moe (rmoe)]

Floating IPs can be disassociated manually in 4..1.1 ISO # 336

[Tatyanka (tatyana-leontovich)]

Also it should be released as well

[Tatyanka (tatyana-leontovich)]

I am not sure that problems is in test. 500 from server on this call, more over it reproduce 2-3 times from 50 attempts

[Tatyanka (tatyana-leontovich)]

Also it is not reproduce on 5.0( ostf tests are the same as in 4.1) seems we should look at the Openstack code and find the difference between isehouse and Havana for this API call

[Tatyanka (tatyana-leontovich)]

Is it reproduced now?, move to incomplete according to we need to reproduce it

[Tatyanka (tatyana-leontovich)]

there is no update over 2 month, so move to the invalid. If you faced the issue gain, please reopen this one