[staging mirrors] MOS 6.1 CentOS, OSTF test Check network connectivity from instance via floating IP (failure)

[Timur Nurlygayanov]

Job:
http://jenkins-product.srt.mirantis.net:8080/job/6.1.centos.bvt_1/74/console

Traceback:
2015-01-27 14:20:28,682 - INFO nailgun_client.py:205 -- Run OSTF tests at cluster #1: ['smoke', 'sanity']
2015-01-27 14:20:29,044 - INFO http.py:72 -- self url is http://10.108.27.2:8000
2015-01-27 14:20:29,317 - INFO fuel_web_client.py:592 -- Try to run assert ostf with expected fail name None
2015-01-27 14:20:29,317 - INFO fuel_web_client.py:154 -- Assert OSTF run at cluster #1. Should fail 0 tests named None
2015-01-27 14:20:29,317 - INFO fuel_web_client.py:80 -- Wait OSTF tests at cluster #1 for 1800 seconds
2015-01-27 14:27:37,043 - INFO fuel_web_client.py:182 -- OSTF test statuses are : {u'Create volume and boot instance from it': u'skipped', u'Request snapshot list': u'success', u'Create instance flavor': u'success', u'Request volume list': u'success', u'Create keypair': u'success', u'Request flavor list': u'success', u'Request user list': u'success', u'Launch instance, create snapshot, launch instance from snapshot': u'success', u'Create volume and attach it to instance': u'skipped', u'Request instance list': u'success', u'Create security group': u'success', u'Request stack list': u'success', u'Create user and authenticate with it to Horizon': u'success', u'Request image list': u'success', u'Request active services list': u'success', u'Check network parameters': u'success', u'Launch instance': u'success', u'Check that required services are running': u'success', u'Check network connectivity from instance via floating IP': u'failure', u'Request absolute limits list': u'success'}
2015-01-27 14:27:37,219 - INFO fuel_web_client.py:623 -- Wait for task {u'status': u'running', u'name': u'dump', u'cluster': None, u'result': {}, u'progress': 0, u'message': None, u'id': 10, u'uuid': u'b06ccbc9-dd70-4d38-af7c-c677b1cb50a0'} 300 seconds
2015-01-27 14:29:22,655 - INFO decorators.py:41 -- Saving logs to "/home/jenkins/workspace/6.1.centos.bvt_1/logs/fail_error_deploy_neutron_vlan_ha-2015_01_27__14_29_22.tar.gz" file
2015-01-27 14:30:10,424 - INFO decorators.py:132 -- <<<<<****************************************************************************************************>>>>>
2015-01-27 14:30:10,424 - INFO decorators.py:133 -- Failed in method 'deploy_neutron_vlan_ha'. Make snapshot: error_deploy_neutron_vlan_ha
2015-01-27 14:30:10,424 - INFO decorators.py:142 -- You could revert this snapshot using [dos.py revert 6.1.main.CentOS.bvt_1.74.2015-01-27_12-18-29 --snapshot-name error_deploy_neutron_vlan_ha && dos.py resume 6.1.main.CentOS.bvt_1.74.2015-01-27_12-18-29 && virsh net-dumpxml 6.1.main.CentOS.bvt_1.74.2015-01-27_12-18-29_admin | grep -P "(\d+\.){3}" -o | awk '{print "Admin node IP: "$0"2"}']
2015-01-27 14:30:10,424 - INFO decorators.py:147 -- <<<<<****************************************************************************************************>>>>>
FAIL

======================================================================
FAIL: Deploy cluster in HA mode with Neutron VLAN
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/usr/lib/python2.7/dist-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/usr/lib/python2.7/dist-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/6.1.centos.bvt_1/fuelweb_test/helpers/decorators.py", line 60, in wrapper
    return func(*args, **kwagrs)
  File "/home/jenkins/workspace/6.1.centos.bvt_1/fuelweb_test/tests/test_neutron.py", line 317, in deploy_neutron_vlan_ha
    cluster_id=cluster_id)
  File "/home/jenkins/workspace/6.1.centos.bvt_1/fuelweb_test/__init__.py", line 48, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/6.1.centos.bvt_1/fuelweb_test/models/fuel_web_client.py", line 596, in run_ostf
    failed_test_name=failed_test_name)
  File "/home/jenkins/workspace/6.1.centos.bvt_1/fuelweb_test/__init__.py", line 48, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/6.1.centos.bvt_1/fuelweb_test/models/fuel_web_client.py", line 195, in assert_ostf_run
    failed_tests_res))
AssertionError: Failed tests, fails: 1 should fail: 0 failed tests name: [{u'Check network connectivity from instance via floating IP (failure)': u'Network can not be deleted Please refer to OpenStack logs for more details.'}]

----------------------------------------------------------------------

[Timur Nurlygayanov (tnurlygayanov)]

I can see many Neutron errors in OSTF logs, need to download the ISO and check deployment on CentOS

[Nastya Urlapova (aurlapova)]

Change to High, due next build has passed test.

[Artem Panchenko (apanchenko-8)]

Here is the part of OSTF logs:

neutronclient.client: DEBUG: RESP:404 CaseInsensitiveDict({'content-length': '131', 'via': '1.1 apache_api_proxy:9696', 'connection': 'close', 'date': 'Tue, 27 Jan 2015 14:23:48 GMT', 'content-type': 'application/json; charset=UTF-8', 'x-openstack-request-id': 'req-34eba2bd-e1d3-4f82-8b8e-667fdcd77bd4'}) {"NeutronError": {"message": "Port 84b6e407-ced7-4fc1-ad36-7fa482dcd795 could not be found", "type": "PortNotFound", "detail": ""}}

full log with traceback: http://paste.openstack.org/show/163798/

Neutron server logs contain the following error:

"StaleDataError: UPDATE statement on table 'ports' expected to update 1 row(s); 0 were matched.
2015-01-27 14:23:32.015 18384 TRACE oslo.messaging.rpc.dispatcher "

http://paste.openstack.org/show/163799/

[Timur Nurlygayanov (tnurlygayanov)]

Reproduced for staging MOS 6.1 as well:
http://jenkins-product.srt.mirantis.net:8080/view/Staging%20Mirrors/job/6.1.staging.centos.bvt_1/59/console