Check network connectivity from instance via floating IP. SSHExecCommandFailed: Command 'ping -q -c1 -w10

[Sergey Kraynev]

OSTF test fails with message:

2016-01-20 13:44:08 FAILURE Check network connectivity from instance via floating IP (fuel_health.tests.smoke.test_neutron_actions.TestNeutron.test_check_neutron_objects_creation) VM connectivity doesn`t function properly. Please refer to OpenStack logs for more details. File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 67, in testPartExecutor
    yield
  File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 601, in run
    testMethod()
  File "/usr/lib/python2.7/site-packages/fuel_health/tests/smoke/test_neutron_actions.py", line 118, in test_check_neutron_objects_creation
    30, (9, 60))
  File "/usr/lib/python2.7/site-packages/fuel_health/common/test_mixins.py", line 183, in verify
    " Please refer to OpenStack logs for more details.")
  File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 666, in fail
    raise self.failureException(msg)
Step 9 failed: VM connectivity doesn`t function properly. Please refer to OpenStack logs for more details.

There is follow error in traceback:

fuel_health.nmanager: DEBUG: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/fuel_health/nmanager.py", line 422, in retry_command
    result = method(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/fuel_health/common/ssh.py", line 166, in exec_command
    strerror=''.join(err_data).join(out_data))
SSHExecCommandFailed: Command 'ping -q -c1 -w10 10.109.18.130', exit status: 1, Error:

which fails in result with:

--- 10.109.18.130 ping statistics ---^M
3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2017ms^M
pipe 3^M
. Another effort needed.
paramiko.transport: DEBUG: EOF in transport thread
fuel_health.common.test_mixins: DEBUG: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/fuel_health/common/test_mixins.py", line 177, in verify
    result = func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/fuel_health/nmanager.py", line 842, in _check_vm_connectivity
    self.assertTrue(self._ping_ip_address(ip_address, timeout, retries),
  File "/usr/lib/python2.7/site-packages/fuel_health/nmanager.py", line 779, in _ping_ip_address
    return fuel_health.test.call_until_true(ping, 40, 1)
  File "/usr/lib/python2.7/site-packages/fuel_health/test.py", line 65, in call_until_true
    elif func():
  File "/usr/lib/python2.7/site-packages/fuel_health/nmanager.py", line 771, in ping
    ssh.exec_command, cmd)
  File "/usr/lib/python2.7/site-packages/fuel_health/nmanager.py", line 436, in retry_command
    if 'ping' not in kwargs.get('command'):
TypeError: argument of type 'NoneType' is not iterable

tests failsm because FIP is not accessible. However all neutron logs have information, that FIP was created and moved to ACTIVE status.
Looks like need review of neutron-team or Infra ream, because issues may be related with some network problems on deployment.

[Nastya Urlapova (aurlapova)]

What actually a version of Fuel? Why it is critical, how I can understand a impact from your description?

[Ivan Ponomarev (ivanzipfer)]

Please provide step to reproduce

[Sergey Shevorakov (sshevorakov)]

@Sergei, please provide description of the steps how to reproduce this bug.

[Sergey Kraynev (skraynev)]

hi Sergey.

steps for reproducing:
1. re-trigger 8.0-pkg-systest-ubuntu job

[Nastya Urlapova (aurlapova)]

@Sergey, you didn't attach any diagnostic information - version, diagnostic snapshot.