Tempest test test_server_connectivity_rebuild is failed

[Sofiia Andriichenko]

Configuration:
    ISO: 9.0 mos iso #457
Settings:
Compute - QEMU.
Network - Neutron with VLAN segmentation.
Storage Backends - LVM
Additional services - Install Ironic, Install Sahara

In tab Settings->Compute check Nova quotas
In tab Settings->OpenStack Services check enable Install Ceilometer and Aodh
In tab Networks->Other check enable Neutron DVR
In tab Settings->Security check enable TLS for OpenStack public endpoints, HTTPS for Horizon

Nodes: controller, compute, ironic,cinder, Telemetry - MongoDB

Steps to reproduce:
    1. Deploy ISO in configuration see (Detailed bug description)
    2. Navigate to controller node
    3. Install git (use apt-get install git)
    4. Clone script to deploy rally + tempest
       # git clone https://github.com/obutenko/mos-rally-verify.git
    5. Navigate to the https://github.com/obutenko/mos-rally-verify
    6. Execute necessary steps to deploy tempest
    7. Tun test in debug mode
 rally --debug verify start --regex tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_rebuild

Expected results:
Test is passed

Actual result:
Test is Failed
(see comments)

Reproducibility:
See attachment

Workaround:
---

Impact:
---

Additional information:
Traceback (most recent call last):

  File "/home/rally/.rally/tempest/for-deployment-8421d60f-8eb3-4dd8-80b4-f374c0f0b05f/tempest/test.py", line 113, in wrapper

    return f(self, *func_args, **func_kwargs)

  File "/home/rally/.rally/tempest/for-deployment-8421d60f-8eb3-4dd8-80b4-f374c0f0b05f/tempest/scenario/test_network_advanced_server_ops.py", line 120, in test_server_connectivity_rebuild

    server, keypair, floating_ip = self._setup_network_and_servers()

  File "/home/rally/.rally/tempest/for-deployment-8421d60f-8eb3-4dd8-80b4-f374c0f0b05f/tempest/scenario/test_network_advanced_server_ops.py", line 69, in _setup_network_and_servers

    server, keypair, floating_ip)

  File "/home/rally/.rally/tempest/for-deployment-8421d60f-8eb3-4dd8-80b4-f374c0f0b05f/tempest/scenario/test_network_advanced_server_ops.py", line 93, in _wait_server_status_and_check_network_connectivity

    self._check_network_connectivity(server, keypair, floating_ip)

  File "/home/rally/.rally/tempest/for-deployment-8421d60f-8eb3-4dd8-80b4-f374c0f0b05f/tempest/scenario/test_network_advanced_server_ops.py", line 86, in _check_network_connectivity

    servers=[server])

  File "/home/rally/.rally/tempest/for-deployment-8421d60f-8eb3-4dd8-80b4-f374c0f0b05f/tempest/scenario/manager.py", line 593, in check_public_network_connectivity

    should_connect=should_connect)

  File "/home/rally/.rally/tempest/for-deployment-8421d60f-8eb3-4dd8-80b4-f374c0f0b05f/tempest/scenario/manager.py", line 577, in check_vm_connectivity

    msg=msg)

  File "/usr/local/lib/python2.7/dist-packages/unittest2/case.py", line 702, in assertTrue

    raise self.failureException(msg)

AssertionError: False is not true : Timed out waiting for 10.109.4.216 to become reachable

snapshot snapshot https://drive.google.com/a/mirantis.com/file/d/0BxPLDs6wcpbDV2FMV213QmtjcU0/view?usp=sharing

[Sofiia Andriichenko (sandriichenko)]

Reproduce only on CI

[Alexander Gubanov (ogubanov)]

I can't reproduce it (even on CI where it sometimes fails).
It maybe the same issue as here https://bugs.launchpad.net/mos/+bug/1590420 , where floating IP time by time is not available, but I'm not sure about that.

[Roman Podoliaka (rpodolyaka)]

Based on the traceback in Tempest logs it fails here:

https://github.com/openstack/tempest/blob/777a307b3c9f4284facf081e6b951b5755333adf/tempest/scenario/test_network_advanced_server_ops.py#L123

i.e. *prior* to rebuild of a server. At this point it must be no different from a simple instance boot and assigning of a floating IP address.

From what I see in the VM console it boots correctly and gets an IP address:

    === network info ===
    if-info: lo,up,127.0.0.1,8,::1
    if-info: eth0,up,10.100.0.5,28,fe80::f816:3eff:fe37:f9b9
    ip-route:default via 10.100.0.1 dev eth0
    ip-route:10.100.0.0/28 dev eth0 src 10.100.0.5
    === datasource: configdrive local ===
    instance-id: c7ee84a1-fd0f-4176-87fa-771136e223c1
    name: tempest-server-smoke-2001453871
    availability-zone: nova
    local-hostname: tempest-server-smoke-2001453871.novalocal
    launch-index: 0
    === cirros: current=unreleased uptime=25.30 ===

No errors in L3 agent logs for this particular floating IP address:

$ grep 10.109.4.216 node-2/var/log/neutron/neutron-*
node-2/var/log/neutron/neutron-l3-agent.log:2016-06-09 01:07:18.926 19181 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-e7df4dc7-8222-47c0-a7e4-4fbf41c6fcb0', 'ip', '-4', 'addr', 'add', '10.109.4.216/32', 'scope', 'global', 'dev', 'rfp-e7df4dc7-8', 'brd', '10.109.4.216'] create_process /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:84
node-2/var/log/neutron/neutron-l3-agent.log:2016-06-09 01:07:19.522 19181 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'fip-14b997d3-1a80-4b02-8b9e-2d1429916bb7', 'ip', '-4', 'route', 'replace', '10.109.4.216/32', 'via', '169.254.110.76', 'dev', 'fpr-e7df4dc7-8'] create_process /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:84
node-2/var/log/neutron/neutron-l3-agent.log:2016-06-09 01:07:19.779 19181 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'fip-14b997d3-1a80-4b02-8b9e-2d1429916bb7', 'arping', '-A', '-I', 'fg-a7d563ef-93', '-c', '3', '-w', '4.5', '10.109.4.216'] create_process /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:84

Neutron team, could you please take a look at this one?

[Oleg Bondarev (obondarev)]

Indeed according to logs everything should be ok: floating up configured correctly and moved to ACTIVE state, tempest then tries to ping ip during 2 mins and fails. Need repro to investigate why pings fail, I'll try to reproduce according to bug description.

[Oleg Bondarev (obondarev)]

I was unable to reproduce the bug. Live repro is needed for further investigation

[Sofiia Andriichenko (sandriichenko)]

Can't reproduce manual and on CI
9.1 snapshot #77