Several tempest tests are failed

[Sofiia Andriichenko]

Detailed bug description:
test_network_basic_ops
test_update_instance_port_admin_state
test_update_router_admin_state
test_subnet_details

Configuration:
Settings:
Compute - QEMU.
Network - Neutron with VLAN segmentation.
Storage Backends - LVM
Additional services - Install Ironic, Install Sahara

In tab Settings->Compute check Nova quotas
In tab Settings->OpenStack Services check enable Install Ceilometer and Aodh
In tab Networks->Other check enable Neutron DVR

Nodes: controller, compute, ironic,cinder, Telemetry - MongoDB

Steps to reproduce:
    1. Deploy ISO in configuration see (Detailed bug description)
    2. Navigate to controller node
    3. Install git (use apt-get install git)
    4. Clone script to deploy rally + tempest
       # git clone https://github.com/obutenko/mos-rally-verify.git
    5. Navigate to the https://github.com/obutenko/mos-rally-verify
    6. Execute necessary steps to deploy tempest
    7. Tun test in debug mode
        #rally --debug verify start --regex tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops

Expected results:
Test is passed

Actual result:
Test is Failed
(see comments)

Reproducibility:
See attachment

Workaround:
---

Impact:
---

Description of the environment:
See (Detailed bug description)

Additional information:
Traceback (most recent call last):
  File "/home/rally/.rally/tempest/for-deployment-f7e28faf-7954-4f54-8988-41285a68de0e/tempest/test.py", line 106, in wrapper
    return f(self, *func_args, **func_kwargs)
  File "/home/rally/.rally/tempest/for-deployment-f7e28faf-7954-4f54-8988-41285a68de0e/tempest/scenario/test_network_basic_ops.py", line 399, in test_network_basic_ops
    self.check_public_network_connectivity(should_connect=True)
  File "/home/rally/.rally/tempest/for-deployment-f7e28faf-7954-4f54-8988-41285a68de0e/tempest/scenario/test_network_basic_ops.py", line 209, in check_public_network_connectivity
    self.check_floating_ip_status(floating_ip, floatingip_status)
  File "/home/rally/.rally/tempest/for-deployment-f7e28faf-7954-4f54-8988-41285a68de0e/tempest/scenario/manager.py", line 928, in check_floating_ip_status
    st=status))
  File "/usr/local/lib/python2.7/dist-packages/testtools/testcase.py", line 411, in assertEqual
    self.assertThat(observed, matcher, message)
  File "/usr/local/lib/python2.7/dist-packages/testtools/testcase.py", line 498, in assertThat
    raise mismatch_error
testtools.matchers._impl.MismatchError: 'ACTIVE' != u'DOWN': FloatingIP: {u'router_id': u'2d938276-4bf5-4d44-b997-7b781f05fbff', u'status': u'DOWN', u'description': u'', u'dns_name': u'', u'dns_domain': u'', u'floating_network_id': u'dfd371e6-a711-40d7-90b2-e6f73b7f439d', u'fixed_ip_address': u'10.100.0.5', u'floating_ip_address': u'10.109.4.221', u'tenant_id': u'bf6d4d7fd3234fb5b826c80b944113d0', u'port_id': u'401732f6-549e-4256-8f01-e79dd2c0be39', u'id': u'23f7f98e-26d6-4284-9188-7aec1a77a03f'} is at status: DOWN. failed to reach status: ACTIVE

[Sofiia Andriichenko (sandriichenko)]

snapshot: https://drive.google.com/a/mirantis.com/file/d/0BxPLDs6wcpbDMDNZYjd0cnhzXzQ/view?usp=sharing

[Oleg Bondarev (obondarev)]

Grep snapshot for router_id from traceback, as well as for floating_ip id returns nothing. I believe the snapshot and traceback are from different runs. Please either point to where I can find tempest logs in snapshot, or give a traceback corresponding to the snapshot. I need an id of failed resource in order to figure out something from snapshot.

[Oleg Bondarev (obondarev)]

On a reverted env tried to run tempest tests and saw a weird behavior of l3 agent on one of compute nodes: for some reason all green threads responsible for routers create/update are hanging. Other threads (e.g. state reporting thread) are ok so agents are alive from server point of view. However they are not able to process new routers/updates so.. tests are failing.

All threads hanged while executing 'ip netns exec <ns> ip link show' command through rootwrap daemon. Guru meditation report [1] shows that problem is somewhere in oslo_rootwrap to daemon communications through unix socket.

[1] http://paste.openstack.org/show/562198/

This might cause failures of all tests checking l3 connectivity.

[Dmitry Mescheryakov (dmitrymex)]

Sofia, please provide us an env with reproduction. Unfortunately we did not have time to investigate it while it was active.

For the record, Kirill Bespalov tried to reproduce the bug syntactically, but he did not succeed. Also, the issue did not reproduce in the next two days. Probably it is rather rare.

[Sofiia Andriichenko (sandriichenko)]

Dmitry, if this test fail again I can give you access to environment

[Sofiia Andriichenko (sandriichenko)]

Can't reproduce manual and on CI