Tempest test test_server_basic_ops is Failed

[Sofiia Andriichenko]

Detailed bug description:
tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basic_ops

Configuration:
Storage Backends - Ceph RBD for volumes (Cinder), Ceph RBD for ephemeral volumes (Nova), Ceph RBD for images (Glance), Ceph RadosGW for objects (Swift API)
Additional services - Install Sahara

In tab Settings->Compute check Nova quotas
In tab Settings->OpenStack Services check enable Install Ceilometer and Aodh
In tab Networks->Other check enable Neutron DVR

Nodes: controller, compute, Ceph, Telemetry - MongoDB

Steps to reproduce:
    1. Deploy ISO in configuration see (Detailed bug description)
    2. Navigate to controller node
    3. Install git (use apt-get install git)
    4. Clone script to deploy rally + tempest
       # git clone https://github.com/obutenko/mos-rally-verify.git
    5. Navigate to the https://github.com/obutenko/mos-rally-verify
    6. Execute necessary steps to deploy tempest
    7. Tun test in debug mode
        #rally --debug verify start --regex tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basic_ops

Expected results:
Test is passed

Actual result:
Test is Failed
(see comments)

Reproducibility:
See attachment

Workaround:
---

Impact:
---

Description of the environment:
See (Detailed bug description)

Additional information:
Error Message

test failed

Stacktrace

Traceback (most recent call last):
  File "/home/rally/.rally/tempest/for-deployment-777332f2-8189-46a5-a7e4-01f79ddb81b7/tempest/scenario/manager.py", line 108, in delete_wrapper
    delete_thing(*args, **kwargs)
  File "/home/rally/.rally/tempest/for-deployment-777332f2-8189-46a5-a7e4-01f79ddb81b7/tempest/lib/services/compute/floating_ips_client.py", line 62, in delete_floating_ip
    resp, body = self.delete(url)
  File "/home/rally/.rally/tempest/for-deployment-777332f2-8189-46a5-a7e4-01f79ddb81b7/tempest/lib/common/rest_client.py", line 290, in delete
    return self.request('DELETE', url, extra_headers, headers, body)
  File "/home/rally/.rally/tempest/for-deployment-777332f2-8189-46a5-a7e4-01f79ddb81b7/tempest/lib/services/compute/base_compute_client.py", line 53, in request
    method, url, extra_headers, headers, body)
  File "/home/rally/.rally/tempest/for-deployment-777332f2-8189-46a5-a7e4-01f79ddb81b7/tempest/lib/common/rest_client.py", line 641, in request
    resp, resp_body)
  File "/home/rally/.rally/tempest/for-deployment-777332f2-8189-46a5-a7e4-01f79ddb81b7/tempest/lib/common/rest_client.py", line 760, in _error_checker
    message=message)
tempest.lib.exceptions.ServerFault: Got server fault
Details: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'neutronclient.common.exceptions.PortNotFoundClient'>

[Roman Podoliaka (rpodolyaka)]

Sofiia, something must be wrong with your environment.

{0} tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basic_ops [46.027969s] ... ok

It works for me on the latest ISO (#284):

======
Totals
======
Ran: 1 tests in 65.0000 sec.
 - Passed: 1
 - Skipped: 0
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 46.0280 sec.

Please give it another try. Or give access to your environment and ask devs for assistance, so that they can jump in for debugging it live.

[Oleksiy Butenko (obutenko)]

Change to Invalid
Can't reproduce this bug manual and on ci

[Sofiia Andriichenko (sandriichenko)]

bug reproduced only on ci and can't reproduce manual
9.0 mos iso #519

[Timofey Durakov (tdurakov)]

moving to 10

[Timur Nurlygayanov (tnurlygayanov)]

Is it actual? Can we reproduce it manually on 9.1?

[Sergii Turivnyi (sturivnyi)]

Yes it is actual.
We can reproduce it manually on 9.1.
See attachment.

[Oleg Bondarev (obondarev)]

Upstream bug: https://bugs.launchpad.net/tempest/+bug/1586931
mitaka backport: https://review.openstack.org/#/c/338184
Will be fixed in MOS with sync: https://review.fuel-infra.org/#/c/23625/

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/23625
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0/mitaka

Commit: 40100dc6213a8d2693c654aec2cc478e41772ac7
Author: Jenkins <email address hidden>
Date: Wed Aug 3 04:21:03 2016

Merge the tip of origin/stable/mitaka into origin/9.0/mitaka

d57fd0a Fix for check_vlan_transparency on mech drivers not called
a98b6eb Wait for vswitchd to add interfaces in native ovsdb
f49a25c ovsdb: Don't let block() wait indefinitely
ae6a2ec Move state reporting initialization to after worker forking
505c58d Fix broken URL in Mitaka Neutron release note
7238c8e Avoid duplicate ipset processing for security groups
63f9470 Macvtap: Allow noop alias as FW driver
198bb55 Fix of ping usage in net_helpers.async_ping()
989cf32 Lower ML2 message severity.
b0d16a0 Fix OVSBridge.set_protocols arg
c4f1dab Allow tox to be run with python 3
0038b3f Fix misuse of assertTrue in L3 DVR test case
e448b7e Check compatibility when auto schedule ha routers
4b7e7cb Imported Translations from Zanata
251d680 DVR: handle floating IP reassociation on the same host
33650bf Not auto schedule router when sync routers from agent
706abe2 Remove unhelpful test of oslo.service ServiceLauncher
cda226b Revert "Fix _get_id_for"
9e403b1 When deleting floating IP catch PortNotFound
dcbf753 Fix unicode bug for password-authenticated BGP peer
c72a240 DVR: Ensure that only one fg device can exist at a time in fip ns
412ba9f Allow min_l3_agents_per_router to equal one
b49841e Skip INVALID and UNASSIGNED ofport in vlan restore
95ddc01 sriov: Fix macvtap vf interface regex pattern
873ae30 Fix _get_id_for
ce479da OVS: UnboundLocalError on switch timeout fixed
aaa1486 Cap tempest to < 12.1.0
aa10d01 Imported Translations from Zanata
20c2bfb Imported Translations from Zanata
97088cf tests: clean up designate client session mock on test exit
b685bf7 ovs: set device MTU after it's moved into a namespace
c705e2f Fix designate dns driver for SSL based endpoints
344f7eb add PROTO_NUM_IPV6_ICMP for _validate_port_range
71686cd Update ml2 delete_subnet to deallocate via ipam
f9d5064 Fixed help messages for path_mtu and global_physnet_mtus options
85ac60c DVR: Fix allowed_address_pair port binding with delayed fip
83ca27f After a migration clean up the floating ip on the source host
eed1eb0 Updated from global requirements
7a969d4 OVS Mech: Set hybrid plug based on agent config
1d18475 Refactor the rpc callback version discovery mechanism
1eccf15 Revert "DVR: Clear SNAT namespace when agent restarts after router move"
ed82f8b qos: Add API test for shared policy
adc7418 Fix get_free_namespace_port to actually avoid used ports
e539089 Force "out-of-band" controller connection mode
cb36ef3 Fix validation of floating-ip association
f178c3a DVR: Fix check multiprefix when delete ipv4 router interface
263a161 Adopt to config_dir option being a list and not a string
870f82e [qos] section is missing from neutron.conf
a868433 neutron-db-manage: revision: fail for --autogenerate and branch
26cf043 Fix invalid mock name in test_ovs_neutron_agent
e026550 Fix keepalived functional tests
0796a15 Avoid testing oslo.service library internals
eac7256 Onl...

[Sofiia Andriichenko (sandriichenko)]

Verified on 9.1 snapshot #173

[Oleksiy Butenko (obutenko)]

reproduced again on snapshot #315

[Oleg Bondarev (obondarev)]

Please provide a link to failed job and/or a snapshot

[Oleksiy Butenko (obutenko)]

cant' reproduce manual and on ci

[Sofiia Andriichenko (sandriichenko)]

reproduced again on CI 9.1 snapshot #367 but cant' reproduce manual

Trace:

Traceback (most recent call last):

  File "tempest/lib/common/utils/test_utils.py", line 84, in call_and_ignore_notfound_exc

    return func(*args, **kwargs)

  File "tempest/lib/services/compute/floating_ips_client.py", line 74, in delete_floating_ip

    resp, body = self.delete(url)

  File "tempest/lib/common/rest_client.py", line 307, in delete

    return self.request('DELETE', url, extra_headers, headers, body)

  File "tempest/lib/services/compute/base_compute_client.py", line 48, in request

    method, url, extra_headers, headers, body, chunked)

  File "tempest/lib/common/rest_client.py", line 665, in request

    resp, resp_body)

  File "tempest/lib/common/rest_client.py", line 829, in _error_checker

    message=message)

tempest.lib.exceptions.ServerFault: Got server fault

Details: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.

<class 'neutronclient.common.exceptions.PortNotFoundClient'>

snapshot: https://drive.google.com/a/mirantis.com/file/d/0BxPLDs6wcpbDVGpzQVJNWFZXS3c/view?usp=sharing

[Alexander Ignatov (aignatov)]

Not reproduced on my environment, and seems no appearances from CI as well. Marked as Invalid. Please reopen if it occurs again.