"QoSTest:test_qos_basic_and_update" failing in DVR node scenario

Bug #1838068 reported by Rodolfo Alonso on 2019-07-26
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Undecided
Rodolfo Alonso

Bug Description

According to [1], this problem has been seen several times in the log reports. The BW testing method is failing with a timeout exception. No logs for this BW check are stored in the reports, that could means the test is blocked in the following command [2]:

 File "/opt/stack/tempest/.tox/tempest/lib/python3.6/site-packages/neutron_tempest_plugin/scenario/test_qos.py", line 110, in _check_bw
    data = client_socket.recv(QoSTestMixin.BUFFER_SIZE)

[1] http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22test_connectivity_through_2_routers%5C%22%20AND%20build_status:%5C%22FAILURE%5C%22%20AND%20build_branch:%5C%22master%5C%22%20AND%20build_name:%5C%22neutron-tempest-plugin-dvr-multinode-scenario%5C%22%20AND%20project:%5C%22openstack%2Fneutron%5C%22
[2] http://logs.openstack.org/65/672465/1/check/neutron-tempest-plugin-dvr-multinode-scenario/1d68297/testr_results.html.gz

Changed in neutron:
assignee: nobody → Rodolfo Alonso (rodolfo-alonso-hernandez)

Fix proposed to branch: master
Review: https://review.opendev.org/673023

Changed in neutron:
status: New → In Progress

Reviewed: https://review.opendev.org/673023
Committed: https://git.openstack.org/cgit/openstack/neutron-tempest-plugin/commit/?id=fa5ebc4b98e4964b73210e38772af50e2524be3b
Submitter: Zuul
Branch: master

commit fa5ebc4b98e4964b73210e38772af50e2524be3b
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Fri Jul 26 17:02:40 2019 +0000

    Refactor the test_qos.QoSTestMixin._check_bw function

    Improved the robustness of this function:
    - The remote process ("nc"), will be stopped both at the beginning and
      the end of this function, just to ensure the process is not left
      running at the beginning of the next check.
    - The socket connection is closed when the file is received (or in case
      of exception). This will close the client socket and will end the "nc"
      process in the remote VM.
    - Set a socket timeout. This timeout, specifically defined for each
      iteration, depending on the expected bandwidth, will force a quick
      timeout while receiving from the remote VM. The check will fail and it
      will be tested again, if the "FILE_DOWNLOAD_TIMEOUT" didn't happen yet.

    Change-Id: I5ce1a34f7d5d635002baa1e5b14c288e6d2bc43e
    Closes-Bug: #1838068

Changed in neutron:
status: In Progress → Fix Released

Reviewed: https://review.opendev.org/674312
Committed: https://git.openstack.org/cgit/openstack/neutron-tempest-plugin/commit/?id=31993d50fd449e2d3ba02f3d4d94989130cd2118
Submitter: Zuul
Branch: master

commit 31993d50fd449e2d3ba02f3d4d94989130cd2118
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Fri Aug 2 13:42:18 2019 +0000

    Force the kill of the "nc" process only if timeout

    If the file is received correctly, the "nc" process in the remote
    machine will end gracefully. Only in case of socket timeout, will force
    the completion of the process before returning.

    Change-Id: I6cec868daa0139f6a49807883f901d2eac84fe14
    Related-Bug: #1838068

This issue was fixed in the openstack/neutron-tempest-plugin 0.5.0 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers