ncat process isn't always spawned on guest vm in scenario tests

[Slawek Kaplonski]

Some scenario tests are using ncat process spawned on guest vm to perform some checks.
It's like that e.g. for port_forwarding and qos tests.
Recently I found that for unknown for me reason, sometimes "nc" process isn't spawned properly in the guest vm. That causes test failure.

[Slawek Kaplonski (slaweq)]

Patch https://review.opendev.org/#/c/713208/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/713208
Committed: https://git.openstack.org/cgit/openstack/neutron-tempest-plugin/commit/?id=fd4141f2015d25f1b009d7cf2ebdd2907cd8e81a
Submitter: Zuul
Branch: master

commit fd4141f2015d25f1b009d7cf2ebdd2907cd8e81a
Author: Slawek Kaplonski <email address hidden>
Date: Sat Mar 14 14:34:00 2020 +0100

    Fix how nc is called in qos test

    We have already nc_listen method in base scenario tests class.
    It was since now used only in port_forwarding tests but we can
    reuse it in QoS tests also.

    There was also problem with spawning ncat process, that sometimes,
    without any clear reason for me, process wasn't spawned at all.
    That caused failure of test.

    So this patch adds new method ensure_nc_listen() which spawns ncat
    process on remote host and checkes if process is really spawned. That
    way we can avoid problems with not spawned ncat process.

    This patch also makes "server" attribute to be optional in nc_listen
    method. It is used only to log console output in case when ssh to the
    server wouldn't work as expected. And if server is not given,
    _log_console_output() method will list all servers which belongs to
    tenant and log console for each of them.

    Closes-Bug: #1868100

    Change-Id: I54c9f041f2f971219c32005b3fa573c06f0110ef

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron-tempest-plugin 1.0.0 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/738916

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/738916
Committed: https://git.openstack.org/cgit/openstack/neutron-tempest-plugin/commit/?id=b056ac25c90b4cd8459291f1eb12081b7ace304d
Submitter: Zuul
Branch: master

commit b056ac25c90b4cd8459291f1eb12081b7ace304d
Author: Flavio Fernandes <email address hidden>
Date: Wed Jul 1 14:57:13 2020 -0400

    test_port_forwarding_to_2_servers: netcat listen stops too soon

    This change is a continuation of https://review.opendev.org/#/c/713208/

    Even with the changes from commit fd4141f2015d25f1b009d7cf2ebdd2907cd8e81a
    our tests have intermittent failures where nc listen process terminates as
    soon as the shell that spawns it closes. By (1) breaking the cmd script to
    have an extra line, (2) adding a 0.1 sec delay and (3) setting stdin of
    process the proposed changes ensures that this termination no longer
    takes place.

    In order to reproduce this issue, follow these set of steps as a reference:

    From a separate shell session in the cirros vm, watch out for nc processes

    $ watch -n1 -d "ps -elf | grep nc"

    Then, start a python session to interact with that vm via ssh session:

    $ python3

    from neutron_tempest_plugin.common import ssh
    cirros_vm_ip='10.0.0.10'
    c=ssh.Client(cirros_vm_ip, 'cirros', 'gocubsgo', port=22)
    c.execute_script('nc udp -p 12346 &')

    If you are lucky (or unlucky), you will see that there are no nc
    processes running, unless you use a variation to that command that
    looks like this:

    c.execute_script('nc udp -p 12349 &\nsleep 0')

    Than then in the watch session you will see something that look like:

    _<PID>_ cirros nc udp -p 12349

    And that will remain running even after the python session closes.

    Closes-Bug: #1868100
    Change-Id: I8192391d6f2e86f9e486805374f710444e770ea4