Bug #1914043 “Octavia TrafficOperationsScenarioTest tests fail o...” : Bugs : tripleo

Revision history for this message

Sagi (Sergey) Shnaidman (sshnaidm) wrote on 2021-02-01:

#1

Download full text (4.8 KiB)

Same with job periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master:

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master/fcea517/logs/undercloud/var/log/tempest/stestr_results.html.gz

TrafficOperationsScenarioTest)
fail
[x]

ft1.1: setUpClass (octavia_tempest_plugin.tests.scenario.v2.test_traffic_ops.TrafficOperationsScenarioTest)testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tempest/test.py", line 188, in setUpClass
    six.reraise(etype, value, trace)
  File "/usr/local/lib/python3.6/site-packages/six.py", line 703, in reraise
    raise value
  File "/usr/lib/python3.6/site-packages/tempest/test.py", line 181, in setUpClass
    cls.resource_setup()
  File "/usr/lib/python3.6/site-packages/octavia_tempest_plugin/tests/scenario/v2/test_traffic_ops.py", line 53, in resource_setup
    super(TrafficOperationsScenarioTest, cls).resource_setup()
  File "/usr/lib/python3.6/site-packages/octavia_tempest_plugin/tests/test_base.py", line 689, in resource_setup
    cls.webserver1_response)
  File "/usr/lib/python3.6/site-packages/octavia_tempest_plugin/tests/test_base.py", line 877, in _install_start_webserver
    stdout, stderr)
tempest.lib.exceptions.CommandFailed: Command 'scp -v -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o ConnectTimeout=5 -o ConnectionAttempts=20 -i /tmp/tmp58qwl5b2 /opt/octavia-tempest-plugin/test_server.bin cirros@192.168.24.112:/dev/shm/test_server.bin' returned non-zero exit status 1.
stdout:
b"Executing: program /usr/bin/ssh host 192.168.24.112, user cirros, command scp -v -t /dev/shm/test_server.bin\nOpenSSH_8.0p1, OpenSSL 1.1.1g FIPS 21 Apr 2020\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf\r\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\r\ndebug1: configuration requests final Match pass\r\ndebug1: re-parsing configuration\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf\r\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\r\ndebug1: Connecting to 192.168.24.112 [192.168.24.112] port 22.\r\ndebug1: fd 3 clearing O_NONBLOCK\r\ndebug1: Connection established.\r\ndebug1: identity file /tmp/tmp58qwl5b2 type -1\r\ndebug1: identity file /tmp/tmp58qwl5b2-cert type -1\r\ndebug1: Local version string SSH-2.0-OpenSSH_8.0\r\ndebug1: Remote protocol version 2.0, remote software version dropbear_2015.67\r\ndebug1: no match: dropbear_2015.67\r\ndebug1: Authenticating to 192.168.24.112:22 as 'cirros'\r\ndebug1: SSH2_MSG_KEXINIT sent\r\ndebug1: SSH2_MSG_KEXINIT received\r\ndebug1: kex: algorithm: <email address hidden>\r\ndebug1: kex: host key algorithm: ecdsa-sha2-nistp521\r\ndebug1: kex: server->client cipher: aes256-ctr MAC: hmac-sha2-256 compression: none\r\ndebug1: kex: client->server cipher: aes256-ctr MAC: hmac-sha2-256 compression: none\r\ndebug...

Same with job periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master:

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master/fcea517/logs/undercloud/var/log/tempest/stestr_results.html.gz

TrafficOperationsScenarioTest)
	fail
[x]

ft1.1: setUpClass (octavia_tempest_plugin.tests.scenario.v2.test_traffic_ops.TrafficOperationsScenarioTest)testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tempest/test.py", line 188, in setUpClass
    six.reraise(etype, value, trace)
  File "/usr/local/lib/python3.6/site-packages/six.py", line 703, in reraise
    raise value
  File "/usr/lib/python3.6/site-packages/tempest/test.py", line 181, in setUpClass
    cls.resource_setup()
  File "/usr/lib/python3.6/site-packages/octavia_tempest_plugin/tests/scenario/v2/test_traffic_ops.py", line 53, in resource_setup
    super(TrafficOperationsScenarioTest, cls).resource_setup()
  File "/usr/lib/python3.6/site-packages/octavia_tempest_plugin/tests/test_base.py", line 689, in resource_setup
    cls.webserver1_response)
  File "/usr/lib/python3.6/site-packages/octavia_tempest_plugin/tests/test_base.py", line 877, in _install_start_webserver
    stdout, stderr)
tempest.lib.exceptions.CommandFailed: Command 'scp -v -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o ConnectTimeout=5 -o ConnectionAttempts=20 -i /tmp/tmp58qwl5b2 /opt/octavia-tempest-plugin/test_server.bin cirros@192.168.24.112:/dev/shm/test_server.bin' returned non-zero exit status 1.
stdout:
b"Executing: program /usr/bin/ssh host 192.168.24.112, user cirros, command scp -v -t /dev/shm/test_server.bin\nOpenSSH_8.0p1, OpenSSL 1.1.1g FIPS  21 Apr 2020\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf\r\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\r\ndebug1: configuration requests final Match pass\r\ndebug1: re-parsing configuration\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf\r\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\r\ndebug1: Connecting to 192.168.24.112 [192.168.24.112] port 22.\r\ndebug1: fd 3 clearing O_NONBLOCK\r\ndebug1: Connection established.\r\ndebug1: identity file /tmp/tmp58qwl5b2 type -1\r\ndebug1: identity file /tmp/tmp58qwl5b2-cert type -1\r\ndebug1: Local version string SSH-2.0-OpenSSH_8.0\r\ndebug1: Remote protocol version 2.0, remote software version dropbear_2015.67\r\ndebug1: no match: dropbear_2015.67\r\ndebug1: Authenticating to 192.168.24.112:22 as 'cirros'\r\ndebug1: SSH2_MSG_KEXINIT sent\r\ndebug1: SSH2_MSG_KEXINIT received\r\ndebug1: kex: algorithm: curve25519-sha256@libssh.org\r\ndebug1: kex: host key algorithm: ecdsa-sha2-nistp521\r\ndebug1: kex: server->client cipher: aes256-ctr MAC: hmac-sha2-256 compression: none\r\ndebug1: kex: client->server cipher: aes256-ctr MAC: hmac-sha2-256 compression: none\r\ndebug1: kex: curve25519-sha256@libssh.org need=32 dh_need=32\r\ndebug1: kex: curve25519-sha256@libssh.org need=32 dh_need=32\r\ndebug1: expecting SSH2_MSG_KEX_ECDH_REPLY\r\ndebug1: Server host key: ecdsa-sha2-nistp521 SHA256:UTDCkWD/yMvQK8lz+DQUYDodvN5dh/bQcVShA9heiJk\r\nWarning: Permanently added '192.168.24.112' (ECDSA) to the list of known hosts.\r\ndebug1: rekey out after 4294967296 blocks\r\ndebug1: SSH2_MSG_NEWKEYS sent\r\ndebug1: expecting SSH2_MSG_NEWKEYS\r\ndebug1: SSH2_MSG_NEWKEYS received\r\ndebug1: rekey in after 4294967296 blocks\r\ndebug1: Will attempt key: /tmp/tmp58qwl5b2  explicit\r\ndebug1: SSH2_MSG_SERVICE_ACCEPT received\r\ndebug1: Authentications that can continue: publickey,password\r\ndebug1: Next authentication method: publickey\r\ndebug1: Trying private key: /tmp/tmp58qwl5b2\r\ndebug1: Authentication succeeded (publickey).\r\nAuthenticated to 192.168.24.112 ([192.168.24.112]:22).\r\ndebug1: channel 0: new [client-session]\r\ndebug1: Entering interactive session.\r\ndebug1: pledge: network\r\ndebug1: Sending environment.\r\ndebug1: Sending env LANG = en_US.UTF-8\r\ndebug1: Sending command: scp -v -t /dev/shm/test_server.bin\r\n/opt/octavia-tempest-plugin/test_server.bin: No such file or directory\nSink: \x01scp: /opt/octavia-tempest-plugin/test_server.bin: No such file or directory\ndebug1: client_input_channel_req: channel 0 rtype exit-status reply 0\r\ndebug1: channel 0: free: client-session, nchannels 1\r\ndebug1: fd 0 clearing O_NONBLOCK\r\ndebug1: fd 2 clearing O_NONBLOCK\r\nTransferred: sent 2472, received 1328 bytes, in 0.1 seconds\r\nBytes per second: sent 40751.6, received 21892.5\r\ndebug1: Exit status 1\r\n"
stderr:
None

summary:

- Octaia TrafficOperationsScenarioTest tests fail or timeout
+ Octavia TrafficOperationsScenarioTest tests fail or timeout

Revision history for this message

Gregory Thiemonge (gthiemonge) wrote on 2021-02-01:

#2

The fix for this issue was merged yesterday as part of https://review.opendev.org/c/openstack/tripleo-quickstart/+/770840

(Configuration of load_balancer.test_server_path in octavia-tempest-plugin)

Revision history for this message

Slawek Kaplonski (slaweq) wrote on 2021-02-03:

#3

Download full text (4.7 KiB)

@Gregory, I think it may be different issue. Recently every day setUp() of one class is failing. See: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master/c856c73/logs/undercloud/var/log/tempest/stestr_results.html.gz

Stacktrace:

ft1.1: setUpClass (octavia_tempest_plugin.tests.scenario.v2.test_traffic_ops.TrafficOperationsScenarioTest)testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tempest/test.py", line 188, in setUpClass
    six.reraise(etype, value, trace)
  File "/usr/local/lib/python3.6/site-packages/six.py", line 703, in reraise
    raise value
  File "/usr/lib/python3.6/site-packages/tempest/test.py", line 181, in setUpClass
    cls.resource_setup()
  File "/usr/lib/python3.6/site-packages/octavia_tempest_plugin/tests/scenario/v2/test_traffic_ops.py", line 53, in resource_setup
    super(TrafficOperationsScenarioTest, cls).resource_setup()
  File "/usr/lib/python3.6/site-packages/octavia_tempest_plugin/tests/test_base.py", line 689, in resource_setup
    cls.webserver1_response)
  File "/usr/lib/python3.6/site-packages/octavia_tempest_plugin/tests/test_base.py", line 877, in _install_start_webserver
    stdout, stderr)
tempest.lib.exceptions.CommandFailed: Command 'scp -v -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o ConnectTimeout=5 -o ConnectionAttempts=20 -i /tmp/tmpu8v8o6mm /opt/octavia-tempest-plugin/test_server.bin cirros@192.168.24.114:/dev/shm/test_server.bin' returned non-zero exit status 1.
stdout:
b"Executing: program /usr/bin/ssh host 192.168.24.114, user cirros, command scp -v -t /dev/shm/test_server.bin\nOpenSSH_8.0p1, OpenSSL 1.1.1g FIPS 21 Apr 2020\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf\r\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\r\ndebug1: configuration requests final Match pass\r\ndebug1: re-parsing configuration\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf\r\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\r\ndebug1: Connecting to 192.168.24.114 [192.168.24.114] port 22.\r\ndebug1: fd 3 clearing O_NONBLOCK\r\ndebug1: Connection established.\r\ndebug1: identity file /tmp/tmpu8v8o6mm type -1\r\ndebug1: identity file /tmp/tmpu8v8o6mm-cert type -1\r\ndebug1: Local version string SSH-2.0-OpenSSH_8.0\r\ndebug1: Remote protocol version 2.0, remote software version dropbear_2015.67\r\ndebug1: no match: dropbear_2015.67\r\ndebug1: Authenticating to 192.168.24.114:22 as 'cirros'\r\ndebug1: SSH2_MSG_KEXINIT sent\r\ndebug1: SSH2_MSG_KEXINIT received\r\ndebug1: kex: algorithm: <email address hidden>\r\ndebug1: kex: host key algorithm: ecdsa-sha2-nistp521\r\ndebug1: kex: server->client cipher: aes256-ctr MAC: hmac-sha2-256 compression: none\r\ndebug1: kex: client->server cipher: aes256-ctr MAC: hmac-sha2-256 compression: none\r\ndebug1: kex: curve25519-sha25...

@Gregory, I think it may be different issue. Recently every day setUp() of one class is failing. See: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master/c856c73/logs/undercloud/var/log/tempest/stestr_results.html.gz

Stacktrace:

ft1.1: setUpClass (octavia_tempest_plugin.tests.scenario.v2.test_traffic_ops.TrafficOperationsScenarioTest)testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tempest/test.py", line 188, in setUpClass
    six.reraise(etype, value, trace)
  File "/usr/local/lib/python3.6/site-packages/six.py", line 703, in reraise
    raise value
  File "/usr/lib/python3.6/site-packages/tempest/test.py", line 181, in setUpClass
    cls.resource_setup()
  File "/usr/lib/python3.6/site-packages/octavia_tempest_plugin/tests/scenario/v2/test_traffic_ops.py", line 53, in resource_setup
    super(TrafficOperationsScenarioTest, cls).resource_setup()
  File "/usr/lib/python3.6/site-packages/octavia_tempest_plugin/tests/test_base.py", line 689, in resource_setup
    cls.webserver1_response)
  File "/usr/lib/python3.6/site-packages/octavia_tempest_plugin/tests/test_base.py", line 877, in _install_start_webserver
    stdout, stderr)
tempest.lib.exceptions.CommandFailed: Command 'scp -v -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o ConnectTimeout=5 -o ConnectionAttempts=20 -i /tmp/tmpu8v8o6mm /opt/octavia-tempest-plugin/test_server.bin cirros@192.168.24.114:/dev/shm/test_server.bin' returned non-zero exit status 1.
stdout:
b"Executing: program /usr/bin/ssh host 192.168.24.114, user cirros, command scp -v -t /dev/shm/test_server.bin\nOpenSSH_8.0p1, OpenSSL 1.1.1g FIPS  21 Apr 2020\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf\r\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\r\ndebug1: configuration requests final Match pass\r\ndebug1: re-parsing configuration\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf\r\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\r\ndebug1: Connecting to 192.168.24.114 [192.168.24.114] port 22.\r\ndebug1: fd 3 clearing O_NONBLOCK\r\ndebug1: Connection established.\r\ndebug1: identity file /tmp/tmpu8v8o6mm type -1\r\ndebug1: identity file /tmp/tmpu8v8o6mm-cert type -1\r\ndebug1: Local version string SSH-2.0-OpenSSH_8.0\r\ndebug1: Remote protocol version 2.0, remote software version dropbear_2015.67\r\ndebug1: no match: dropbear_2015.67\r\ndebug1: Authenticating to 192.168.24.114:22 as 'cirros'\r\ndebug1: SSH2_MSG_KEXINIT sent\r\ndebug1: SSH2_MSG_KEXINIT received\r\ndebug1: kex: algorithm: curve25519-sha256@libssh.org\r\ndebug1: kex: host key algorithm: ecdsa-sha2-nistp521\r\ndebug1: kex: server->client cipher: aes256-ctr MAC: hmac-sha2-256 compression: none\r\ndebug1: kex: client->server cipher: aes256-ctr MAC: hmac-sha2-256 compression: none\r\ndebug1: kex: curve25519-sha256@libssh.org need=32 dh_need=32\r\ndebug1: kex: curve25519-sha256@libssh.org need=32 dh_need=32\r\ndebug1: expecting SSH2_MSG_KEX_ECDH_REPLY\r\ndebug1: Server host key: ecdsa-sha2-nistp521 SHA256:UxForkUFr7w64fRFbLyYpWzMedTYhRyhLCqYwfxFFOg\r\nWarning: Permanently added '192.168.24.114' (ECDSA) to the list of known hosts.\r\ndebug1: rekey out after 4294967296 blocks\r\ndebug1: SSH2_MSG_NEWKEYS sent\r\ndebug1: expecting SSH2_MSG_NEWKEYS\r\ndebug1: SSH2_MSG_NEWKEYS received\r\ndebug1: rekey in after 4294967296 blocks\r\ndebug1: Will attempt key: /tmp/tmpu8v8o6mm  explicit\r\ndebug1: SSH2_MSG_SERVICE_ACCEPT received\r\ndebug1: Authentications that can continue: publickey,password\r\ndebug1: Next authentication method: publickey\r\ndebug1: Trying private key: /tmp/tmpu8v8o6mm\r\ndebug1: Authentication succeeded (publickey).\r\nAuthenticated to 192.168.24.114 ([192.168.24.114]:22).\r\ndebug1: channel 0: new [client-session]\r\ndebug1: Entering interactive session.\r\ndebug1: pledge: network\r\ndebug1: Sending environment.\r\ndebug1: Sending env LANG = en_US.UTF-8\r\ndebug1: Sending command: scp -v -t /dev/shm/test_server.bin\r\n/opt/octavia-tempest-plugin/test_server.bin: No such file or directory\nSink: \x01scp: /opt/octavia-tempest-plugin/test_server.bin: No such file or directory\ndebug1: client_input_channel_req: channel 0 rtype exit-status reply 0\r\ndebug1: channel 0: free: client-session, nchannels 1\r\ndebug1: fd 0 clearing O_NONBLOCK\r\ndebug1: fd 2 clearing O_NONBLOCK\r\nTransferred: sent 2472, received 1336 bytes, in 0.1 seconds\r\nBytes per second: sent 36569.6, received 19764.1\r\ndebug1: Exit status 1\r\n"
stderr:
None

Revision history for this message

Arx Cruz (arxcruz) wrote on 2021-02-04:

#4

The SSH issue is fixed by https://review.opendev.org/c/openstack/tripleo-quickstart/+/770840 and https://review.rdoproject.org/r/#/c/31817/ however, the test is failing with timeout in some tests [1]:

{0} octavia_tempest_plugin.tests.scenario.v2.test_traffic_ops.TrafficOperationsScenarioTest.test_source_ip_udp_traffic [570.473338s] ... FAILED

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/usr/lib/python3.6/site-packages/octavia_tempest_plugin/tests/scenario/v2/test_traffic_ops.py", line 896, in test_source_ip_udp_traffic
        traffic_member_count=1, persistent=False)
      File "/usr/lib/python3.6/site-packages/octavia_tempest_plugin/tests/scenario/v2/test_traffic_ops.py", line 193, in _test_basic_traffic
        CONF.load_balancer.check_timeout)
      File "/usr/lib/python3.6/site-packages/octavia_tempest_plugin/tests/waiters.py", line 96, in wait_for_status
        raise exceptions.TimeoutException(message)
    tempest.lib.exceptions.TimeoutException: Request timed out
    Details: (TrafficOperationsScenarioTest:test_source_ip_udp_traffic) show_loadbalancer provisioning_status failed to update to ACTIVE within the required time 120. Current status of show_loadbalancer: PENDING_UPDATE

1 - https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-standalone-master/bb9415c/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz

Marios Andreou (marios-b) on 2021-03-17

Changed in tripleo:
milestone:	wallaby-3 → wallaby-rc1

Marios Andreou (marios-b) on 2021-05-06

Changed in tripleo:
milestone:	wallaby-rc1 → xena-1

Sagi (Sergey) Shnaidman (sshnaidm) on 2021-05-26

Changed in tripleo:
status:	Triaged → Fix Released

tripleo

Octavia TrafficOperationsScenarioTest tests fail or timeout

Bug Description

Other bug subscribers

Remote bug watches