tripleo

[Train][CentOS7/8][fs020] cold migration and resize tempest tests failing

Bug #1923403 reported by yatin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Unassigned
tripleo wallaby-rc1

Bug Description

Following tempest tests are failing since 27th March:-
tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_resize
tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_cold_migration

Failing with below traceback:-

Captured traceback:
~~~~~~~~~~~~~~~~~~~
{3} tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_resize [393.569761s] ... FAILED

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/usr/lib/python2.7/site-packages/tempest/common/utils/__init__.py", line 89, in wrapper
        return f(*func_args, **func_kwargs)
      File "/usr/lib/python2.7/site-packages/tempest/scenario/test_network_advanced_server_ops.py", line 196, in test_server_connectivity_resize
        'VERIFY_RESIZE')
      File "/usr/lib/python2.7/site-packages/tempest/common/waiters.py", line 96, in wait_for_server_status
        raise lib_exc.TimeoutException(message)
    tempest.lib.exceptions.TimeoutException: Request timed out
    Details: (TestNetworkAdvancedServerOps:test_server_connectivity_resize) Server 83bc9d74-97dc-4bd3-860d-b91ac34846b7 failed to reach VERIFY_RESIZE status and task state "None" within the required time (300 s). Current status: ACTIVE. Current task state: None.
{3} tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_cold_migration [408.793853s] ... FAILED

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/usr/lib/python2.7/site-packages/tempest/common/utils/__init__.py", line 89, in wrapper
        return f(*func_args, **func_kwargs)
      File "/usr/lib/python2.7/site-packages/tempest/scenario/test_network_advanced_server_ops.py", line 229, in test_server_connectivity_cold_migration
        'VERIFY_RESIZE')
      File "/usr/lib/python2.7/site-packages/tempest/common/waiters.py", line 96, in wait_for_server_status
        raise lib_exc.TimeoutException(message)
    tempest.lib.exceptions.TimeoutException: Request timed out
    Details: (TestNetworkAdvancedServerOps:test_server_connectivity_cold_migration) Server a1972a7a-4758-4168-8c2b-9fd16a1e3c18 failed to reach VERIFY_RESIZE status and task state "None" within the required time (300 s). Current status: ACTIVE. Current task state: None.

Nova compute reports:-
2021-03-27 07:15:13.589 7 ERROR oslo_messaging.rpc.server [req-17af7b03-e541-4f78-bf07-db30b340edf5 98c7f53e74a94e22a41031fc2f6e9d51 07c9ff470ac74345a7cf4f3bdc6a4f20 - default default] Exception during message handling: nova.exception.ResizeError: Resize error: not able to execute ssh command: Unexpected error while running command.
Command: ssh -o BatchMode=yes 172.17.0.51 mkdir -p /var/lib/nova/instances/723871da-467d-4185-aadb-f31d01285b18
Exit code: 255
Stdout: ''
Stderr: 'Host key verification failed.\r\n'

Log link:-
https://logserver.rdoproject.org/openstack-periodic-integration-stable3/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-train/cabe901/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz
https://logserver.rdoproject.org/openstack-periodic-integration-stable3/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-train/cabe901/logs/overcloud-novacompute-0/var/log/containers/nova/nova-compute.log.txt.gz

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-train
https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-train

yatin (yatinkarel)
Changed in tripleo:
status: New → Triaged
importance: Undecided → Critical
milestone: none → wallaby-rc1
tags: added: promotion-blocker
Revision history for this message
Martin Schuppert (mschuppert) wrote :

looks like the issue is that /etc/ssh/ssh_known_hosts files are empty

https://logserver.rdoproject.org/openstack-periodic-integration-stable3/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-train/cabe901/logs/overcloud-novacompute-0/etc/ssh/ssh_known_hosts.gz

Revision history for this message
wes hayutin (weshayutin) wrote :

added to skip list.. still an issue https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/786094

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-common (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/tripleo-common/+/786103

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-ansible (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/786159

Changed in tripleo:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-heat-templates (master)

Change abandoned by "Martin Schuppert <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/786089
Reason: Abandon in favor of https://review.opendev.org/c/openstack/tripleo-ansible/+/786159

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-ansible (master)

Reviewed: https://review.opendev.org/c/openstack/tripleo-ansible/+/786159
Committed: https://opendev.org/openstack/tripleo-ansible/commit/30df6f54375bb3ffa0cdd40ca219905b2fa92cf5
Submitter: "Zuul (22348)"
Branch: master

commit 30df6f54375bb3ffa0cdd40ca219905b2fa92cf5
Author: yatinkarel <email address hidden>
Date: Wed Apr 14 10:29:42 2021 +0530

    Use ansible_facts in tripleo_ssh_known_hosts role

    It was missed in original patch to switch to
    ansible_facts[1].

    For providing test value for 'ssh_host_key_rsa_public' overriding
    it in 'ansible_facts' with set_fact.

    [1] https://review.opendev.org/c/openstack/tripleo-ansible/+/776666

    Closes-Bug: #1923403
    Related-Bug: #1915761
    Change-Id: I9d5ae576c57eefe5496f9dda71e8eac23e45e89f

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-ansible (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/786233

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-ansible (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/786234

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-ansible (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/786355

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-common (stable/train)

Change abandoned by "yatin <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/tripleo-common/+/786103
Reason: In favor of https://review.opendev.org/q/I9d5ae576c57eefe5496f9dda71e8eac23e45e89f

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-ansible (stable/victoria)

Reviewed: https://review.opendev.org/c/openstack/tripleo-ansible/+/786233
Committed: https://opendev.org/openstack/tripleo-ansible/commit/c582b6584e78922403e296701c0ade252af7a8fe
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit c582b6584e78922403e296701c0ade252af7a8fe
Author: yatinkarel <email address hidden>
Date: Wed Apr 14 10:29:42 2021 +0530

    Use ansible_facts in tripleo_ssh_known_hosts role

    It was missed in original patch to switch to
    ansible_facts[1].

    For providing test value for 'ssh_host_key_rsa_public' overriding
    it in 'ansible_facts' with set_fact.

    [1] https://review.opendev.org/c/openstack/tripleo-ansible/+/776666

    Closes-Bug: #1923403
    Related-Bug: #1915761
    Change-Id: I9d5ae576c57eefe5496f9dda71e8eac23e45e89f
    (cherry picked from commit 30df6f54375bb3ffa0cdd40ca219905b2fa92cf5)

tags: added: in-stable-victoria
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-ansible (stable/ussuri)

Reviewed: https://review.opendev.org/c/openstack/tripleo-ansible/+/786234
Committed: https://opendev.org/openstack/tripleo-ansible/commit/b52dc1e7fd69fdc360f1281c53ac92abb7dddfee
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit b52dc1e7fd69fdc360f1281c53ac92abb7dddfee
Author: yatinkarel <email address hidden>
Date: Wed Apr 14 10:29:42 2021 +0530

    Use ansible_facts in tripleo_ssh_known_hosts role

    It was missed in original patch to switch to
    ansible_facts[1].

    For providing test value for 'ssh_host_key_rsa_public' overriding
    it in 'ansible_facts' with set_fact.

    [1] https://review.opendev.org/c/openstack/tripleo-ansible/+/776666

    Closes-Bug: #1923403
    Related-Bug: #1915761
    Change-Id: I9d5ae576c57eefe5496f9dda71e8eac23e45e89f
    (cherry picked from commit 30df6f54375bb3ffa0cdd40ca219905b2fa92cf5)

tags: added: in-stable-ussuri
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-ansible (stable/train)

Reviewed: https://review.opendev.org/c/openstack/tripleo-ansible/+/786355
Committed: https://opendev.org/openstack/tripleo-ansible/commit/a29918a4aff5802f8c86d8f79b3846da755c4aa2
Submitter: "Zuul (22348)"
Branch: stable/train

commit a29918a4aff5802f8c86d8f79b3846da755c4aa2
Author: yatinkarel <email address hidden>
Date: Wed Apr 14 10:29:42 2021 +0530

    Use ansible_facts in tripleo_ssh_known_hosts role

    It was missed in original patch to switch to
    ansible_facts[1].

    For providing test value for 'ssh_host_key_rsa_public' overriding
    it in 'ansible_facts' with set_fact.

    [1] https://review.opendev.org/c/openstack/tripleo-ansible/+/776666

    Closes-Bug: #1923403
    Related-Bug: #1915761
    Change-Id: I9d5ae576c57eefe5496f9dda71e8eac23e45e89f
    (cherry picked from commit 30df6f54375bb3ffa0cdd40ca219905b2fa92cf5)
    (cherry picked from commit b52dc1e7fd69fdc360f1281c53ac92abb7dddfee)

tags: added: in-stable-train
Revision history for this message
yatin (yatinkarel) wrote :

Fix merged https://review.opendev.org/q/I9d5ae576c57eefe5496f9dda71e8eac23e45e89f,
skiplist reverted https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/786104, closing it.
Tested with test patch https://review.rdoproject.org/r/c/rdo-jobs/+/33261

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-ansible 3.1.0

This issue was fixed in the openstack/tripleo-ansible 3.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-ansible 2.3.0

This issue was fixed in the openstack/tripleo-ansible 2.3.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-ansible 0.8.0

This issue was fixed in the openstack/tripleo-ansible 0.8.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-ansible 1.5.4

This issue was fixed in the openstack/tripleo-ansible 1.5.4 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.