ovn migration create-resources is not stable

Bug #1945299 reported by Eduardo Olivares
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Medium
Unassigned

Bug Description

Reported at: https://bugzilla.redhat.com/show_bug.cgi?id=2008425

As part of the ovs2ovn migration process, some workload is created before the migration begins. The intention is to verify that those workloads are healthy after the migration.
The script that creates those workloads checks they are pingable and ssh'able. Sometimes the ssh fails because the created VMs did not get the public keys during the cloud-init phase:
+ ssh -i /home/stack/ovn_migration/pre_migration_resources/ovn_migration_ssh_key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null cirros@10.0.0.243 date
Warning: Permanently added '10.0.0.243' (ECDSA) to the list of known hosts.
Permission denied, please try again.
Permission denied, please try again.
cirros@10.0.0.243: Permission denied (publickey,password).
+ rc=255
+ echo 'Done with the resource creation : exiting with 255'
Done with the resource creation : exiting with 255
+ exit 255

It seems the issue is related to a problem in cirros 0.4.0, which was resolved in a later release of cirros:
https://github.com/cirros-dev/cirros/pull/11/commits/e40bcd2964aa496a9d03e1aaf95cf7a86938f129
So, using cirros 0.5.2 could stabilize this script.

This script can be found in two different formats:
https://opendev.org/openstack/neutron/src/branch/master/tools/ovn_migration/tripleo_environment/playbooks/roles/resources/create/templates/create-resources.sh.j2
https://opendev.org/openstack/neutron/src/branch/master/tools/ovn_migration/infrared/tripleo-ovn-migration/roles/create-resources/templates/create-resources.sh.j2

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/811317

Changed in neutron:
status: New → In Progress
tags: added: ovn
Changed in neutron:
status: In Progress → Triaged
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/811317
Committed: https://opendev.org/openstack/neutron/commit/459f63439beb81a33e5e65d0c54c61832186f42e
Submitter: "Zuul (22348)"
Branch: master

commit 459f63439beb81a33e5e65d0c54c61832186f42e
Author: Eduardo Olivares <email address hidden>
Date: Tue Sep 28 10:51:51 2021 +0200

    Replace cirros 0.4.0 by 0.5.2 in ovn migration create-resources.sh.j2

    Some VMs are created before the ovn mgiration process starts in order to
    verify they are healthy after the migration
    Sometimes these VMs are not accessible via ssh due to an issue in cirros
    0.4.0 that was fixed in a later release [1]

    Closes-Bug: #1945299
    [1] https://github.com/cirros-dev/cirros/pull/11

    Change-Id: Ib133b5e1bed19aeac8514e3c6690ca768991bbd4

Changed in neutron:
status: Triaged → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/xena)

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/811340

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/811341

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/811342

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/811343

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/ussuri)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/811343
Committed: https://opendev.org/openstack/neutron/commit/38dc5d3a641503985e0f8ca1b9cd040e5775f347
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 38dc5d3a641503985e0f8ca1b9cd040e5775f347
Author: Eduardo Olivares <email address hidden>
Date: Tue Sep 28 10:51:51 2021 +0200

    Replace cirros 0.4.0 by 0.5.2 in ovn migration create-resources.sh.j2

    Some VMs are created before the ovn mgiration process starts in order to
    verify they are healthy after the migration
    Sometimes these VMs are not accessible via ssh due to an issue in cirros
    0.4.0 that was fixed in a later release [1]

    Closes-Bug: #1945299
    [1] https://github.com/cirros-dev/cirros/pull/11

    Change-Id: Ib133b5e1bed19aeac8514e3c6690ca768991bbd4
    (cherry picked from commit 459f63439beb81a33e5e65d0c54c61832186f42e)

tags: added: in-stable-ussuri
tags: added: in-stable-victoria
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/victoria)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/811342
Committed: https://opendev.org/openstack/neutron/commit/c0147aa0b988f8354cd7d407a08eeb0e6a43bdd6
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit c0147aa0b988f8354cd7d407a08eeb0e6a43bdd6
Author: Eduardo Olivares <email address hidden>
Date: Tue Sep 28 10:51:51 2021 +0200

    Replace cirros 0.4.0 by 0.5.2 in ovn migration create-resources.sh.j2

    Some VMs are created before the ovn mgiration process starts in order to
    verify they are healthy after the migration
    Sometimes these VMs are not accessible via ssh due to an issue in cirros
    0.4.0 that was fixed in a later release [1]

    Closes-Bug: #1945299
    [1] https://github.com/cirros-dev/cirros/pull/11

    Change-Id: Ib133b5e1bed19aeac8514e3c6690ca768991bbd4
    (cherry picked from commit 459f63439beb81a33e5e65d0c54c61832186f42e)

tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/811341
Committed: https://opendev.org/openstack/neutron/commit/d2b28dee76960530a9fb0398f9c4a5599b6613a6
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit d2b28dee76960530a9fb0398f9c4a5599b6613a6
Author: Eduardo Olivares <email address hidden>
Date: Tue Sep 28 10:51:51 2021 +0200

    Replace cirros 0.4.0 by 0.5.2 in ovn migration create-resources.sh.j2

    Some VMs are created before the ovn mgiration process starts in order to
    verify they are healthy after the migration
    Sometimes these VMs are not accessible via ssh due to an issue in cirros
    0.4.0 that was fixed in a later release [1]

    Closes-Bug: #1945299
    [1] https://github.com/cirros-dev/cirros/pull/11

    Change-Id: Ib133b5e1bed19aeac8514e3c6690ca768991bbd4
    (cherry picked from commit 459f63439beb81a33e5e65d0c54c61832186f42e)

tags: added: in-stable-xena
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/811340
Committed: https://opendev.org/openstack/neutron/commit/5fbd86fd8f8096e7d823ff8bbe7c134c720e71fc
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 5fbd86fd8f8096e7d823ff8bbe7c134c720e71fc
Author: Eduardo Olivares <email address hidden>
Date: Tue Sep 28 10:51:51 2021 +0200

    Replace cirros 0.4.0 by 0.5.2 in ovn migration create-resources.sh.j2

    Some VMs are created before the ovn mgiration process starts in order to
    verify they are healthy after the migration
    Sometimes these VMs are not accessible via ssh due to an issue in cirros
    0.4.0 that was fixed in a later release [1]

    Closes-Bug: #1945299
    [1] https://github.com/cirros-dev/cirros/pull/11

    Change-Id: Ib133b5e1bed19aeac8514e3c6690ca768991bbd4
    (cherry picked from commit 459f63439beb81a33e5e65d0c54c61832186f42e)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 19.0.0.0rc2

This issue was fixed in the openstack/neutron 19.0.0.0rc2 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 16.4.2

This issue was fixed in the openstack/neutron 16.4.2 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 17.3.0

This issue was fixed in the openstack/neutron 17.3.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 18.2.0

This issue was fixed in the openstack/neutron 18.2.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 20.0.0.0rc1

This issue was fixed in the openstack/neutron 20.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/networking-ovn train-eol

This issue was fixed in the openstack/networking-ovn train-eol release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.