test_cold_migrate_unshelved_instance failing with cat: can't open '/mnt/timestamp': No such file or directory

[Lee Yarwood]

https://zuul.opendev.org/t/openstack/build/13400ea7d7af4dd88fca244b82301c79/log/job-output.txt#65297

2020-12-01 11:03:11.055150 | controller | 2020-12-01 10:52:58,178 102645 ERROR [tempest.lib.common.utils.linux.remote_client] (TestShelveInstance:test_cold_migrate_unshelved_instance) Executing command on 172.24.5.37 failed. Error: Command 'set -eu -o pipefail; PATH=$PATH:/sbin:/usr/sbin; sudo cat /mnt/timestamp', exit status: 1, stderr:
2020-12-01 11:03:11.055160 | controller | cat: can't open '/mnt/timestamp': No such file or directory

Add related test to Bug #1732428
https://review.opendev.org/c/openstack/tempest/+/743708

[Balazs Gibizer (balazs-gibizer)]

It seems it fails all the time https://zuul.opendev.org/t/openstack/builds?job_name=nova-multi-cell&branch=master So I making this critical as blocking the nova CI

[Balazs Gibizer (balazs-gibizer)]

A fix has been proposed in nova https://review.opendev.org/c/openstack/tempest/+/765079

[Balazs Gibizer (balazs-gibizer)]

And there is a logstash signature https://review.opendev.org/c/opendev/elastic-recheck/+/765108

[Lee Yarwood (lyarwood)]

This looks more like an issue with cross cell resize.

The following pastebin shows example qemu-img commands from a failing run where the final cold migration / resize ends up using the original image with a fresh overlay, instead of the snapshot disk from the source host:

http://paste.openstack.org/show/800637/

I'm assuming that we've not passed the snapshot_id correctly as part of the cross cell resize:

https://github.com/openstack/nova/blob/f0efcae6975a99044ef7052453f905f60fcecac6/nova/compute/manager.py#L5906

Skipping the test for now and adding a DNM debug change to troubleshoot this more.

[Lee Yarwood (lyarwood)]

https://review.opendev.org/c/openstack/nova/+/765141 skips the test in the nova-multi-cell job.

[Ghanshyam Mann (ghanshyammann)]

either we can disable it explicitly in nova-cell-job or disable it in devstack for this job but we need to add job var for that https://review.opendev.org/c/openstack/nova/+/765141

[Alexandre arents (aarents)]

Agree with Lee that it is more a bug in nova:
https://review.opendev.org/c/openstack/nova/+/765561
And tempest job is correct and reveal the issue.

[Balazs Gibizer (balazs-gibizer)]

The disablement of the test is merged https://review.opendev.org/c/openstack/nova/+/765141

[Martin Kopec (mkopec)]

gerrit doesn't update status of the bugs automatically again ... this is supposed to be fixed for nova by https://review.opendev.org/c/openstack/nova/+/765561

[Martin Kopec (mkopec)]

Based on the discussion above, it was agreed that the bug was on nova side (got fixed already) so marking this as Invalid for Tempest .. feel free to correct me

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 23.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/793356

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/nova/+/793373

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/nova/+/793373
Reason: stable/ussuri branch of openstack/nova transitioned to End of Life and is about to be deleted. To be able to do that, all open patches need to be abandoned.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/793356
Reason: stable/victoria branch of openstack/nova is about to be deleted. To be able to do that, all open patches need to be abandoned. Please cherry pick the patch to unmaintained/victoria if you want to further work on this patch.