functional tests in rocky randomly fail with "Build of instance was re-scheduled: Cannot modify readonly field uuid"

[Matt Riedemann]

Seen here:

https://561c74891cdc1994ef28-bf4df36ee9e72417a89c120068385812.ssl.cf2.rackcdn.com/690721/4/check/nova-tox-functional/5b7bec9/testr_results.html.gz

From logstash this is probably the earliest patch to hit it:

https://review.opendev.org/#/c/687563/

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22was%20re-scheduled%3A%20Cannot%20modify%20readonly%20field%20uuid%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d

Without something like this https://review.opendev.org/#/c/669545/ we can't tell where the original error is coming from though.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/669545
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=9062fd392374f546da58651a785f47c3906204dc
Submitter: Zuul
Branch: master

commit 9062fd392374f546da58651a785f47c3906204dc
Author: Gary Kotton <email address hidden>
Date: Sat Jul 6 00:31:40 2019 -0700

    Nova compute: add in log exception to help debug failures

    When the system is under load, there can be boot failures. One
    exception seen in the log file is:

    ''NoneType' object has no attribute 'status_code''

    There is no log trace which helps pinpoint this client call.
    Note that this is only one example.

    We now add in a LOG.exception to have the stack trace.

    Related-Bug: #1850682

    Change-Id: I2ab56005db363f81eecf07c3db62c267ba7f8ca9

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/692662

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/692662
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=35b5dc0d9793742d759a8cae2de33f450c75f2a1
Submitter: Zuul
Branch: stable/train

commit 35b5dc0d9793742d759a8cae2de33f450c75f2a1
Author: Gary Kotton <email address hidden>
Date: Sat Jul 6 00:31:40 2019 -0700

    Nova compute: add in log exception to help debug failures

    When the system is under load, there can be boot failures. One
    exception seen in the log file is:

    ''NoneType' object has no attribute 'status_code''

    There is no log trace which helps pinpoint this client call.
    Note that this is only one example.

    We now add in a LOG.exception to have the stack trace.

    Related-Bug: #1850682

    Change-Id: I2ab56005db363f81eecf07c3db62c267ba7f8ca9
    (cherry picked from commit 9062fd392374f546da58651a785f47c3906204dc)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/692936

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/692936
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=defe542eb130cd9ad621493fbf98a6a3458c4169
Submitter: Zuul
Branch: stable/stein

commit defe542eb130cd9ad621493fbf98a6a3458c4169
Author: Gary Kotton <email address hidden>
Date: Sat Jul 6 00:31:40 2019 -0700

    Nova compute: add in log exception to help debug failures

    When the system is under load, there can be boot failures. One
    exception seen in the log file is:

    ''NoneType' object has no attribute 'status_code''

    There is no log trace which helps pinpoint this client call.
    Note that this is only one example.

    We now add in a LOG.exception to have the stack trace.

    Related-Bug: #1850682

    Change-Id: I2ab56005db363f81eecf07c3db62c267ba7f8ca9
    (cherry picked from commit 9062fd392374f546da58651a785f47c3906204dc)
    (cherry picked from commit 35b5dc0d9793742d759a8cae2de33f450c75f2a1)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/rocky
Review: https://review.opendev.org/693009

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/693009
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=eab5da85eae72d7e6bd91813c6773e7347c4798b
Submitter: Zuul
Branch: stable/rocky

commit eab5da85eae72d7e6bd91813c6773e7347c4798b
Author: Gary Kotton <email address hidden>
Date: Sat Jul 6 00:31:40 2019 -0700

    Nova compute: add in log exception to help debug failures

    When the system is under load, there can be boot failures. One
    exception seen in the log file is:

    ''NoneType' object has no attribute 'status_code''

    There is no log trace which helps pinpoint this client call.
    Note that this is only one example.

    We now add in a LOG.exception to have the stack trace.

    Related-Bug: #1850682

    Change-Id: I2ab56005db363f81eecf07c3db62c267ba7f8ca9
    (cherry picked from commit 9062fd392374f546da58651a785f47c3906204dc)
    (cherry picked from commit 35b5dc0d9793742d759a8cae2de33f450c75f2a1)
    (cherry picked from commit defe542eb130cd9ad621493fbf98a6a3458c4169)