compute resource tracker errors regarding stale allocations should be logged at WARNING instead of DEBUG

[Lars Kellogg-Stedman]

We ran into a situation with a Pike deployment (upgraded from Ocata) in which new instances were failing to schedule on a target compute node that was empty. Nova-conductor would throw a NoValidHost exception without any details, and there were no other error messages related to the failure. It turns out this was due to a number of stale resource allocations on the target compute node.

If nova-compute finds stale allocations, it logs a message at DEBUG priority that says:

  There are allocations remaining against the source host that might need to be removed:

Since this is logged at DEBUG priority, the message will not typically be seen by operators in a production environment. However, stale allocations can prevent Nova from scheduling new instances on the compute node, leading to mysterious failures for which there are no obvious errors.

This message should be logged at WARNING priority so that it is visible.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/564964
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a54e5ed00f98b8536080647e5caa90d556d6e809
Submitter: Zuul
Branch: master

commit a54e5ed00f98b8536080647e5caa90d556d6e809
Author: Lars Kellogg-Stedman <email address hidden>
Date: Fri Apr 27 22:43:35 2018 -0400

    log stale allocations as WARNING instead of DEBUG

    Stale resource allocations can prevent Nova from scheduling new
    instances on a host that otherwise has sufficient resources available.
    Nova would log a message about this situation on the affected compute
    node, but it was logged at DEBUG priority and so would typically not
    be visible in a production environment.

    This commit modifies Nova to log the message as a warning instead.

    Change-Id: I347d2ab0077c4d9b62bc169af9e039d00ec214fa
    Closes-bug: #1767553

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/567271

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/567273

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/567273
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=12a3b698d17672ec0277dff4e96c44a92186aab1
Submitter: Zuul
Branch: stable/queens

commit 12a3b698d17672ec0277dff4e96c44a92186aab1
Author: Lars Kellogg-Stedman <email address hidden>
Date: Fri Apr 27 22:43:35 2018 -0400

    log stale allocations as WARNING instead of DEBUG

    Stale resource allocations can prevent Nova from scheduling new
    instances on a host that otherwise has sufficient resources available.
    Nova would log a message about this situation on the affected compute
    node, but it was logged at DEBUG priority and so would typically not
    be visible in a production environment.

    This commit modifies Nova to log the message as a warning instead.

    Change-Id: I347d2ab0077c4d9b62bc169af9e039d00ec214fa
    Closes-bug: #1767553
    (cherry picked from commit a54e5ed00f98b8536080647e5caa90d556d6e809)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/567271
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=852cf3bca7e4bf768f9c3e478e5b1e919df57977
Submitter: Zuul
Branch: stable/pike

commit 852cf3bca7e4bf768f9c3e478e5b1e919df57977
Author: Lars Kellogg-Stedman <email address hidden>
Date: Fri Apr 27 22:43:35 2018 -0400

    log stale allocations as WARNING instead of DEBUG

    Stale resource allocations can prevent Nova from scheduling new
    instances on a host that otherwise has sufficient resources available.
    Nova would log a message about this situation on the affected compute
    node, but it was logged at DEBUG priority and so would typically not
    be visible in a production environment.

    This commit modifies Nova to log the message as a warning instead.

    Change-Id: I347d2ab0077c4d9b62bc169af9e039d00ec214fa
    Closes-bug: #1767553
    (cherry picked from commit a54e5ed00f98b8536080647e5caa90d556d6e809)
    (cherry picked from commit 12a3b698d17672ec0277dff4e96c44a92186aab1)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.4 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 16.1.3 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 18.0.0.0b2 development milestone.