Noisy DEBUG log: Defaulting the value of the field 'numa_membw_values' to None in MetricPayload

[Balazs Gibizer]

There are plenty of similar DEBUG log is visible in the nova log:

DEBUG nova.notifications.objects.base [None req-af08f64a-756e-4b43-8357-fd879d3bbbc9 None None] Defaulting the value of the field 'numa_membw_values' to None in MetricPayload due to 'Cannot load 'numa_membw_values' in the base class' {{(pid=145045) populate_schema /opt/stack/nova/nova/notifications/objects/base.py:142}}

This logs coming from the notification sending code path [1] that tries to populate the notification payload from a nova OVO and encounters an unset / non-lazy loadable field in the source OVO.

As there are fields like MonitorMetric.numa_membw_values that are notorious unset in many circumstances but used in the MetricPayload we got excessive debug logging.

[1] https://github.com/openstack/nova/blob/cccba02be8dda0350a215de78b0acde3dbf19716/nova/notifications/objects/base.py#L138

[Matthew Edmonds (edmondsw)]

As near as I can tell, numa_membw_values is never set. Maybe in some out-of-tree monitor? But it's certainly not set in https://github.com/openstack/nova/blob/cccba02be8dda0350a215de78b0acde3dbf19716/nova/compute/monitors/cpu/virt_driver.py#L42 which is where I'd expect it.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/595810

[Matt Riedemann (mriedem)]

@Matthew: because the implementation stalled halfway:

https://review.openstack.org/#/c/329783/

When we came to this point:

https://docs.openstack.org/nova/latest/contributor/policies.html#metrics-gathering

[Matt Riedemann (mriedem)]

Taking http://logs.openstack.org/97/579897/4/check/tempest-full/53d05dc/controller/logs/screen-n-cpu.txt.gz as a sample, the "Defaulting the value of the field" DEBUG log shows up over 1000 times.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/595810
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=375632cbc9019f0e4e887afb5d6e48978f2a1a26
Submitter: Zuul
Branch: master

commit 375632cbc9019f0e4e887afb5d6e48978f2a1a26
Author: Balazs Gibizer <email address hidden>
Date: Thu Aug 23 16:57:51 2018 +0200

    Remove noisy DEBUG log

    There is a DEBUG log coming from the notification sending code path
    that tries to populate the notification payload from a nova OVO and
    encounters an unset / non-lazy loadable field in the source OVO.
    As there are fields like MonitorMetric.numa_membw_values that are
    notorious unset in many circumstances but used in the MetricPayload
    we got excessive DEBUG logging from this code path.

    This patch removes the offending logging statement as soo far we failed
    to found cases based on this log where anything else we could do than
    what the code anyhow does, defaulting the payload field to None.

    Change-Id: I680e1d9860d839a76e229c4891933872388eb050
    Closes-Bug: #1788634

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/596264

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/596264
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=3724280716703bad54d6dc434b4b1679243113d9
Submitter: Zuul
Branch: stable/rocky

commit 3724280716703bad54d6dc434b4b1679243113d9
Author: Balazs Gibizer <email address hidden>
Date: Thu Aug 23 16:57:51 2018 +0200

    Remove noisy DEBUG log

    There is a DEBUG log coming from the notification sending code path
    that tries to populate the notification payload from a nova OVO and
    encounters an unset / non-lazy loadable field in the source OVO.
    As there are fields like MonitorMetric.numa_membw_values that are
    notorious unset in many circumstances but used in the MetricPayload
    we got excessive DEBUG logging from this code path.

    This patch removes the offending logging statement as soo far we failed
    to found cases based on this log where anything else we could do than
    what the code anyhow does, defaulting the payload field to None.

    Change-Id: I680e1d9860d839a76e229c4891933872388eb050
    Closes-Bug: #1788634
    (cherry picked from commit 375632cbc9019f0e4e887afb5d6e48978f2a1a26)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/600372

[Elod Illes (elod-illes)]

The problem seems to be there in stable/queens, too. (e.g. more than 1000 occurrences in http://logs.openstack.org/01/600101/1/check/tempest-full/25c9d13/controller/logs/screen-n-cpu.txt.gz )

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/600372
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=6ece888c790475e529163810a8f01b5ffa65c8ea
Submitter: Zuul
Branch: stable/queens

commit 6ece888c790475e529163810a8f01b5ffa65c8ea
Author: Balazs Gibizer <email address hidden>
Date: Thu Aug 23 16:57:51 2018 +0200

    Remove noisy DEBUG log

    There is a DEBUG log coming from the notification sending code path
    that tries to populate the notification payload from a nova OVO and
    encounters an unset / non-lazy loadable field in the source OVO.
    As there are fields like MonitorMetric.numa_membw_values that are
    notorious unset in many circumstances but used in the MetricPayload
    we got excessive DEBUG logging from this code path.

    This patch removes the offending logging statement as soo far we failed
    to found cases based on this log where anything else we could do than
    what the code anyhow does, defaulting the payload field to None.

    Change-Id: I680e1d9860d839a76e229c4891933872388eb050
    Closes-Bug: #1788634
    (cherry picked from commit 375632cbc9019f0e4e887afb5d6e48978f2a1a26)
    (cherry picked from commit 3724280716703bad54d6dc434b4b1679243113d9)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.6 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 18.0.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 19.0.0.0rc1 release candidate.