Excessive warnings in nova-compute logs about unexpected network-vif-unplugged events during live migration

Bug #1819764 reported by Matt Riedemann
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Matt Riedemann
Rocky
Fix Committed
Medium
Matt Riedemann
Stein
Fix Committed
Medium
Matt Riedemann

Bug Description

Seen here:

http://logs.openstack.org/63/631363/22/check/nova-grenade-live-migration/1a03cbc/logs/subnode-2/screen-n-cpu.txt.gz?level=WARNING

Mar 11 07:50:38.515198 ubuntu-xenial-ovh-bhs1-0003641906 nova-compute[3321]: WARNING nova.compute.manager [req-883bec65-937b-4ed6-9d05-f0282ed17b75 req-11ffd59a-bf70-4cec-9678-36d3335ef766 service nova] [instance: faf3444e-f788-4ff4-992a-b14ce53ef8f8] Received unexpected event network-vif-unplugged-85392fed-4d68-4e01-818a-e004a33d95b6 for instance with vm_state paused and task_state migrating.

There are 22 instances of that same type of warning in that compute log, and it's all over our CI jobs that run live migration tests:

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Received%20unexpected%20event%20network-vif-unplugged%5C%22%20AND%20message%3A%5C%22and%20task_state%20migrating.%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d

We don't listen for network-vif-unplugged events during live migration and the unplug events are expected so we shouldn't warn about those.

We see quite a bit of this for other move operations like resize revert and shelve offload as well, and hard reboots, but we could deal with those separately:

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Received%20unexpected%20event%20network-vif-unplugged%5C%22%20AND%20NOT%20message%3A%5C%22and%20task_state%20None%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/642877

Changed in nova:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/642877
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=521e59224e8a595af66ec898f1bb739a9dbe7d97
Submitter: Zuul
Branch: master

commit 521e59224e8a595af66ec898f1bb739a9dbe7d97
Author: Matt Riedemann <email address hidden>
Date: Tue Mar 12 16:11:04 2019 -0400

    Don't warn on network-vif-unplugged event during live migration

    The "network-vif-unplugged" event is expected during live migration
    and nothing listens for it so we should not log a warning in that case.

    Change-Id: I8fd8df211670f1abbcb7b496e62589295922bdc1
    Closes-Bug: #1819764

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.openstack.org/650060

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/stein)

Reviewed: https://review.openstack.org/650060
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=17e6ca727207a21028f2878a1e01047c1be2ad36
Submitter: Zuul
Branch: stable/stein

commit 17e6ca727207a21028f2878a1e01047c1be2ad36
Author: Matt Riedemann <email address hidden>
Date: Tue Mar 12 16:11:04 2019 -0400

    Don't warn on network-vif-unplugged event during live migration

    The "network-vif-unplugged" event is expected during live migration
    and nothing listens for it so we should not log a warning in that case.

    Change-Id: I8fd8df211670f1abbcb7b496e62589295922bdc1
    Closes-Bug: #1819764
    (cherry picked from commit 521e59224e8a595af66ec898f1bb739a9dbe7d97)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/651797

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/rocky)

Reviewed: https://review.openstack.org/651797
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b8ca269a679ceecfcfa4e95ded148df07e8dd3b2
Submitter: Zuul
Branch: stable/rocky

commit b8ca269a679ceecfcfa4e95ded148df07e8dd3b2
Author: Matt Riedemann <email address hidden>
Date: Tue Mar 12 16:11:04 2019 -0400

    Don't warn on network-vif-unplugged event during live migration

    The "network-vif-unplugged" event is expected during live migration
    and nothing listens for it so we should not log a warning in that case.

    Change-Id: I8fd8df211670f1abbcb7b496e62589295922bdc1
    Closes-Bug: #1819764
    (cherry picked from commit 521e59224e8a595af66ec898f1bb739a9dbe7d97)
    (cherry picked from commit 17e6ca727207a21028f2878a1e01047c1be2ad36)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 19.0.1

This issue was fixed in the openstack/nova 19.0.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 18.2.1

This issue was fixed in the openstack/nova 18.2.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 20.0.0.0rc1

This issue was fixed in the openstack/nova 20.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.