Excessive warnings in nova-compute logs about unexpected network-vif-unplugged events during live migration

Bug #1819764 reported by Matt Riedemann on 2019-03-12
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Medium
Matt Riedemann
Rocky
Medium
Matt Riedemann
Stein
Medium
Matt Riedemann

Bug Description

Seen here:

http://logs.openstack.org/63/631363/22/check/nova-grenade-live-migration/1a03cbc/logs/subnode-2/screen-n-cpu.txt.gz?level=WARNING

Mar 11 07:50:38.515198 ubuntu-xenial-ovh-bhs1-0003641906 nova-compute[3321]: WARNING nova.compute.manager [req-883bec65-937b-4ed6-9d05-f0282ed17b75 req-11ffd59a-bf70-4cec-9678-36d3335ef766 service nova] [instance: faf3444e-f788-4ff4-992a-b14ce53ef8f8] Received unexpected event network-vif-unplugged-85392fed-4d68-4e01-818a-e004a33d95b6 for instance with vm_state paused and task_state migrating.

There are 22 instances of that same type of warning in that compute log, and it's all over our CI jobs that run live migration tests:

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Received%20unexpected%20event%20network-vif-unplugged%5C%22%20AND%20message%3A%5C%22and%20task_state%20migrating.%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d

We don't listen for network-vif-unplugged events during live migration and the unplug events are expected so we shouldn't warn about those.

We see quite a bit of this for other move operations like resize revert and shelve offload as well, and hard reboots, but we could deal with those separately:

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Received%20unexpected%20event%20network-vif-unplugged%5C%22%20AND%20NOT%20message%3A%5C%22and%20task_state%20None%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d

Fix proposed to branch: master
Review: https://review.openstack.org/642877

Changed in nova:
status: Confirmed → In Progress

Reviewed: https://review.openstack.org/642877
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=521e59224e8a595af66ec898f1bb739a9dbe7d97
Submitter: Zuul
Branch: master

commit 521e59224e8a595af66ec898f1bb739a9dbe7d97
Author: Matt Riedemann <email address hidden>
Date: Tue Mar 12 16:11:04 2019 -0400

    Don't warn on network-vif-unplugged event during live migration

    The "network-vif-unplugged" event is expected during live migration
    and nothing listens for it so we should not log a warning in that case.

    Change-Id: I8fd8df211670f1abbcb7b496e62589295922bdc1
    Closes-Bug: #1819764

Changed in nova:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/650060
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=17e6ca727207a21028f2878a1e01047c1be2ad36
Submitter: Zuul
Branch: stable/stein

commit 17e6ca727207a21028f2878a1e01047c1be2ad36
Author: Matt Riedemann <email address hidden>
Date: Tue Mar 12 16:11:04 2019 -0400

    Don't warn on network-vif-unplugged event during live migration

    The "network-vif-unplugged" event is expected during live migration
    and nothing listens for it so we should not log a warning in that case.

    Change-Id: I8fd8df211670f1abbcb7b496e62589295922bdc1
    Closes-Bug: #1819764
    (cherry picked from commit 521e59224e8a595af66ec898f1bb739a9dbe7d97)

Reviewed: https://review.openstack.org/651797
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b8ca269a679ceecfcfa4e95ded148df07e8dd3b2
Submitter: Zuul
Branch: stable/rocky

commit b8ca269a679ceecfcfa4e95ded148df07e8dd3b2
Author: Matt Riedemann <email address hidden>
Date: Tue Mar 12 16:11:04 2019 -0400

    Don't warn on network-vif-unplugged event during live migration

    The "network-vif-unplugged" event is expected during live migration
    and nothing listens for it so we should not log a warning in that case.

    Change-Id: I8fd8df211670f1abbcb7b496e62589295922bdc1
    Closes-Bug: #1819764
    (cherry picked from commit 521e59224e8a595af66ec898f1bb739a9dbe7d97)
    (cherry picked from commit 17e6ca727207a21028f2878a1e01047c1be2ad36)

This issue was fixed in the openstack/nova 19.0.1 release.

This issue was fixed in the openstack/nova 18.2.1 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers