libvirt: "Instance failed to shutdown in 60 seconds." in the gate
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Confirmed
|
Medium
|
Unassigned |
Bug Description
I noticed today that when shutting down an instance, like for a snapshot test, it's sometimes taking a full 60 seconds to wait for the graceful shutdown before just destroying the guest:
May 16 19:47:26.017509 ubuntu-
This could contribute to overall job/test timeouts since a lot of tempest test wait operations are using the build_timeout config and set to 196 seconds. I'm not sure why these guests aren't stopping, but they should be since they shouldn't be doing anything. Looking at logstash affected branches it seems this started in Stein.
This is with libvirt 4.0.0-1ubuntu8.10 and qemu 1:2.11+
I don't see any errors in the libvirtd logs, the shutdown request returns with code 0.
Hmm, I'm completely puzzled as well, on the root cause. And indeed, the libvirtd log[1] is utterly boring with the shutdown request returing "0".
2019-05-16 19:46:25.777+0000: 10502: debug : virThreadJobCle ar:121 : Thread 10502 (virNetServerHa ndleJob) finished job remoteDispatchD omainShutdown with ret=0
And I'm not even sure if I can reproduce this problem locally to track it down.
Nothing in the syslog: http:// logs.openstack. org/56/ 656656/ 12/check/ nova-multi- cell/6a403ba/ compute1/ logs/syslog. txt.gz
Maybe some clues in the guest boot log? Do we capture it? Let's look...okay, there is this compressed, system-wide journal log[2], maybe that has it? Let's extract it and examine:
$> gunzip devstack. journal. xz.gz systemd- journal- remote <(xzcat ./devstack. journal. xz) -o output.journal
$> /lib/systemd/
Finishing after writing 6333 entries
`grep`ing around `sudo journalctl --file ./output.journal` didn't yield anything interesting either. Still not clear, I see a guest serial console log (from boot time).
[1]http:// logs.openstack. org/56/ 656656/ 12/check/ nova-multi- cell/6a403ba/ compute1/ logs/libvirt/ libvirtd_ log.txt. gz logs.openstack. org/56/ 656656/ 12/check/ nova-multi- cell/6a403ba/ compute1/ logs/devstack. journal. xz.gz
[2] http://