live block migration results in loss of console log

Bug #1203193 reported by Loganathan Parthipan on 2013-07-19
46
This bug affects 7 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Medium
Unassigned

Bug Description

I'm not sure if this has been reported before or is a known limitation. But the command

nova console-log <server>

after a live-migration with --block-migrate results in a zero console log. Prior to the live migration it hadn't been the case.

To recreate:

1. Boot an instance --> server A
2. nova console-log A --> console log is returned
3. As admin: nova live-migration --block-migrate A
4. As admin: nova show A --> confirms that the vm has moved
5. Check that it's accessible etc.
6. Now get the console log: nova console-log A --> returns empty

description: updated
description: updated
summary: - live migration results in loss of console log
+ live block migration results in loss of console log
Matt Riedemann (mriedem) wrote :

What kind of hypervisor are you using? libvirt?

I'm using libvirt with libvirt-type = qemu. There's no VMX support on the host hypervisor.

tags: added: libvirt
Yang Yu (yuyangbj) wrote :

Is not it reasonable for zero console log after live-migration? After live-migration, we do not restart the VM, so it is correct there is no console log.

The original contents of the console log is lost after a migration and that's the issue here.

Why should a VM user lose the ability to inspect the console log when a service provider decided to migrate the VM for maintenance when everything else is torn and setup at the destination host correctly?

Hua Zhang (zhhuabj) wrote :

After reading live-migration related codes in openstack side just now, I guess most likely it's a bug of kvm.
openstack only checks the configuration and configures network for the vm by those two methods (pre_live_migration & post_live_migration),
the rest thing is left to kvm to do by invoking it's virDomainMigrateToURI method of libvirtmod module which is realized using C language.

I don't see the code of virDomainMigrateToURI method, but I guess it's possible steps are as bellow:
1, share pubkey between the two hosts with ssh-copy id root@targethost from "sourcehost"
2, copy the vm's disk from the source host to the target host by: dd if=/dev/<vgname>/<lvname> | ssh root@targethost 'dd of=/dev/<vgname>/<lvname>'
3, dump the xml definitions virsh dumpxml <vm> dump.xml
4, transfer xml-file scp dump.xml root@targethost:/root/
5, define the machine on target virsh define dump.xml the machine is then created from the xml.
6, virsh start vm

so parthipan, could you help confirm following two things:
1) are you configuring the share ssh pubkey for root account ?
2) have you tried block live migration by libvirt's CLI directly, rather than openstack's CLI. like:
   virsh migrate --live --copy-storage-all --verbose <vm-name> qemu+ssh://<target-host>/system

GuoHui Liu (guohliu) wrote :

actually it is a bug of libvirt, we already opened a issue in libvirt community to tracking that issue. will post the url to here later.

Changed in nova:
status: New → Confirmed
importance: Undecided → Medium

I have registered a blueprint to slightly alter the way console logs are read in the libvirt driver. Please comment on this design. Thanks.

https://blueprints.launchpad.net/nova/+spec/support-rotated-console-logs-in-libvirt-driver

Changed in nova:
assignee: nobody → Loganathan Parthipan (parthipan)
tags: added: live-migration
tags: added: live-migrate
removed: live-migration

I have a feeling that this bug is not related to nova. It is libvirt's/hypervisor's job to copy every needed file to destination host, nova only triggers live migration operation. I believe that it should be solved as part of bug that GuoHui mentioned: https://bugzilla.redhat.com/show_bug.cgi?id=994882

Michael Still (mikal) wrote :

No action on this one in a very long time, so unassigning.

Changed in nova:
assignee: Loganathan Parthipan (parthipan) → nobody
Paul Murray (pmurray) on 2015-11-06
tags: added: live-migration
removed: live-migrate
lvmxh (shaohef) on 2015-11-17
Changed in nova:
assignee: nobody → lvmxh (shaohef)
lvmxh (shaohef) on 2015-12-03
Changed in nova:
assignee: lvmxh (shaohef) → nobody
stgleb (gstepanov) on 2016-02-19
Changed in nova:
assignee: nobody → stgleb (gstepanov)

Fix proposed to branch: master
Review: https://review.openstack.org/284674

Changed in nova:
status: Confirmed → In Progress
Kashyap Chamarthy (kashyapc) wrote :

[In reply to comment #9]

Pawel, seems like there's general agreement from libvirt upstream that this needs to be handled at a higher-layer than libvirt.

[Below is a comment from IRC, by Dan Berrangé, on this topic, from here: https://bugzilla.redhat.com/show_bug.cgi?id=994882#c3 ]
-------
"This bug is basically asking for the block migration functionality to extend to other qemu devices backed by files. This is pretty tricky in general, because with latest libvirt & qemu, qemu doesn't even get acess to the log files. It just gets given an anonymous pipe file descriptor, so it has no ability to migrate the log file even if it wanted to.

libvirt tries to only concern itself with what happens on a single-node
so moving stuff between nodes is left to the app using libvirt which knows better what to do".
-------

And, Pawel just agreed on #openstack-nova that it "sounds fair that higher-layer should handle this, yes".

Matt Riedemann (mriedem) on 2016-03-11
tags: added: console

Change abandoned by Michael Still (<email address hidden>) on branch: master
Review: https://review.openstack.org/284674
Reason: This code hasn't been updated in a long time, and is in merge conflict. I am going to abandon this review, but feel free to restore it if you're still working on this.

@stgleb: Are you still working on this bug, if not please unassign yourself and set it to 'confirmed'.

stgleb (gstepanov) on 2017-01-11
Changed in nova:
assignee: stgleb (gstepanov) → nobody
assignee: nobody → stgleb (gstepanov)
Changed in nova:
status: In Progress → Confirmed

@siva-radhakrishnan: Are you working on this bug still?

Maciej Szankin (mszankin) wrote :

This bug report has an assignee for a while now but there is no patch
for that. It looks like that the chance of getting a patch is low.
I'm going to remove the assignee to signal to others that they can take
over if they like.
If you want to work on this, please:
* add yourself as assignee AND
* set the status to "In Progress" AND
* provide a (WIP) patch within the next 2 weeks after that.
If you need assistance, reach out on the IRC channel #openstack-nova or
use the mailing list.

Changed in nova:
assignee: stgleb (gstepanov) → nobody
Adrien Cunin (adri2000) wrote :

I can reproduce this issue without block migration. Setup with /var/lib/nova/instances/ on shared NFS, using openstack server migrate --live ... --shared-migration ...

Would it be the same root cause?

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.