enabling serial console breaks live migration

[Chris Stone]

Version: openstack-nova-compute-2014.2.2-1.el7.noarch

Enabling serial console causes the live migration process to fail with the following error in nova-compute.log:

nova[12362]: 2015-05-14 21:57:21.108 ERROR nova.virt.libvirt.driver [-] [instance: a4d0bd8f-666e-4606-9e2a-4c583719719e] Live Migration failure: internal error: process exited while connecting to monitor: Warning: option deprecated, use lost_tick_policy property of kvm-pit instead.
qemu-kvm: -chardev socket,id=charserial0,host=1.2.3.4,port=10000,server,nowait: Failed to bind socket: Cannot assign requested address
qemu-kvm: -chardev socket,id=charserial0,host=1.2.3.4,port=10000,server,nowait: chardev: opening backend "socket" failed

The host IP is coming from the proxyclient_address setting which can't be 0.0.0.0, but even if it is changed to 0.0.0.0 the port number stays the same and will likely conflict on the destination. If there is no port conflict the live migration will succeed.

[Yaguang Tang (heut2008)]

before we find a way to fix this bug, we should document it somewhere to notify operators.

[Alexander Schmidt (alexs-h)]

I also ran into this problem using kilo code.

[Chris Stone (cstone-0)]

We used the same config post Kilo upgrade and got the same issue, so this is also present in the 2105.1.1 release in RDO as well.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/191035

[Alexander Schmidt (alexs-h)]

The patch solves the first part of the problem: using the proxyclient_address of the target host for the live migration.

Avoiding collisions on the ports used for the serial console will be a separate patch.

It would be great if other people could also test/verify the solution.

[Markus Zoeller (markus_z) (mzoeller)]

This is a serious issue, in my opinion. I add the "kilo-backport-potential" tag and set the importance to "High".

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/215102

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/191035
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=984cc474efc6ecbeb1240f49479b6439bc9a9416
Submitter: Jenkins
Branch: master

commit 984cc474efc6ecbeb1240f49479b6439bc9a9416
Author: Alexander Schmidt <email address hidden>
Date: Fri Jun 12 14:21:07 2015 +0200

    libvirt: enable live migration with serial console

    When migrating an instance that uses serial console, the domain XML
    must be changed to reflect the serial console configuration of the
    target host.

    Also prevent live migration when serial console is enabled and the
    libvirt version is not able to process modified domain XML.

    Change-Id: I654e444584124334e34d4673db6464cc2f2b822e
    Partial-Bug: #1455252

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/275800

[Markus Zoeller (markus_z) (mzoeller)]

The official tag "documentation" got changed to "doc" to match it
with the bug reports which get generated by merged commits with a
"DocImpact" in the commit message.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Markus Zoeller (markus_z) (<email address hidden>) on branch: master
Review: https://review.openstack.org/275800
Reason: Sahid is solving bug 1489853 in change https://review.openstack.org/301247

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/215102
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c8ceeb46c7589a9159c4ad2ad50ac19249981aa0
Submitter: Jenkins
Branch: master

commit c8ceeb46c7589a9159c4ad2ad50ac19249981aa0
Author: Markus Zoeller <email address hidden>
Date: Wed Feb 3 16:04:29 2016 +0100

    libvirt: Add serial ports to the migration data object

    The serial console code in nova picks TCP ports for each serial
    interface of an instance. When doing a live migration, the
    TCP ports used for the migrating instance might be in use on
    the target hypervisor which causes the live migration to fail.

    This patch adds a field to the migration data object so that a follow
    up patch can use that to rewrite the guest XML.

    Change-Id: I970014ee0ee8059f5976797bf344fea9a7ccb32d
    Related-Bug: #1455252

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/347471

[Matt Riedemann (mriedem)]

There is a revert proposed to fix this, Markus Zoeller tested the serial console live migration and said it worked:

https://review.openstack.org/#/c/368732/

[Matt Riedemann (mriedem)]

Ignore comment 14, that revert doesn't fix the serial console issue, Markus said that https://review.openstack.org/#/c/275801/ is the fix.

[Matt Riedemann (mriedem)]

I'm dropping the newton-rc-potential tag for this as it's a latent bug, not a regression in newton.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/275801
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=898bb13304b68f33a130281d5acf425a1b23d171
Submitter: Jenkins
Branch: master

commit 898bb13304b68f33a130281d5acf425a1b23d171
Author: Markus Zoeller <email address hidden>
Date: Tue Nov 22 17:08:14 2016 +0100

    libvirt: Acquire TCP ports for console during live migration

    During a live migration process we reserve serial ports on
    destination host and update domain XML during migration to
    use the new reserved ports from destination.

    If the migration succeeds we release ports on the source host.
    If the migration fails we release reserved ports on the
    destination host.

    Co-Authored-By: Sahid Ferdjaoui <email address hidden>
    Change-Id: Ie2524191d22cca2287eb7dbaa22b74d43e43c896
    Closes-Bug: #1455252

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/406898

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 15.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/406898
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=7cdd8b2bf3583db78b9dc02f7341cb8a3472aed1
Submitter: Jenkins
Branch: stable/newton

commit 7cdd8b2bf3583db78b9dc02f7341cb8a3472aed1
Author: Markus Zoeller <email address hidden>
Date: Tue Nov 22 17:08:14 2016 +0100

    libvirt: Acquire TCP ports for console during live migration

    During a live migration process we reserve serial ports on
    destination host and update domain XML during migration to
    use the new reserved ports from destination.

    If the migration succeeds we release ports on the source host.
    If the migration fails we release reserved ports on the
    destination host.

    Conflicts:
     nova/tests/unit/virt/libvirt/test_driver.py
     nova/virt/libvirt/driver.py

    Co-Authored-By: Sahid Ferdjaoui <email address hidden>
    Change-Id: Ie2524191d22cca2287eb7dbaa22b74d43e43c896
    Closes-Bug: #1455252
    (cherry picked from commit 898bb13304b68f33a130281d5acf425a1b23d171)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 14.0.5 release.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Markus Zoeller (markus_z) (<email address hidden>) on branch: master
Review: https://review.openstack.org/347471
Reason: I give up, I don't know what to do here.