live migration fails after "Modify libvirt port range for live-migration" change

Bug #1779820 reported by Martin Schuppert
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Medium
Martin Schuppert

Bug Description

change implemented in https://review.openstack.org/#/c/561784/ is not complete as nova::migration::qemu::configure_qemu is not set and new min/max port in qemu.conf is not set. As a result we have changed iptables rules to the new port range and connection to the default port range time out:

compute-0]$ egrep "migration_port_min|migration_port_max" var/lib/config-data/puppet-generated/nova_libvirt/etc/libvirt/qemu.conf
#migration_port_min = 49152
#migration_port_max = 49215

compute-0]$ grep libvirt etc/sysconfig/iptables
-A INPUT -p tcp -m multiport --dports 16514,61152:61215,5900:6923 -m state --state NEW -m comment --comment "200 nova_libvirt ipv4" -j ACCEPT

migration results in:
2018-07-03 06:03:00.786 1 ERROR nova.virt.libvirt.driver [req-1756a25c-95bb-45ad-b78d-1f235ee3155c 9646b8286b0b40f798e27d27aa529ece 8f01287c00c94895ae6639a198db4ed2 - default default] [instance: 80e82e13-6479-4b63-83bb-077cdc9c9c69] Live Migration failure: unable to connect to server at 'compute-0.localdomain:49152': Connection timed out: libvirtError: unable to connect to server at 'compute-0.localdomain:49152': Connection timed out

Changed in tripleo:
assignee: nobody → Martin Schuppert (mschuppert)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to puppet-tripleo (master)

Fix proposed to branch: master
Review: https://review.openstack.org/579807

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.openstack.org/579813

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to puppet-tripleo (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/580059

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to puppet-tripleo (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/580061

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/580066

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/580068

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on puppet-tripleo (master)

Change abandoned by Emilien Macchi (<email address hidden>) on branch: master
Review: https://review.openstack.org/579807
Reason: The gate is suffering of timeouts, we need to clear it. Please do not restore or recheck this patch, I'll take care of it when gate is stable again.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to puppet-tripleo (master)

Reviewed: https://review.openstack.org/579807
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=ed16fdc55d76fe8f7fdf7c9ca3bd9acd66cbc3af
Submitter: Zuul
Branch: master

commit ed16fdc55d76fe8f7fdf7c9ca3bd9acd66cbc3af
Author: Martin Schuppert <email address hidden>
Date: Tue Jul 3 10:24:17 2018 +0200

    Make sure we apply qemu config changes

    With the change in https://review.openstack.org/#/c/561784/3 we need to
    make sure that the new port range get applied to the the qemu.conf file.
    This change includes ::nova::migration::qemu to
    ::tripleo::profile::base::nova::libvirt

    Change-Id: Idadfc7b3507977f1385e846a48a734ed0e5f0a32
    Closes-bug: 1779820

Changed in tripleo:
status: In Progress → Fix Released
Changed in tripleo:
importance: Undecided → Medium
milestone: none → rocky-3
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on puppet-tripleo (stable/pike)

Change abandoned by Martin Schuppert (<email address hidden>) on branch: stable/pike
Review: https://review.openstack.org/580061

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on puppet-tripleo (stable/queens)

Change abandoned by Martin Schuppert (<email address hidden>) on branch: stable/queens
Review: https://review.openstack.org/580059

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.openstack.org/579813
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=547552fc7b6730f7d0670c5b0d76f63e5372f8c7
Submitter: Zuul
Branch: master

commit 547552fc7b6730f7d0670c5b0d76f63e5372f8c7
Author: Martin Schuppert <email address hidden>
Date: Tue Jul 3 10:39:28 2018 +0200

    Set configure_qemu to true to get qemu port range applied

    With https://review.openstack.org/#/c/561784 we change the default
    migration port range to '61152-61215'.
    nova::migration::qemu::configure_qemu needs to be set to true that
    the config gets applied via puppet-nova.

    Change-Id: Iad4b392c9fe7426f2ce10a02fadd8b1aeee34ef6
    Closes-bug: 1779820
    Depends-On: Idadfc7b3507977f1385e846a48a734ed0e5f0a32

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to puppet-tripleo (stable/queens)

Reviewed: https://review.openstack.org/580059
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=3c2dc2a5d47fb9365ff6e8b430a569fd029472dc
Submitter: Zuul
Branch: stable/queens

commit 3c2dc2a5d47fb9365ff6e8b430a569fd029472dc
Author: Martin Schuppert <email address hidden>
Date: Tue Jul 3 10:24:17 2018 +0200

    Make sure we apply qemu config changes

    With the change in https://review.openstack.org/#/c/561784/3 we need to
    make sure that the new port range get applied to the the qemu.conf file.
    This change includes ::nova::migration::qemu to
    ::tripleo::profile::base::nova::libvirt

    Change-Id: Idadfc7b3507977f1385e846a48a734ed0e5f0a32
    Closes-bug: 1779820
    (cherry picked from commit ed16fdc55d76fe8f7fdf7c9ca3bd9acd66cbc3af)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to puppet-tripleo (stable/pike)

Reviewed: https://review.openstack.org/580061
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=8deff93827aa5438be96cc72788c847ba682c402
Submitter: Zuul
Branch: stable/pike

commit 8deff93827aa5438be96cc72788c847ba682c402
Author: Martin Schuppert <email address hidden>
Date: Tue Jul 3 10:24:17 2018 +0200

    Make sure we apply qemu config changes

    With the change in https://review.openstack.org/#/c/561784/3 we need to
    make sure that the new port range get applied to the the qemu.conf file.
    This change includes ::nova::migration::qemu to
    ::tripleo::profile::base::nova::libvirt

    Change-Id: Idadfc7b3507977f1385e846a48a734ed0e5f0a32
    Closes-bug: 1779820
    (cherry picked from commit ed16fdc55d76fe8f7fdf7c9ca3bd9acd66cbc3af)
    (cherry picked from commit 3c2dc2a5d47fb9365ff6e8b430a569fd029472dc)

tags: added: in-stable-pike
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/queens)

Reviewed: https://review.openstack.org/580066
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=322747fa6e62fd74bc6e40515b9563d8e62f4654
Submitter: Zuul
Branch: stable/queens

commit 322747fa6e62fd74bc6e40515b9563d8e62f4654
Author: Martin Schuppert <email address hidden>
Date: Tue Jul 3 10:39:28 2018 +0200

    Set configure_qemu to true to get qemu port range applied

    With https://review.openstack.org/#/c/561784 we change the default
    migration port range to '61152-61215'.
    nova::migration::qemu::configure_qemu needs to be set to true that
    the config gets applied via puppet-nova.

    Change-Id: Iad4b392c9fe7426f2ce10a02fadd8b1aeee34ef6
    Closes-bug: 1779820
    Depends-On: Idadfc7b3507977f1385e846a48a734ed0e5f0a32
    (cherry picked from commit 547552fc7b6730f7d0670c5b0d76f63e5372f8c7)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/pike)

Reviewed: https://review.openstack.org/580068
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=ec472adc0c2e443ec7816a70d80ba683891c759c
Submitter: Zuul
Branch: stable/pike

commit ec472adc0c2e443ec7816a70d80ba683891c759c
Author: Martin Schuppert <email address hidden>
Date: Tue Jul 3 10:39:28 2018 +0200

    Set configure_qemu to true to get qemu port range applied

    With https://review.openstack.org/#/c/561784 we change the default
    migration port range to '61152-61215'.
    nova::migration::qemu::configure_qemu needs to be set to true that
    the config gets applied via puppet-nova.

    Change-Id: Iad4b392c9fe7426f2ce10a02fadd8b1aeee34ef6
    Closes-bug: 1779820
    Depends-On: Idadfc7b3507977f1385e846a48a734ed0e5f0a32
    (cherry picked from commit 547552fc7b6730f7d0670c5b0d76f63e5372f8c7)
    (cherry picked from commit 322747fa6e62fd74bc6e40515b9563d8e62f4654)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/puppet-tripleo 9.2.0

This issue was fixed in the openstack/puppet-tripleo 9.2.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 9.0.0.0b4

This issue was fixed in the openstack/tripleo-heat-templates 9.0.0.0b4 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/puppet-tripleo 7.4.15

This issue was fixed in the openstack/puppet-tripleo 7.4.15 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 7.0.15

This issue was fixed in the openstack/tripleo-heat-templates 7.0.15 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/puppet-tripleo 8.3.5

This issue was fixed in the openstack/puppet-tripleo 8.3.5 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 8.0.5

This issue was fixed in the openstack/tripleo-heat-templates 8.0.5 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.