Port TCP 16509 is not allowed in compute firewall which breaks instance live migration

Bug #1635427 reported by James Slagle
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
James Slagle

Bug Description

Live migration of instance fails with:

2016-10-20 18:29:41.470 9410 ERROR nova.virt.libvirt.driver [req-abe8c10c-a496-49dd-b414-fa0fe0a66a6f 2cc73868cdb84c85a142794fe852c7ed b424e6e8863240f7a5edb629a5db834d - - -] [instance: 029ad115-38f8-49e7-89f1-7161d76b0ed3] Live Migration failure: operation failed: Failed to connect to remote libvirt URI qemu+tcp://comp-r00-01.redhat.local/system: unable to connect to server at 'comp-r00-01.redhat.local:16509': No route to host

Version-Release number of selected component (if applicable):
openstack-heat-templates-0.0.1-0.20161004223740.f123aa1.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy overcloud with 2 compute nodes
2. Live migrate instance from one host to another

Actual results:
Live migration fails with the following error in /var/log/nova/nova-compute.log:
2016-10-20 18:29:41.470 9410 ERROR nova.virt.libvirt.driver [req-abe8c10c-a496-49dd-b414-fa0fe0a66a6f 2cc73868cdb84c85a142794fe852c7ed b424e6e8863240f7a5edb629a5db834d - - -] [instance: 029ad115-38f8-49e7-89f1-7161d76b0ed3] Live Migration failure: operation failed: Failed to connect to remote libvirt URI qemu+tcp://comp-r00-01.redhat.local/system: unable to connect to server at 'comp-r00-01.redhat.local:16509': No route to host

Expected results:

Additional info:
From source to destination:
[heat-admin@comp-r00-00 ~]$ nc comp-r00-01.redhat.local 16509
Ncat: No route to host.

On the destination host:
[heat-admin@comp-r00-01 ~]$ nc comp-r00-01.redhat.local 16509
Ncat: Broken pipe.

iptables rules:
[heat-admin@comp-r00-01 ~]$ sudo iptables -nL
Chain INPUT (policy ACCEPT)
target prot opt source destination
neutron-openvswi-INPUT all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 /* 000 accept related established rules */ state RELATED,ESTABLISHED
ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 /* 001 accept all icmp */ state NEW
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 /* 002 accept all to lo interface */ state NEW
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 22 /* 003 accept ssh */ state NEW
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 123 /* 105 ntp */ state NEW
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 4789 /* 118 neutron vxlan networks */ state NEW
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 161 /* 127 snmp */ state NEW
ACCEPT 47 -- 0.0.0.0/0 0.0.0.0/0 /* 136 neutron gre networks */ state NEW
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:22
REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited
LOG all -- 0.0.0.0/0 0.0.0.0/0 /* 998 log all */ LOG flags 0 level 4
DROP all -- 0.0.0.0/0 0.0.0.0/0 /* 999 drop all */ state NEW

Chain FORWARD (policy ACCEPT)
target prot opt source destination
neutron-filter-top all -- 0.0.0.0/0 0.0.0.0/0
neutron-openvswi-FORWARD all -- 0.0.0.0/0 0.0.0.0/0
REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
neutron-filter-top all -- 0.0.0.0/0 0.0.0.0/0
neutron-openvswi-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0

Chain neutron-filter-top (2 references)
target prot opt source destination
neutron-openvswi-local all -- 0.0.0.0/0 0.0.0.0/0

Chain neutron-openvswi-FORWARD (1 references)
target prot opt source destination
neutron-openvswi-sg-chain all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out tap2b63863d-d4 --physdev-is-bridged /* Direct traffic from the VM interface to the security group chain. */
neutron-openvswi-sg-chain all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tap2b63863d-d4 --physdev-is-bridged /* Direct traffic from the VM interface to the security group chain. */

Chain neutron-openvswi-INPUT (1 references)
target prot opt source destination
neutron-openvswi-o2b63863d-d all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tap2b63863d-d4 --physdev-is-bridged /* Direct incoming traffic from VM to the security group chain. */

Chain neutron-openvswi-OUTPUT (1 references)
target prot opt source destination

Chain neutron-openvswi-i2b63863d-d (1 references)
target prot opt source destination
RETURN all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED /* Direct packets associated with a known session to the RETURN chain. */
RETURN udp -- 172.16.19.11 0.0.0.0/0 udp spt:67 udp dpt:68
RETURN udp -- 172.16.19.10 0.0.0.0/0 udp spt:67 udp dpt:68
RETURN udp -- 172.16.19.12 0.0.0.0/0 udp spt:67 udp dpt:68
RETURN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:22
RETURN icmp -- 0.0.0.0/0 0.0.0.0/0
DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID /* Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack. */
neutron-openvswi-sg-fallback all -- 0.0.0.0/0 0.0.0.0/0 /* Send unmatched traffic to the fallback chain. */

Chain neutron-openvswi-local (1 references)
target prot opt source destination

Chain neutron-openvswi-o2b63863d-d (2 references)
target prot opt source destination
RETURN udp -- 0.0.0.0 255.255.255.255 udp spt:68 dpt:67 /* Allow DHCP client traffic. */
neutron-openvswi-s2b63863d-d all -- 0.0.0.0/0 0.0.0.0/0
RETURN udp -- 0.0.0.0/0 0.0.0.0/0 udp spt:68 dpt:67 /* Allow DHCP client traffic. */
DROP udp -- 0.0.0.0/0 0.0.0.0/0 udp spt:67 udp dpt:68 /* Prevent DHCP Spoofing by VM. */
RETURN all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED /* Direct packets associated with a known session to the RETURN chain. */
RETURN all -- 0.0.0.0/0 0.0.0.0/0
DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID /* Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack. */
neutron-openvswi-sg-fallback all -- 0.0.0.0/0 0.0.0.0/0 /* Send unmatched traffic to the fallback chain. */

Chain neutron-openvswi-s2b63863d-d (1 references)
target prot opt source destination
RETURN all -- 172.16.19.19 0.0.0.0/0 MAC FA:16:3E:E1:CC:59 /* Allow traffic from defined IP/MAC pairs. */
DROP all -- 0.0.0.0/0 0.0.0.0/0 /* Drop traffic without an IP/MAC allow rule. */

Chain neutron-openvswi-sg-chain (2 references)
target prot opt source destination
neutron-openvswi-i2b63863d-d all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out tap2b63863d-d4 --physdev-is-bridged /* Jump to the VM specific chain. */
neutron-openvswi-o2b63863d-d all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tap2b63863d-d4 --physdev-is-bridged /* Jump to the VM specific chain. */
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0

Chain neutron-openvswi-sg-fallback (2 references)
target prot opt source destination
DROP all -- 0.0.0.0/0 0.0.0.0/0 /* Default drop rule for unmatched traffic. */

Revision history for this message
James Slagle (james-slagle) wrote :

tcp port 16509 should be opened for libvirt live migration per:
http://docs.openstack.org/admin-guide/compute-configuring-migrations.html

Changed in tripleo:
assignee: nobody → James Slagle (james-slagle)
importance: Undecided → Critical
milestone: none → ocata-1
status: New → In Progress
tags: added: newton-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.openstack.org/389358

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.openstack.org/389358
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=1c4ade1d66450a49da9cb17528d21d47e000bf80
Submitter: Jenkins
Branch: master

commit 1c4ade1d66450a49da9cb17528d21d47e000bf80
Author: James Slagle <email address hidden>
Date: Thu Oct 20 17:25:21 2016 -0400

    Open port 16509 for libvirt for live migration

    Port 16509 should be opened for tcp traffic to enable live migration.

    See Also:
    http://docs.openstack.org/admin-guide/compute-configuring-migrations.html

    Previously, we were not enabling any iptables rules on the Compute
    Roles, so this is a regression.

    Change-Id: Ie4abf53dc2a8171af48d02e34a1a3ad43f27cfb3
    Closes-Bug: #1635427

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/389662

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/newton)

Reviewed: https://review.openstack.org/389662
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=fa5c89c58569bed19bfb9a6794c528b891161a10
Submitter: Jenkins
Branch: stable/newton

commit fa5c89c58569bed19bfb9a6794c528b891161a10
Author: James Slagle <email address hidden>
Date: Thu Oct 20 17:25:21 2016 -0400

    Open port 16509 for libvirt for live migration

    Port 16509 should be opened for tcp traffic to enable live migration.

    See Also:
    http://docs.openstack.org/admin-guide/compute-configuring-migrations.html

    Previously, we were not enabling any iptables rules on the Compute
    Roles, so this is a regression.

    Change-Id: Ie4abf53dc2a8171af48d02e34a1a3ad43f27cfb3
    Closes-Bug: #1635427
    (cherry picked from commit 1c4ade1d66450a49da9cb17528d21d47e000bf80)

tags: added: in-stable-newton
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 5.1.0

This issue was fixed in the openstack/tripleo-heat-templates 5.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 6.0.0.0b1

This issue was fixed in the openstack/tripleo-heat-templates 6.0.0.0b1 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 5.1.0

This issue was fixed in the openstack/tripleo-heat-templates 5.1.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.