Inner-project Floating IP communication is not working

[Koji]

Floating IP communication works from the outside-in as well as between projects. However, instances in the same project are unable to communicate with each other (or themselves) via Floating IP. Private IP communication works fine.

For more information
https://answers.launchpad.net/nova/+question/198681

[Koji (kj-tanaka)]

My OpenStack is built on multiple nodes with Ubuntu package as shown below.

OS: Ubuntu 12.04
Nova: 2012.1-0ubuntu2.1
network_manager: VlanManager
libvirt_type: kvm

[Daneyon Hansen (danehans)]

I too have a multi-node OpenStack environment and my instances are able to communicate by Floating IP within the same project.

OS: Ubuntu 12.04
Nova: 2012.1+23-0~precise1
network_manager: FlatDHCP
libvirt_type: kvm

[Joe T (joe-topjian-v)]

Hi Daneyon,

I think this issue might be local to VlanManager only. Thank you for confirming that FlatDHCP works.

Thanks,
Joe

[Mark McLoughlin (markmc)]

bug #956474 shows the opposite problem - duplicate packages when pinging between floating IPs

The fix for bug #933460 should have made this work. Could you try running:

  $> for iii in $(ls -d /sys/class/net/*); do echo -n "$iii "; cat $iii/brport/hairpin_mode 2>/dev/null || echo; done

and paste the output?

You could also try the fix for bug #1040537, but that looks unrelated.

[Joe T (joe-topjian-v)]

Hi Mark,

On which server would you like that for loop to be run? The server that runs nova-network, the compute node (which does not run nova-network), or the instances themselves?

Thanks,
Joe

[Mark McLoughlin (markmc)]

Good question

Run it on both the compute node and network node

Ah, I see in https://answers.launchpad.net/nova/+question/198681 you already looked at hairpin_mode and it wasn't set for the vlan1XX interfaces

It makes sense to me that hairpin_mode should be enabled on the vlan interfaces too. And, indeed, for the bridge interface in flat networking.

hairpin_mode means that a packet can be forwarded back over the port it arrived from. In the case of nova-network on a remote host, the packet will arrive on the vlan interface, be DNATed and need to be forwarded back over the vlan interface. So, that definitely sounds like we need it enabled.

If you manually do "echo 1 > /sys/class/net/vlan1XX/brport/hairpin_mode" on the network node, does that work?

[Joe T (joe-topjian-v)]

Hi Mark,

I ran some tests.

Let my nova-network node be called "cloud" and let my compute node be called "c02".

I have two projects, everything hosted on "c02" at the moment (I have c01, but I will not introduce that yet).

I have both fixed and floating IPs on two instances in project1 and one instance in project2.

hairpin is off on both cloud and c02 (cat /sys/class/net/vlan300 returns 0).

project1 can ping project2 via floating ip.

project1 vm1 cannot ping project1 vm2.

cloud can ping both vm1 and vm2 via floating and fixed ip.

hairpin is set on cloud for vlan300, there are no changes.

Odd: If I leave ping running on cloud, and then ping vm2 from vm1, ping on cloud stops. I must kill both pings in order for the initial ping on cloud to resume.

I then set hairpin on c02, no change and at times I even saw packet loss.

Eventually from doing tests, I lost total connectivity to one vm and had to reboot. After more time, I was unable to reboot or create new instances because the instances could not reach dhcp on cloud. I turned hairpin off on cloud and things are back to "normal".

Let me know if you need clarifications on any certain tests or if I can provide output of anything.

Regarding the "for loop" output, I'm sure you can visualize the output from the above descriptions but just for reference:

before setting hairpin:

cloud:
/sys/class/net/bond0
/sys/class/net/bonding_masters
/sys/class/net/br300
/sys/class/net/br301
/sys/class/net/eth0
/sys/class/net/eth1
/sys/class/net/eth2
/sys/class/net/eth3
/sys/class/net/lo
/sys/class/net/vlan20
/sys/class/net/vlan30
/sys/class/net/vlan300 0
/sys/class/net/vlan301 0

c02:
/sys/class/net/bond0
/sys/class/net/bonding_masters
/sys/class/net/br300
/sys/class/net/br301
/sys/class/net/eth0
/sys/class/net/eth1
/sys/class/net/eth2
/sys/class/net/eth3
/sys/class/net/lo
/sys/class/net/virbr0
/sys/class/net/vlan30
/sys/class/net/vlan300 0
/sys/class/net/vlan301 0
/sys/class/net/vnet0 1
/sys/class/net/vnet1 1
/sys/class/net/vnet2 0

After hairpin was set, the only change was that vlan300 was set to 1.

[Joe T (joe-topjian-v)]

Oops, let me clarify one line:

"project1 vm1 cannot ping project1 vm2."

should read

"project1 vm1 cannot ping project1 vm2 by floating ip."

[Mark McLoughlin (markmc)]

Thanks much for the carefully testing

I don't have any great theories, but could you try:

  firewall_driver=nova.virt.firewall.NoopFirewallDriver

One way to dig into this might be to do e.g. 'iptables -L -v -n' and keep an eye on the counters for each rule while the ping is running

[Thierry Carrez (ttx)]

Could anyone test what Mark suggested in comment 9 ?

[Davanum Srinivas (DIMS) (dims-v)]

We cannot solve the issue you reported without more information. Could you please provide the requested information ? (See comment #9 from Mark)

[Thierry Carrez (ttx)]

This bug lacks the necessary information to effectively reproduce and fix it, therefore it has been closed. Feel free to reopen the bug by providing the requested information and set the bug status back to ''New''.

[Lorin Hochstein (lorinh)]

I just hit this issue on my Folsom deployment

OS: Ubuntu 12.04
Nova: 2012.2.3-0ubuntu2.1~cloud0
network_manager: VlanManager (Not running multihost)
libvirt_type: kvm

I tried a ping test and was unable to ping the floating IP of an instance from inside of another instance that also had a floating IP.

When I watched the icmp packets move across the network, I noticed that SNAT'ing was not occurring to the ICMP packet when it got to the network node.

To be specific, I was trying to ping from "atlantis" (fixed=10.40.1.10, 10.20.0.7) to "dc" (fixed=10.40.1.4, floating=10.20.4).

When atlantis sends out the packet, it's "FROM: 10.40.1.10 to: 10.20.0.4"

When it hits the iptables NAT rules on the nova-network node, the packet header becomes: "FROM: 10.40.1.10 to 10.40.1.4". It then reaches the "dc" node, which sends the response directly back to "atlantis" since it's on the same subnet. The return packet doesn't even reach atlantis, though. It reaches br150 on the atlantis's host, but doesn't get to the vnet interface. I'm not sure why the packet was being dropped by the compute host.

Also, I noticed the oddest thing: the ICMP packets don't reach "dc" unless I'm doing a tcpdump on br150 of the nova-network node! I've never seen such a thing.

[James Troup (elmo)]

Hi,

I'm also seeing the same problem on Folsom from the Ubuntu Cloud
Archive running on Ubuntu 12.04 LTS.

Instance A can not talk to instance B's floating IP address. Instance
A can talk to instance B's fixed IP address.

I see traffic go from instance A, hit the cloud controller, at which
point it hits the DNAT rule in nova-network-PREROUTING and gets
dropped on the floor¹. I assume this is because the bridge isn't
running in hairpin mode but it's a production cloud and I didn't want
to experiment to confirm that.

In any event, turning on hairpin mode wouldn't actually help me
because at that point we'd have traffic going:

 * [A] -> src=A-fixed, dest=B-floating -> [Cloud controller]
 * [Cloud controller] -> src=A-fixed, dest=B-fixed -> [B]

But [A] and [B] are on the same subnet, so B's reply would be direct,
i.e.:

 * [B] -> src=B-> fixed dest=A-fixed -> [A]

Unfortunately [A] is expecting a reply from the [Cloud Controller],
not [B] so it would throw the packet from [B] away.

This could be fixed with SNAT on the cloud controller but that would
mess with the ability to restrict access by IP to floating IPs via
security groups.

I think the correct solution is for the floating IP DNAT rules to also
be run on the hypervisors/compute hosts; doing so fixed the problem
for me.

| ubuntu@juju-machine-A:~$ nc -w1 -v -q0 -z 91.189.92.32 443
| nc: connect to 91.189.92.32 port 443 (tcp) timed out: Operation now in progress
| ubuntu@juju-machine-A:~$

If I then go onto the compute host and add a fake DNAT rule similar to
what is on the cloud controller:

| root@leuce:~# iptables -t nat -I PREROUTING -d 91.189.92.32/32 -j DNAT --to-destination 10.33.16.157

Machine A can then talk to machine B on it's floating IP:

| ubuntu@juju-machine-A:~$ nc -w1 -v -q0 -z 91.189.92.32 443
| Connection to 91.189.92.32 443 port [tcp/https] succeeded!
| ubuntu@juju-machine-A:~$

From a quick check of one of our development clouds that runs grizzly,
it looks like this isn't fixed there either.

--
James

¹ This is an assumption based on the debug output of an iptables -j
  TRACE rule.

[Russell Bryant (russellb)]

Let's please try to limit the use of tags to the ones defined here:

https://wiki.openstack.org/wiki/BugTags

If there's something missing, let me know.

[Tom Haddon (mthaddon)]

Er, Russell, surely the point of tags is the flexibility of them. We added the tags because we use them to keep track of bugs we care about. What's the problem with having "canonical-webops prodstack" tags for this bug?

[Joe T (joe-topjian-v)]

I ran into this issue again on a single-host VlanManager environment.

I saw the same behavior as Lorin: running tcpdump on the bridge caused everything to work. I believe tcpdump is placing the interface in promiscuous mode, so I tried that outside of tcpdump:

ifconfig brXXX promisc

And sure enough, floating IP traffic began working.

I also tried the iptables rules as suggested from James, but that didn't have any effect. I wonder if James' environment was multi-host?

Finally, I see that there was a suggestion to try the NoopFirewallDriver years ago. I apologize, but I never saw those comments.