Unable to ping VM by second floating ip

[Mikhail Laptev]

During the check, I've deployed OpenStack with VBox scripts.
After successful deployment following list of command was executed:

As a result, I can ping the VM by first IP address, but unable to do it by another IP address.
p.s. I've attached full log to the issue.
p.p.s. list of commands could be found below:

root@node-2:~# glance image-list
root@node-2:~# glance image-create --name='Ubuntu 14.04 LTS' --disk-format=qcow2 --container-format=bare --visibility=public --file=/tmp/trusty-server-cloudimg-amd64-disk1.img --progress
root@node-2:~# nova flavor-create m1.medium.1cpu 3-1 4096 40 1
root@node-2:~# nova boot --flavor 3-1 --image "Ubuntu 14.04 LTS" --nic net-id="7c97fd51-5ec9-4971-ab09-afbd71d61ece" MyUbuntuWithNova --poll
root@node-2:~# nova secgroup-add-rule default icmp -1 -1 0.0.0.0/0
root@node-2:~# nova secgroup-add-rule default tcp 22 22 0.0.0.0/0
root@node-2:~# nova floating-ip-create
root@node-2:~# nova floating-ip-associate 3e32cdff-7150-473e-9a57-97cda79017ff 172.16.0.229
root@node-2:~# ping 172.16.0.229
root@node-2:~# neutron net-create new_internal_net
root@node-2:~# neutron subnet-create --name new_internal_net__subnet --allocation-pool start=192.168.112.2,end=192.168.112.254 --ip-version 4 new_internal_net 192.168.112.0/24
root@node-2:~# neutron subnet-update --dns-nameserver 8.8.4.4 --dns-nameserver 8.8.8.8 9d3c6b9f-eb47-464e-aa82-c9da89f3dbd5
root@node-2:~# nova floating-ip-create
root@node-2:~# nova interface-attach --net-id 2a44d0d1-7df8-4972-98a0-b0e736570276 3e32cdff-7150-473e-9a57-97cda79017ff
root@node-2:~# nova floating-ip-associate 3e32cdff-7150-473e-9a57-97cda79017ff 172.16.0.230
ERROR (BadRequest): Unable to associate floating ip 172.16.0.230 to fixed ip 192.168.111.190 for instance 3e32cdff-7150-473e-9a57-97cda79017ff. Error: Cannot associate floating IP 172.16.0.230 (759690f4-73d8-497c-9ac3-1980d0f9eff4) with port ab44d743-1342-4f57-84e2-4ffee4416546 using fixed IP 192.168.111.190, as that fixed IP already has a floating IP on external network d9a522f6-d360-4358-9b95-7bf55a48eaad. (HTTP 400) (Request-ID: req-583bfd15-34e5-49a1-8fd3-466b01ef02e1)
root@node-2:~# neutron router-interface-add 21dd5c13-d5f0-4dd1-a753-256af40923b5 9d3c6b9f-eb47-464e-aa82-c9da89f3dbd5
root@node-2:~# nova floating-ip-associate --fixed-address 192.168.112.4 3e32cdff-7150-473e-9a57-97cda79017ff 172.16.0.230
root@node-2:~# ping 172.16.0.230
PING 172.16.0.230 (172.16.0.230) 56(84) bytes of data.
From 172.16.0.230 icmp_seq=1 Destination Host Unreachable
From 172.16.0.230 icmp_seq=2 Destination Host Unreachable
From 172.16.0.230 icmp_seq=3 Destination Host Unreachable
From 172.16.0.230 icmp_seq=4 Destination Host Unreachable
From 172.16.0.230 icmp_seq=5 Destination Host Unreachable
From 172.16.0.230 icmp_seq=6 Destination Host Unreachable
From 172.16.0.230 icmp_seq=7 Destination Host Unreachable
From 172.16.0.230 icmp_seq=8 Destination Host Unreachable
From 172.16.0.230 icmp_seq=9 Destination Host Unreachable
^C
--- 172.16.0.230 ping statistics ---
11 packets transmitted, 0 received, +9 errors, 100% packet loss, time 10053ms
root@node-2:~# ping 172.16.0.229
PING 172.16.0.229 (172.16.0.229) 56(84) bytes of data.
64 bytes from 172.16.0.229: icmp_seq=1 ttl=63 time=3.46 ms
64 bytes from 172.16.0.229: icmp_seq=2 ttl=63 time=1.77 ms
64 bytes from 172.16.0.229: icmp_seq=3 ttl=63 time=1.27 ms
--- 172.16.0.229 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 1.270/2.171/3.468/0.940 ms

[Matthew Mosesohn (raytrac3r)]

Can you include a diagnostic snapshot?

[Bogdan Dobrelya (bogdando)]

Please also include info which exactly env did you deploy

[Bogdan Dobrelya (bogdando)]

Reproduced the same issue with Vxlan , here are logs, see around Tue Dec 29 09:46:46 UTC 2015

[Bogdan Dobrelya (bogdando)]

I noticed there is no IP at the new interface after the command "nova interface-attach" issued to the test instance.

More snippets http://pastebin.com/LvXxrFCb

[Bogdan Dobrelya (bogdando)]

Few more snippets with flows http://pastebin.com/tFXfUPS2
Note, that even though I added the ip address to the instance's eth1 and set it up, nothing has changed, no connectivity to the secondary floating IP

[Bogdan Dobrelya (bogdando)]

We should clarify the impact of this issue. Seems like a high one.

[Bogdan Dobrelya (bogdando)]

The issue is in Nova or, more likely, Neutron.
I checked how this works for the single fixed IPs network and single instance's interface:
- nova add-fixed-ip replaces the instance's fixed IP to the new one. Shouldn't it add the IP to the interface instead?
- the 2nd floating ip associated with the new fixed IP works, while the old floating IP stops working (because there is no old fixed IP - it was replaced).
- if manually added to the interface, the original fixed IP association works again and both floating IPs are OK.

And this doesn't work if the new fixed IP is from another network and belongs to another attached interface.

[Bogdan Dobrelya (bogdando)]

Looks like the similar issue https://ask.openstack.org/en/question/44793/problem-attaching-a-second-interface-into-a-vm/

[Alexander Ignatov (aignatov)]

Won't fix for 8.0 since it's Medium.

[Elena Ezhova (eezhova)]

There are a number of issues here:
1. When a second interface (let it be eth1) is attached to a VM (or if it is booted with two interfaces) only the first interface (eth0) gets an IP. To configure the second interface it is needed to use dhcp client (dhclient, pump, udhcpc or dhcpcd) or use system configuration method for permanent results (via /etc/network/interfaces for Ubuntu [1]).

2. As a result the eth1 interface gets the IP that was allocated for it and routes are configured in the following way:

root@test2:~# ip r
default via 10.0.0.1 dev eth0
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.6 # eth0 ip
10.0.1.0/24 dev eth1 proto kernel scope link src 10.0.1.6 # eth1 ip
169.254.169.254 via 10.0.0.1 dev eth0

From now on both 10.0.0.6 and 10.0.1.6 can be pinged from the qrouter namespace. It is possible to ping floating ip associated with 10.0.0.6 (FIP1) while floating ip associated with 10.0.1.6 (FIP2) is not accessible.

The reason is that when we ping FIP2 source ip is set to public net gateway IP and destination address is NAT-ed to 10.0.1.6. ICMP echo packet arrives at VMs eth1 interface but then ICMP reply packet would be either go through eth0 (instead of eth1) or would be dropped by rp_filter [2] if it's enabled.

A solution is to have different default gateway for each interface. As routing table can normally have only one default route entry it is needed to create another routing table, populate it with needed routes and add rules that traffic to/from 10.0.1.6 would use this routing table. [3][4] After this both floating ips can be successfully pinged.

The resulting configuration looks like this:

root@test2:~# ip rule show
0: from all lookup local
32764: from all to 10.0.1.6 lookup floating
32765: from 10.0.1.6 lookup floating
32766: from all lookup main
32767: from all lookup default

root@test2:~# ip route list table floating
default via 10.0.1.1 dev eth1
10.0.1.0/24 dev eth1 scope link src 10.0.1.6

root@test2:~# ip route list table main
default via 10.0.0.1 dev eth0
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.6
169.254.169.254 via 10.0.0.1 dev eth0

All in all, this is entirely a guest configuration issue and it can be solved by manual configuration like described above. In case when a VM is booted with multiple interfaces such configuration can be performed by cloud-init scripts, on the other hand, when an interface is being attached to a working VM openstack services no longer have access to it. But probably there is a way to perform advanced configuration using DHCP, this has to be verified.

[1] https://ask.openstack.org/en/question/44793/problem-attaching-a-second-interface-into-a-vm/
[2] http://tldp.org/HOWTO/Adv-Routing-HOWTO/lartc.kernel.rpf.html
[3] https://www.thomas-krenn.com/en/wiki/Two_Default_Gateways_on_One_System
[4] http://lartc.org/howto/lartc.rpdb.html

[Elena Ezhova (eezhova)]

Moving this to Invalid, if manual configuration solution is unsatisfactory, please reopen and we'll consider implementing automatization as a feature.