Devstack (Libvirt driver) install on Ubuntu 14.04.2 floating IPs not working

[Boris Derzhavets]

Tested on VMs (F21 KVM Hypervisor) 4 GB RAM , 2 VCPUs running stack.sh instance with Ubuntu 14.04.2 (1).

    $ git clone https://git.openstack.org/openstack-dev/devstack
    $ cd devstack
    $ ./stack.sh

  My local.conf.

    [[local|localrc]]
    HOST_IP=192.169.142.52
    ADMIN_PASSWORD=secret
    MYSQL_PASSWORD=secret
    RABBIT_PASSWORD=secret
    SERVICE_PASSWORD=secret
    FLOATING_RANGE=192.168.10.0/24
    FLAT_INTERFACE=eth0
    Q_FLOATING_ALLOCATION_POOL=start=192.168.10.150,end=192.168.10.254
    PUBLIC_NETWORK_GATEWAY=192.168.10.15
    SERVICE_TOKEN=super-secret-admin-token

    DEST=$HOME/stack
    SERVICE_DIR=$DEST/status
    DATA_DIR=$DEST/data
    LOGFILE=$DEST/logs/stack.sh.log
    LOGDIR=$DEST/logs

    FIXED_RANGE=10.254.1.0/24
    NETWORK_GATEWAY=10.254.1.1

    # Services
    disable_service n-net
    enable_service q-svc
    enable_service q-agt
    enable_service q-dhcp
    enable_service q-l3
    enable_service q-meta
    enable_service horizon
    disable_service tempest

Security rules ( demo tenant, I ran `cd dev* && . openrc demo` )

    ubuntu@ubuntu-vm:~/devstack$ nova secgroup-list-rules default
    +-------------+-----------+---------+-----------+--------------+
    | IP Protocol | From Port | To Port | IP Range | Source Group |
    +-------------+-----------+---------+-----------+--------------+
    | | | | | default |
    | icmp | -1 | -1 | 0.0.0.0/0 | |
    | | | | | default |
    | tcp | 22 | 22 | 0.0.0.0/0 | |
    +-------------+-----------+---------+-----------+--------------+

I can login to VF21 instance only via qdhcp-namespace

    ubuntu@ubuntu-vm:~/devstack$ . openrc demo
    ubuntu@ubuntu-vm:~/devstack$ sudo ip netns exec qdhcp-94d8a1e6-89bf-4162-9fc3-061a9bc17737 ssh -i osxkey.pem fedora@10.254.1.4
    Last login: Wed Feb 25 22:01:09 2015 from 10.254.1.3
    [fedora@vf21rsx01 ~]$ uname -a
    Linux vf21rsx01.novalocal 3.18.7-200.fc21.x86_64 #1 SMP Wed Feb 11 21:53:17 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

 I have internet access && can run yum -y update.

I ping from 192.169.142.153 (host running stack.sh instance ) floating IP 192.168.10.154 ( private IP 50.0.0.13) . `tcpdump -vv -i eth0` is running inside VM (192.168.10.154, 50.0.0.13)

    20:19:34.729398 IP (tos 0x0, ttl 63, id 42021, offset 0, flags [DF], proto ICMP (1), length 84)
        ip-192-169-142-53.ip.secureserver.net > 50-0-0-13.static.sonic.net: ICMP echo request, id 8588, seq 560, length 64
    20:19:34.729696 IP (tos 0x0, ttl 64, id 41602, offset 0, flags [none], proto ICMP (1), length 84)
        50-0-0-13.static.sonic.net > ip-192-169-142-53.ip.secureserver.net: ICMP echo reply, id 8588, seq 560, length 64
    20:19:35.729432 IP (tos 0x0, ttl 63, id 42096, offset 0, flags [DF], proto ICMP (1), length 84)
        ip-192-169-142-53.ip.secureserver.net > 50-0-0-13.static.sonic.net: ICMP echo request, id 8588, seq 561, length 64
    20:19:35.729742 IP (tos 0x0, ttl 64, id 41605, offset 0, flags [none], proto ICMP (1), length 84)
        50-0-0-13.static.sonic.net > ip-192-169-142-53.ip.secureserver.net: ICMP echo reply, id 8588, seq 561, length 64

Runtime tcpdump tracking devices involved :-

    ubuntu@ubuntu-vm2:~/devstack$ brctl show

    bridge name bridge id STP enabled interfaces
    qbr715a260e-b2 8000.0648d25a79c4 no qvb715a260e-b2
    qbra7a715f5-02 8000.522935fa9c61 no qvba7a715f5-02
                                        tapa7a715f5-02
    virbr0 8000.000000000000 y es

    ubuntu@ubuntu-vm2:~/devstack$ sudo ovs-vsctl show | grep a7a715f5-02
            Port "qvoa7a715f5-02"
                Interface "qvoa7a715f5-02"

    ICMP traffic is OK on "tapa7a715f5-02" , on "qbra7a715f5-02" ICMP replies from VM are already lost.
    So , they don't reach br-int via (qvba7a715f5-02,qvoa7a715f5-02) veth pair

[Boris Derzhavets (bderzhavets)]

TYpo, should be
I ping from 192.169.142.53 . . . . .

[Boris Derzhavets (bderzhavets)]

External network created by devstack didn't have "shared" status. Changing status to "shared" gives ability
ping floating IPs, however ssh connection `ssh -i oskey.pem fedora@floatimg-ip` still hangs and finally times out.
f21 instance has been launched with oskey.

[Boris Derzhavets (bderzhavets)]

Another fresh test on real 14.04.2 box. Sharing public network doesn't help any long.
Floating IP 192.168.10.103 unpingable, however
ubuntu@ubuntu-System-test:~/devstack$ arping -I br-ex 192.168.10.103
ARPING 192.168.10.103 from 192.168.10.15 br-ex
Unicast reply from 192.168.10.103 [FA:16:3E:E5:39:D9] 0.711ms
Unicast reply from 192.168.10.103 [FA:16:3E:E5:39:D9] 0.619ms
Unicast reply from 192.168.10.103 [FA:16:3E:E5:39:D9] 0.531ms
Unicast reply from 192.168.10.103 [FA:16:3E:E5:39:D9] 0.532ms
Unicast reply from 192.168.10.103 [FA:16:3E:E5:39:D9] 0.534ms

[Boris Derzhavets (bderzhavets)]

Another test

root@ubuntu-System-test:~# hping3 -c 3 192.168.10.103
HPING 192.168.10.103 (br-ex 192.168.10.103): NO FLAGS are set, 40 headers + 0 data bytes
len=40 ip=192.168.10.103 ttl=64 DF id=44493 sport=0 flags=RA seq=0 win=0 rtt=4.3 ms
len=40 ip=192.168.10.103 ttl=64 DF id=44612 sport=0 flags=RA seq=1 win=0 rtt=4.3 ms
len=40 ip=192.168.10.103 ttl=64 DF id=44716 sport=0 flags=RA seq=2 win=0 rtt=4.3 ms

--- 192.168.10.103 hping statistic ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 4.3/4.3/4.3 ms

root@ubuntu-System-test:~# hping3 -c 3 -1 192.168.10.103
HPING 192.168.10.103 (br-ex 192.168.10.103): icmp mode set, 28 headers + 0 data bytes

--- 192.168.10.103 hping statistic ---
3 packets transmitted, 0 packets received, 100% packet loss
round-trip min/avg/max = 0.0/0.0/0.0 ms

So qbrXXXX bridge drops ICMP packets.

[Boris Derzhavets (bderzhavets)]

Would be more precise, corresponding VM bridge qbrXXXXXX drops ICMP replies been sent by VM.
It's reproducible behavior under unstable instance of stack.sh. Problems with password request which
appeared during ssh login via qdhcp-namespace I already resolved , and this is not already important , because
tcpdump may be run against corresponding tap device plugged into VM when nova boots VM.
I also verified neutron chains ( responsible for security rules implementation ) in iptables ( I run iptables-persistent firewall , ufw is disabled ) corresponding tap device with wrong behavior and they seem to be OK.

[Boris Derzhavets (bderzhavets)]

Correct command is :-
$ iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
It keeps floating IPs 100% usable and security rules 100% effective
Command like :-
$ iptables -t nat -A POSTROUTING -s 172.24.4.0/24 -j MASQUERADE
disables floating IPs if 172.24.4.0/24 is devstack public network

Thank you for cooperation.

[Sean Dague (sdague)]

This devstack bug was last updated over 180 days ago, as devstack
is a fast moving project and we'd like to get the tracker down to
currently actionable bugs, this is getting marked as Invalid. If the
issue still exists, please feel free to reopen it.