Floating IPs don't work with LBaaS V2 Loadbalancers

[Al Miller]

Summary:

With LBaaS V2, create a loadbalancer and two member instances. Assign floating IPs to the instances, and to the loadbalancers. The instances are accessible via their FIPS, but the loadbalancer is not. The loadbalancer does work via the its fixed IP.

Reproduced in devstack on master as of Wed, Sept 9, 2015.

# ===== BEGIN local.conf =====
[[local|localrc]]

# The name of the RECLONE environment variable is a bit misleading. It doesn't actually
# reclone repositories, rather it uses git fetch to make sure the repos are current.

RECLONE=True
#RECLONE=False

# Load the external LBaaS plugin.

#enable_plugin neutron-lbaas https://git.openstack.org/openstack/neutron-lbaas stable/kilo

enable_plugin neutron-lbaas https://git.openstack.org/openstack/neutron-lbaas

#enable_plugin neutron-lbaas /media/sf_SharedPyCharm/bug-1483100/neutron-lbaas

DATABASE_PASSWORD=password
ADMIN_PASSWORD=password
SERVICE_PASSWORD=password
SERVICE_TOKEN=password
RABBIT_PASSWORD=password
# Enable Logging
LOGFILE=$DEST/logs/stack.sh.log
VERBOSE=True
LOG_COLOR=True
SCREEN_LOGDIR=$DEST/logs
# Pre-requisite
ENABLED_SERVICES=rabbit,mysql,key
# Horizon
#ENABLED_SERVICES+=,horizon
# Nova
ENABLED_SERVICES+=,n-api,n-crt,n-obj,n-cpu,n-cond,n-sch
IMAGE_URLS+=",https://launchpad.net/cirros/trunk/0.3.0/+download/cirros-0.3.0-x86_64-disk.img"
ENABLED_SERVICES+=,g-api,g-reg
# Neutron
ENABLED_SERVICES+=,q-svc,q-agt,q-dhcp,q-l3,q-meta
# Enable LBaaS V2
ENABLED_SERVICES+=,q-lbaasv2

#ENABLED_SERVICES+=,q-vpn
# Cinder
ENABLED_SERVICES+=,c-api,c-vol,c-sch

# enable DVR

Q_PLUGIN=ml2
Q_ML2_TENANT_NETWORK_TYPE=vxlan
Q_DVR_MODE=dvr_snat

LOGFILE=$DEST/logs/stack.sh.log

# Old log files are automatically removed after 7 days to keep things neat. Change
# the number of days by setting ``LOGDAYS``.
LOGDAYS=2

# ===== END local.conf =====

# ===== BEGIN local.sh =====

#!/usr/bin/env bash

set -x

# Sample ``local.sh`` that configures two simple webserver instances and sets
# up a Neutron LBaaS Version 2 loadbalancer.

# Keep track of the DevStack directory
TOP_DIR=$(cd $(dirname "$0") && pwd)
BOOT_DELAY=60

# Import common functions
source ${TOP_DIR}/functions

# Use openrc + stackrc for settings
source ${TOP_DIR}/stackrc

# Destination path for installation ``DEST``
DEST=${DEST:-/opt/stack}

# Additional Variables
IMAGE_NAME="cirros"
SUBNET_NAME="private-subnet"

cat > ${TOP_DIR}/webserver.sh <<EOF
#!/bin/sh

MYIP=\$(/sbin/ifconfig eth0|grep 'inet addr'|awk -F: '{print \$2}'| awk '{print \$1}')
while true; do
    echo -e "HTTP/1.0 200 OK\r\n\r\nWelcome to \$MYIP\r\n" | sudo nc -l -p 80
done
EOF

cat > ${TOP_DIR}/webserver-443.sh <<EOF
#!/bin/sh

MYIP=\$(/sbin/ifconfig eth0|grep 'inet addr'|awk -F: '{print \$2}'| awk '{print \$1}')
while true; do
    echo -e "HTTP/1.0 200 OK\r\n\r\nWelcome to \$MYIP port 443" | sudo nc -l -p 443
done
EOF

chmod 755 ${TOP_DIR}/webserver.sh
chmod 755 ${TOP_DIR}/webserver-443.sh

if is_service_enabled nova; then

    # Get OpenStack demo user auth
    source ${TOP_DIR}/openrc demo demo

    # Create an SSH key to use for the instances
    HOST=$(echo $HOSTNAME | cut -d"." -f1)
    DEVSTACK_LBAAS_SSH_KEY_NAME=${HOST}_DEVSTACK_LBAAS_SSH_KEY_RSA
    DEVSTACK_LBAAS_SSH_KEY_DIR=${TOP_DIR}
    DEVSTACK_LBAAS_SSH_KEY=${DEVSTACK_LBAAS_SSH_KEY_DIR}/${DEVSTACK_LBAAS_SSH_KEY_NAME}
    rm -f ${DEVSTACK_LBAAS_SSH_KEY}.pub ${DEVSTACK_LBAAS_SSH_KEY}
    ssh-keygen -b 2048 -t rsa -f ${DEVSTACK_LBAAS_SSH_KEY} -N ""
    nova keypair-add --pub_key=${DEVSTACK_LBAAS_SSH_KEY}.pub ${DEVSTACK_LBAAS_SSH_KEY_NAME}

    # Add tcp/22,80 and icmp to default security group
    nova secgroup-add-rule default tcp 22 22 0.0.0.0/0
    nova secgroup-add-rule default tcp 80 80 0.0.0.0/0
    nova secgroup-add-rule default tcp 443 443 0.0.0.0/0
    nova secgroup-add-rule default icmp -1 -1 0.0.0.0/0

    # Get Image id
    IMAGE_ID=$(glance image-list | awk -v image=${IMAGE_NAME} '$0 ~ image {print $2}' | head -1)

    # Get Network id
    NET_ID=$(neutron subnet-show ${SUBNET_NAME} | awk '/network_id/ {print $4}')
    SUBNET_ID=$(neutron subnet-show ${SUBNET_NAME} | awk '/ id / {print $4}')

    FIP_ID_1=$(neutron floatingip-create public|grep "| id " |awk '{print $4}')
    FIP_ID_2=$(neutron floatingip-create public|grep "| id " |awk '{print $4}')
# FIP_ID_3=$(neutron floatingip-create public|grep "| id " |awk '{print $4}')
# FIP_ID_4=$(neutron floatingip-create public|grep "| id " |awk '{print $4}')

    FIP_1=$(neutron floatingip-show $FIP_ID_1 | grep "| floating_ip_address " | awk {'print $4 }')
    FIP_2=$(neutron floatingip-show $FIP_ID_2 | grep "| floating_ip_address " | awk {'print $4 }')
# FIP_3=$(neutron floatingip-show $FIP_ID_3 | grep "| floating_ip_address " | awk {'print $4 }')
# FIP_4=$(neutron floatingip-show $FIP_ID_4 | grep "| floating_ip_address " | awk {'print $4 }')

    PORT_ID_1=$(neutron port-create private | grep "| id " | awk '{print $4}')
    PORT_ID_2=$(neutron port-create private | grep "| id " | awk '{print $4}')
# PORT_ID_3=$(neutron port-create private | grep "| id " | awk '{print $4}')
# PORT_ID_4=$(neutron port-create private | grep "| id " | awk '{print $4}')

    neutron floatingip-associate $FIP_ID_1 $PORT_ID_1
    neutron floatingip-associate $FIP_ID_2 $PORT_ID_2
# neutron floatingip-associate $FIP_ID_3 $PORT_ID_3
# neutron floatingip-associate $FIP_ID_4 $PORT_ID_4

    IP_1=$(neutron floatingip-show $FIP_ID_1 | grep "| fixed_ip_address " | awk '{print $4}')
    IP_2=$(neutron floatingip-show $FIP_ID_2 | grep "| fixed_ip_address " | awk '{print $4}')
# IP_3=$(neutron floatingip-show $FIP_ID_3 | grep "| fixed_ip_address " | awk '{print $4}')
# IP_4=$(neutron floatingip-show $FIP_ID_4 | grep "| fixed_ip_address " | awk '{print $4}')

    # Boot some instances
# NOVA_BOOT_ARGS="--key-name ${DEVSTACK_LBAAS_SSH_KEY_NAME} --image ${IMAGE_ID} --flavor 1 --nic net-id=$NET_ID"
    NOVA_BOOT_ARGS="--key-name ${DEVSTACK_LBAAS_SSH_KEY_NAME} --image ${IMAGE_ID} --flavor 1 --nic"

    nova boot ${NOVA_BOOT_ARGS} port-id=$PORT_ID_1 node1 --poll
    nova boot ${NOVA_BOOT_ARGS} port-id=$PORT_ID_2 node2 --poll
# nova boot ${NOVA_BOOT_ARGS} port-id=$PORT_ID_3 node3
# nova boot ${NOVA_BOOT_ARGS} port-id=$PORT_ID_4 node4

# echo "Waiting ${BOOT_DELAY} seconds for instances to boot"
# sleep ${BOOT_DELAY}

    nova list

    ssh-keygen -R ${FIP_1}
    ssh-keygen -R ${FIP_2}
# ssh-keygen -R ${FIP_3}
# ssh-keygen -R ${FIP_4}

    # Run a simple web server on the instances

    sleep 20 # getting "connection refused", might not be fully running yet.
    scp -i ${DEVSTACK_LBAAS_SSH_KEY} -o StrictHostKeyChecking=no ${TOP_DIR}/webserver.sh cirros@${FIP_1}:webserver.sh
    scp -i ${DEVSTACK_LBAAS_SSH_KEY} -o StrictHostKeyChecking=no ${TOP_DIR}/webserver.sh cirros@${FIP_2}:webserver.sh

# scp -i ${DEVSTACK_LBAAS_SSH_KEY} -o StrictHostKeyChecking=no ${TOP_DIR}/webserver.sh cirros@${FIP_3}:webserver.sh
# scp -i ${DEVSTACK_LBAAS_SSH_KEY} -o StrictHostKeyChecking=no ${TOP_DIR}/webserver.sh cirros@${FIP_4}:webserver.sh

    screen_process node1 "ssh -i ${DEVSTACK_LBAAS_SSH_KEY} -o StrictHostKeyChecking=no cirros@${FIP_1} ./webserver.sh"
    screen_process node2 "ssh -i ${DEVSTACK_LBAAS_SSH_KEY} -o StrictHostKeyChecking=no cirros@${FIP_2} ./webserver.sh"
# screen_process node3 "ssh -i ${DEVSTACK_LBAAS_SSH_KEY} -o StrictHostKeyChecking=no cirros@${FIP_3} ./webserver.sh"
# screen_process node4 "ssh -i ${DEVSTACK_LBAAS_SSH_KEY} -o StrictHostKeyChecking=no cirros@${FIP_4} ./webserver.sh"

fi

if is_service_enabled q-lbaasv2; then

    echo "AJM: beginning LB setup"
    ip netns
    neutron lbaas-loadbalancer-create --name lb1 ${SUBNET_NAME}
# neutron lbaas-loadbalancer-create --name lb2 ${SUBNET_NAME}
    sleep 10
    echo "AJM: after loadbalancer create"
    ip netns
    neutron lbaas-listener-create --loadbalancer lb1 --protocol HTTP --protocol-port 80 --name listener1
# neutron lbaas-listener-create --loadbalancer lb2 --protocol HTTP --protocol-port 80 --name listener2
    sleep 10
    echo "AJM: after listener-create"
    ip netns

    neutron lbaas-pool-create --lb-algorithm ROUND_ROBIN --listener listener1 --protocol HTTP --name pool1
# neutron lbaas-pool-create --lb-algorithm ROUND_ROBIN --listener listener2 --protocol HTTP --name pool2
    sleep 10
    echo "AJM: after pool create"
    ip netns

    neutron lbaas-member-create --subnet ${SUBNET_NAME} --address ${IP_1} --protocol-port 80 pool1
    neutron lbaas-member-create --subnet ${SUBNET_NAME} --address ${IP_2} --protocol-port 80 pool1

    echo "AJM: after member create"
    ip netns

# neutron lbaas-member-create --subnet ${SUBNET_NAME} --address ${IP_3} --protocol-port 80 pool2
# neutron lbaas-member-create --subnet ${SUBNET_NAME} --address ${IP_4} --protocol-port 80 pool2

    FIP_ID_LB1=$(neutron floatingip-create public|grep "| id " |awk '{print $4}')
    LB1_VIP_PORT=$(neutron lbaas-loadbalancer-show lb1 | grep "| vip_port_id " | awk '{print $4}')

    echo "AJM: after fip create"
    ip netns

# FIP_ID_LB2=$(neutron floatingip-create public|grep "| id " |awk '{print $4}')
# LB2_VIP_PORT=$(neutron lbaas-loadbalancer-show lb2 | grep "| vip_port_id " | awk '{print $4}')

    neutron floatingip-associate $FIP_ID_LB1 $LB1_VIP_PORT
    echo "AJM: at the end"
    ip netns
# neutron floatingip-associate $FIP_ID_LB2 $LB2_VIP_PORT

# neutron lbaas-member-create --subnet ${SUBNET_NAME} --address ${IP1} --protocol-port 443 pool2
fi
# ===== END local.sh

ubuntu@devstack1:~/devstack$ neutron floatingip-list
+--------------------------------------+------------------+---------------------+--------------------------------------+
| id | fixed_ip_address | floating_ip_address | port_id |
+--------------------------------------+------------------+---------------------+--------------------------------------+
| 1192f5d0-f2a8-4abe-b0f2-68b7977f6334 | 10.0.0.5 | 172.24.4.4 | ac203959-db40-4709-8eea-639563dd4be6 |
| 67846392-e9dc-4f59-b47d-e25345c25dd1 | 10.0.0.4 | 172.24.4.3 | ba173e89-abc1-4b7b-a23c-55bc8808a6a6 |
| ca36b325-6d40-4747-89e6-67b2c7ffbf27 | 10.0.0.6 | 172.24.4.6 | f132d74c-270c-4bd3-abde-b567f01e3f5f |
+--------------------------------------+------------------+---------------------+--------------------------------------+

ubuntu@devstack1:~/devstack$ curl 10.0.0.6
Welcome to 10.0.0.5

ubuntu@devstack1:~/devstack$ curl 10.0.0.6
Welcome to 10.0.0.4

ubuntu@devstack1:~/devstack$ curl 10.0.0.6
Welcome to 10.0.0.5

ubuntu@devstack1:~/devstack$ curl 10.0.0.6
Welcome to 10.0.0.4

ubuntu@devstack1:~/devstack$ curl 172.24.4.3
Welcome to 10.0.0.4

ubuntu@devstack1:~/devstack$ curl 172.24.4.4
Welcome to 10.0.0.5

ubuntu@devstack1:~/devstack$ curl 172.24.4.6
curl: (7) Failed to connect to 172.24.4.6 port 80: No route to host

[Hong Hui Xiao (xiaohhui)]

The reason of this bug seems to be that you are using DVR. First of all, I can't see any benefit using DVR in a all in one env(You are deploying devstack all in one host, right?).
And I can't see the bug with a legacy router. But I do see the bug in DVR. There are forwarding rules for floatingip of vms but no forwarding rules for the vip port. So, you can't reach the vip's floatingip.
I will look into it more.