Bug #1514510 “Delete of last member does not remove it from hapr...” : Bugs : octavia

Eran Raichstein (eranra) on 2015-12-17

Changed in octavia:
assignee:	nobody → Eran Raichstein (eranra)

Revision history for this message

Eran Raichstein (eranra) wrote on 2015-12-22:

#1

Download full text (24.6 KiB)

RE-RUN BUG # 1514510

The bug is still reproducible on current Octavia version. I deleted two members from the pool and both are removed from Neutron member list but only first one is removed from configuration file

1. using latest octavia / devstack versions::

Octavia is on ::
commit cd5a50adb3a1a42bb1dfd5cc6e1d03029cd795e5
Author: Michael Johnson <email address hidden>
Date: Fri Dec 18 20:06:23 2015 +0000

2. Added /etc/octavia/octavia.conf loadbalancer_topology = ACTIVE_STANDBY (to force active standby configuration)
3. Executed ./stack.sh

DONE

4. Executed script to create : members, load balancer, listener, pool, and attach members (script content is listed bellow)

./create_octavia_demo_environment.sh fast

5. Checked that the LB works (didn't check for this bug any fail-over scenario)

curl 20.0.0.13

DONE

6. Listed nova VM's:

nova list --all-tenants

RE-RUN BUG # 1514510

The bug is still reproducible on current Octavia version. I deleted two members from the pool and both are removed from Neutron member list but only first one is removed from configuration file

1. using latest octavia / devstack versions::

Octavia is on ::
commit cd5a50adb3a1a42bb1dfd5cc6e1d03029cd795e5
Author: Michael Johnson <johnsomor@gmail.com>
Date:   Fri Dec 18 20:06:23 2015 +0000

2. Added /etc/octavia/octavia.conf loadbalancer_topology = ACTIVE_STANDBY (to force active standby configuration)
3. Executed ./stack.sh

DONE

4. Executed script to create : members, load balancer, listener, pool, and attach members (script content is listed bellow)

./create_octavia_demo_environment.sh fast

5. Checked that the LB works (didn't check for this bug any fail-over scenario)

curl 20.0.0.13

DONE

6. Listed nova VM's:

nova list --all-tenants

DONE

7. executed 'neutron lbaas-member-list'

neutron lbaas-member-list

+--------------------------------------+----------+---------------+--------+--------------------------------------+----------------+
| id                                   | address  | protocol_port | weight | subnet_id                            | admin_state_up |
+--------------------------------------+----------+---------------+--------+--------------------------------------+----------------+
| 340c12af-82d0-4f07-8a7a-89eb99fa6ae1 | 10.0.0.7 |            80 |      1 | a7c9d665-60db-445f-ad86-49da016b52c9 | True           |
| 3c19fc1f-7a12-4705-9971-46b2c5fc1ae7 | 10.0.0.6 |            80 |      1 | a7c9d665-60db-445f-ad86-49da016b52c9 | True           |
| 5899e0a7-5730-4814-ba4e-05b93022cfb7 | 10.0.0.4 |            80 |      1 | a7c9d665-60db-445f-ad86-49da016b52c9 | True           |
| d19ccd0b-645a-4951-a400-031ea37d9aad | 10.0.0.5 |            80 |      1 | a7c9d665-60db-445f-ad86-49da016b52c9 | True           |
| e411f036-a1d6-465d-97a2-06b8074c32da | 10.0.0.8 |            80 |      1 | a7c9d665-60db-445f-ad86-49da016b52c9 | True           |
+--------------------------------------+----------+---------------+--------+--------------------------------------+----------------+

DONE

8. looked on the 'haproxy configuration file'

ssh -o StrictHostKeyChecking=no -i /etc/octavia/.ssh/octavia_ssh_key ubuntu@20.0.0.14

ubuntu@amphora-9be90952-fa15-40bc-bab2-13d10b6583fd:~$ ps -ef | grep haproxy
haproxy   1301     1  0 12:56 ?        00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
nobody   12497     1  0 13:39 ?        00:00:00 /usr/sbin/haproxy -f /var/lib/octavia/cc38634b-5e5e-4f51-a3b9-b706c7137e74/haproxy.cfg -L _bEiJ83C7eKHa_8qm6iSjbzL04U -p /var/lib/octavia/cc38634b-5e5e-4f51-a3b9-b706c7137e74/cc38634b-5e5e-4f51-a3b9-b706c7137e74.pid -sf 11283
ubuntu   13904 13823  0 13:43 pts/2    00:00:00 grep --color=auto haproxy
ubuntu@amphora-9be90952-fa15-40bc-bab2-13d10b6583fd:~$

ubuntu@amphora-9be90952-fa15-40bc-bab2-13d10b6583fd:~$ cat /var/lib/octavia/cc38634b-5e5e-4f51-a3b9-b706c7137e74/haproxy.cfg
# Configuration for lb1
global
    daemon
    user nobody
    group nogroup
    log /dev/log local0
    log /dev/log local1 notice
    stats socket /var/lib/octavia/cc38634b-5e5e-4f51-a3b9-b706c7137e74.sock mode 0666 level user

defaults
    log global
    retries 3
    option redispatch
    timeout connect 5000
    timeout client 50000
    timeout server 50000

peers cc38634b5e5e4f51a3b9b706c7137e74_peers
    peer _bEiJ83C7eKHa_8qm6iSjbzL04U 20.0.0.14:1025
    peer gIuCnMhBYqG_eFpqe_fF29Rv4tU 20.0.0.15:1025

frontend cc38634b-5e5e-4f51-a3b9-b706c7137e74
    option tcplog
    option forwardfor
    bind 20.0.0.13:80
    mode http
    default_backend 225ad656-2610-4ce5-bf1d-bcedd3cae9ea

backend 225ad656-2610-4ce5-bf1d-bcedd3cae9ea
    mode http
    balance roundrobin
    option forwardfor
    server 340c12af-82d0-4f07-8a7a-89eb99fa6ae1 10.0.0.7:80 weight 1
    server 3c19fc1f-7a12-4705-9971-46b2c5fc1ae7 10.0.0.6:80 weight 1
    server 5899e0a7-5730-4814-ba4e-05b93022cfb7 10.0.0.4:80 weight 1
    server d19ccd0b-645a-4951-a400-031ea37d9aad 10.0.0.5:80 weight 1
    server e411f036-a1d6-465d-97a2-06b8074c32da 10.0.0.8:80 weight 1

9. removed 2 members (using UUID)

(remove 10.0.0.4)
neutron lbaas-member-delete 5899e0a7-5730-4814-ba4e-05b93022cfb7 225ad656-2610-4ce5-bf1d-bcedd3cae9ea
Deleted member: 5899e0a7-5730-4814-ba4e-05b93022cfb7

(remove 10.0.0.5)
neutron lbaas-member-delete d19ccd0b-645a-4951-a400-031ea37d9aad 225ad656-2610-4ce5-bf1d-bcedd3cae9ea
Deleted member: d19ccd0b-645a-4951-a400-031ea37d9aad

10. looked on the 'haproxy configuration file'

ubuntu@amphora-9be90952-fa15-40bc-bab2-13d10b6583fd:~$ cat /var/lib/octavia/cc38634b-5e5e-4f51-a3b9-b706c7137e74/haproxy.cfg
# Configuration for lb1
global
    daemon
    user nobody
    group nogroup
    log /dev/log local0
    log /dev/log local1 notice
    stats socket /var/lib/octavia/cc38634b-5e5e-4f51-a3b9-b706c7137e74.sock mode 0666 level user

defaults
    log global
    retries 3
    option redispatch
    timeout connect 5000
    timeout client 50000
    timeout server 50000

peers cc38634b5e5e4f51a3b9b706c7137e74_peers
    peer _bEiJ83C7eKHa_8qm6iSjbzL04U 20.0.0.14:1025
    peer gIuCnMhBYqG_eFpqe_fF29Rv4tU 20.0.0.15:1025

frontend cc38634b-5e5e-4f51-a3b9-b706c7137e74
    option tcplog
    option forwardfor
    bind 20.0.0.13:80
    mode http
    default_backend 225ad656-2610-4ce5-bf1d-bcedd3cae9ea

backend 225ad656-2610-4ce5-bf1d-bcedd3cae9ea
    mode http
    balance roundrobin
    option forwardfor
    server 340c12af-82d0-4f07-8a7a-89eb99fa6ae1 10.0.0.7:80 weight 1
    server 3c19fc1f-7a12-4705-9971-46b2c5fc1ae7 10.0.0.6:80 weight 1
    server d19ccd0b-645a-4951-a400-031ea37d9aad 10.0.0.5:80 weight 1
    server e411f036-a1d6-465d-97a2-06b8074c32da 10.0.0.8:80 weight 1

11. executed 'neutron lbaas-member-list'

neutron lbaas-member-list

$ neutron lbaas-member-list 225ad656-2610-4ce5-bf1d-bcedd3cae9ea

+--------------------------------------+----------+---------------+--------+--------------------------------------+----------------+
| id                                   | address  | protocol_port | weight | subnet_id                            | admin_state_up |
+--------------------------------------+----------+---------------+--------+--------------------------------------+----------------+
| 340c12af-82d0-4f07-8a7a-89eb99fa6ae1 | 10.0.0.7 |            80 |      1 | a7c9d665-60db-445f-ad86-49da016b52c9 | True           |
| 3c19fc1f-7a12-4705-9971-46b2c5fc1ae7 | 10.0.0.6 |            80 |      1 | a7c9d665-60db-445f-ad86-49da016b52c9 | True           |
| e411f036-a1d6-465d-97a2-06b8074c32da | 10.0.0.8 |            80 |      1 | a7c9d665-60db-445f-ad86-49da016b52c9 | True           |
+--------------------------------------+----------+---------------+--------+--------------------------------------+----------------+

============================================================================================================================================
============================================================================================================================================
============================================================================================================================================
============================================================================================================================================

Adittional information 
============================
OCTAVIA configuration file /etc/octavia/octavia.conf (loadbalancer_topology = ACTIVE_STANDBY)::

[DEFAULT]
verbose = True
debug = True
api_handler = queue_producer
# Print more verbose output (set logging level to INFO instead of default WARNING level).
# verbose = False
# Print debugging output (set logging level to DEBUG instead of default WARNING level).
# debug = False
# bind_host = 0.0.0.0
# bind_port = 9876
# api_handler = simulated_handler
#
# Plugin options are hot_plug_plugin (Hot-pluggable controller plugin)
#
# octavia_plugins = hot_plug_plugin

# Region in Identity service catalog to use for communication with the OpenStack services.
#
# os_region_name =

# Hostname to be used by the host machine for services running on it.
# The default value is the hostname of the host machine.
# host =

[database]
connection = mysql+pymysql://root:password@172.25.8.83:3306/octavia
# This line MUST be changed to actually run the plugin.
# Example:
# connection = mysql+pymysql://root:pass@127.0.0.1:3306/octavia
# Replace 127.0.0.1 above with the IP address of the database used by the
# main octavia server. (Leave it as is if the database runs on this host.)

# connection = mysql+pymysql://

# NOTE: In deployment the [database] section and its connection attribute may
# be set in the corresponding core plugin '.ini' file. However, it is suggested
# to put the [database] section and its connection attribute in this
# configuration file.

[health_manager]
# bind_ip = 0.0.0.0
# bind_port = 5555
# controller_ip_port_list example: 127.0.0.1:5555, 127.0.0.1:5555
# controller_ip_port_list =
# failover_threads = 10
# status_update_threads = 50
# heartbeat_interval = 10
heartbeat_key =insecure
# heartbeat_timeout = 60
# health_check_interval = 3
# sock_rlimit = 0

[keystone_authtoken]
auth_version = 2
admin_password = password
admin_tenant_name = admin
admin_user = admin
auth_uri = http://172.25.8.83:35357/v2.0
# auth_uri = https://localhost:5000/v3
# admin_user = octavia
# admin_password = password
# admin_tenant_name = service
# insecure = False

[keystone_authtoken_v3]
# If using Keystone v3
# admin_user_domain = default
# admin_project_domain = default

[certificates]
ca_private_key_passphrase = foobar
ca_private_key = /etc/octavia/certs/private/cakey.pem
ca_certificate = /etc/octavia/certs/ca_01.pem
# cert_generator = local_cert_generator

# For local certificate signing (development only):
# ca_certificate = /etc/ssl/certs/ssl-cert-snakeoil.pem
# ca_private_key = /etc/ssl/private/ssl-cert-snakeoil.key
# ca_private_key_passphrase =
# signing_digest = sha256
# storage_path = /var/lib/octavia/certificates/

# For the TLS management
# Certificate Manager options are local_cert_manager
#                                 barbican_cert_manager
# cert_manager=local_cert_manager
# For Barbican authentication (if using any Barbican based cert class)
# barbican_auth = barbican_acl_auth

[anchor]
# Use OpenStack anchor to sign the amphora REST API certificates
# url = http://localhost:9999/v1/sign/default
# username = myusername
# password = simplepassword

[networking]
# Network to communicate with amphora
# lb_network_name =
# The maximum attempts to retry an action with the networking service.
# max_retries = 15
# Seconds to wait before retrying an action with the networking service.
# retry_interval = 1

[haproxy_amphora]
server_ca = /etc/octavia/certs/ca_01.pem
client_cert = /etc/octavia/certs/client.pem
key_path = /etc/octavia/.ssh/octavia_ssh_key
base_path = /var/lib/octavia
base_cert_dir = /var/lib/octavia/certs
# haproxy_template = /var/lib/octavia/custom_template
# The following may need to be an absolute location:
# base_log_dir = /logs
connection_max_retries = 300
connection_retry_interval = 5
# Cert manager options are local_cert_manager,
#                          barbican_cert_manager,
#
# cert_manager = barbican_cert_manager

# SSH Driver specific
username = ubuntu
# key_path = /opt/stack/.ssh/id_rsa

# REST Driver specific
# bind_host = 0.0.0.0
# bind_port = 9191
# haproxy_cmd = /usr/sbin/haproxy
# respawn_count = 2
# respawn_interval = 2
# Change for production to a ram drive
# haproxy_cert_dir = /tmp
# use_upstart = True

# Maximum number of entries that can fit in the stick table.
# The size supports "k", "m", "g" suffixes.
# haproxy_stick_table_size = 10k

[controller_worker]
amp_network = 7487b291-f3df-4933-a81b-924bc5e643df
amp_secgroup_list = 13913e98-72b7-4675-a201-1de09ff523ee
amp_image_id = b7deda9c-08ac-45a2-a64c-1dbddbac7a55
amp_ssh_key_name = octavia_ssh_key
network_driver = allowed_address_pairs_driver
compute_driver = compute_nova_driver
amphora_driver = amphora_haproxy_ssh_driver
amp_flavor_id = 10
# amp_active_retries = 10
# amp_active_wait_sec = 10
# Nova parameters to use when booting amphora
# amp_flavor_id =
# amp_image_id =
# amp_ssh_key_name =
# amp_ssh_allowed_access = True
# amp_network =
# amp_secgroup_list =
# client_ca = /etc/octavia/certs/ca_01.pem

# Amphora driver options are amphora_noop_driver,
#                            amphora_haproxy_rest_driver,
#                            amphora_haproxy_ssh_driver
#
# amphora_driver = amphora_noop_driver
#
# Compute driver options are compute_noop_driver
#                            compute_nova_driver
#
# compute_driver = compute_noop_driver
#
# Network driver options are network_noop_driver
#                            allowed_address_pairs_driver
#
# network_driver = network_noop_driver
#
# Certificate Generator options are local_cert_generator
#                                   barbican_cert_generator
#                                   anchor_cert_generator
# cert_generator = local_cert_generator
#
# Load balancer topology options are SINGLE, ACTIVE_STANDBY
loadbalancer_topology = ACTIVE_STANDBY

[task_flow]
# engine = serial
# max_workers = 5

[oslo_messaging_rabbit]
rabbit_hosts = localhost:5672
rabbit_port = 5672
# Rabbit and HA configuration:
# rabbit_userid = octavia
# rabbit_password = password
# rabbit_port = 5672

# For HA, specify queue nodes in cluster, comma delimited:
#   For example: rabbit_hosts = 192.168.50.8:5672, 192.168.50.9:5672
# rabbit_hosts = localhost:5672

[oslo_messaging]
topic = octavia_prov
rpc_thread_pool_size = 2
# Queue Consumer Thread Pool Size
# rpc_thread_pool_size = 2

# Topic (i.e. Queue) Name
# topic = octavia_prov

[house_keeping]
# Interval in seconds to initiate spare amphora checks
# spare_check_interval = 30
# spare_amphora_pool_size = 0

# Cleanup interval for Deleted amphora
# cleanup_interval = 30
# Amphora expiry age in seconds. Default is 1 week
# amphora_expiry_age = 604800

[amphora_agent]
# agent_server_ca = /etc/octavia/certs/client_ca.pem
# agent_server_cert = /etc/octavia/certs/server.pem
# agent_server_network_dir = /etc/network/interfaces.d/
# agent_server_network_file =

[keepalived_vrrp]
# Amphora Role/Priority advertisement interval in seconds
# vrrp_advert_int = 1

# Service health check interval and success/fail count
# vrrp_check_interval = 5
# vrpp_fail_count = 2
# vrrp_success_count = 2

# Amphora MASTER gratuitous ARP refresh settings
# vrrp_garp_refresh_interval = 5
# vrrp_garp_refresh_count = 2

============================

Using automated script to create bug environment::
============================

#!/usr/bin/env bash

# Keep track of the DevStack directory
TOP_DIR=/home/stack/devstack

cd ${TOP_DIR}
source ${TOP_DIR}/openrc admin admin
  
if [ "$1" = "fast" ]; then
	echo "Using small delays (physical machine)"
	WAIT_AMPHORA_DELAY=120
	WAIT_OCTAVIA_DELAY=40
	BOOT_DELAY=80
else
	echo "Using regular delays (virtual machine)"
	WAIT_AMPHORA_DELAY=1200
	WAIT_OCTAVIA_DELAY=240
	BOOT_DELAY=90
fi

# Import common functions
source ${TOP_DIR}/functions

# Use openrc + stackrc for settings
source ${TOP_DIR}/stackrc

# Destination path for installation ``DEST``
DEST=${DEST:-/opt/stack}

if is_service_enabled nova; then

# Get OpenStack admin user auth
    source ${TOP_DIR}/openrc admin admin

# Add tcp/22,80 and icmp to default security group (for admin)
    nova secgroup-add-rule default tcp 22 22 0.0.0.0/0
    nova secgroup-add-rule default tcp 80 80 0.0.0.0/0
    nova secgroup-add-rule default icmp -1 -1 0.0.0.0/0
	
    # Get OpenStack admin user auth for project demo
    source ${TOP_DIR}/openrc admin demo

# Add tcp/22,80 and icmp to default security group (for demo)
    nova secgroup-add-rule default tcp 22 22 0.0.0.0/0
    nova secgroup-add-rule default tcp 80 80 0.0.0.0/0
    nova secgroup-add-rule default icmp -1 -1 0.0.0.0/0

nova keypair-add --pub_key=/etc/octavia/.ssh/octavia_ssh_key.pub nova_ssh_key

# Creating production network
	echo "Creating production network (for load balancer front-end)"
    PRODUCTION_NETWORK_ID=$(neutron net-create --shared production | awk '/ id / {print $4}')
    PRODUCTION_SUBNET_ID=$(neutron subnet-create --name production-subnet --allocation-pool start=20.0.0.10,end=20.0.0.200 production 20.0.0.0/16 | awk '/ id / {print $4}')
	
    neutron router-interface-add router1 production-subnet

# Boot some instances
    NOVA_BOOT_ARGS="--key-name nova_ssh_key --image $(nova image-list | awk '/ cirros-0.3.3-x86_64-disk / {print $2}') --flavor 1 --nic net-id=$(neutron net-list | awk '/ private / {print $2}')"

nova boot ${NOVA_BOOT_ARGS} --meta metering.group=aggregation demonode1
    nova boot ${NOVA_BOOT_ARGS} --meta metering.group=aggregation demonode2
    nova boot ${NOVA_BOOT_ARGS} --meta metering.group=aggregation demonode3
    nova boot ${NOVA_BOOT_ARGS} --meta metering.group=aggregation demonode4
    nova boot ${NOVA_BOOT_ARGS} --meta metering.group=aggregation demonode5

NOVA_BOOT_ARGS="--key-name nova_ssh_key --image $(nova image-list | awk '/ cirros-0.3.3-x86_64-disk / {print $2}') --flavor 1 --nic net-id=$(neutron net-list | awk '/ production / {print $2}')"
	
    nova boot ${NOVA_BOOT_ARGS} stresser1

echo "Waiting ${BOOT_DELAY} seconds for instances to boot"
    sleep ${BOOT_DELAY}

IP1=$(nova show demonode1 | grep "private network" | awk '{print $5}')
    IP2=$(nova show demonode2 | grep "private network" | awk '{print $5}')
    IP3=$(nova show demonode3 | grep "private network" | awk '{print $5}')
    IP4=$(nova show demonode4 | grep "private network" | awk '{print $5}')
    IP5=$(nova show demonode5 | grep "private network" | awk '{print $5}')
    SIP1=$(nova show stresser1 | grep "production network" | awk '{print $5}')

ssh-keygen -R ${IP1}
    ssh-keygen -R ${IP2}
    ssh-keygen -R ${IP3}
    ssh-keygen -R ${IP4}
    ssh-keygen -R ${IP5}
    ssh-keygen -R ${SIP1}

source ${TOP_DIR}/openrc admin admin
	
	### Allow access to production network from host
	EXTERNALIP=$(neutron subnet-show public-subnet | awk ' / allocation_pools / {print $5}' | tr -d '",')
	sudo ip route add "20.0.0.0/24" via $EXTERNALIP dev br-ex
	sudo ip route show
	
    # Run a simple web server on the instances
    scp -i /etc/octavia/.ssh/octavia_ssh_key -o StrictHostKeyChecking=no ~/devstack/SharedRepository/CIL/tools/webserver.sh cirros@${IP1}:webserver.sh
    scp -i /etc/octavia/.ssh/octavia_ssh_key -o StrictHostKeyChecking=no ~/devstack/SharedRepository/CIL/tools/webserver.sh cirros@${IP2}:webserver.sh
    scp -i /etc/octavia/.ssh/octavia_ssh_key -o StrictHostKeyChecking=no ~/devstack/SharedRepository/CIL/tools/webserver.sh cirros@${IP3}:webserver.sh
    scp -i /etc/octavia/.ssh/octavia_ssh_key -o StrictHostKeyChecking=no ~/devstack/SharedRepository/CIL/tools/webserver.sh cirros@${IP4}:webserver.sh
    scp -i /etc/octavia/.ssh/octavia_ssh_key -o StrictHostKeyChecking=no ~/devstack/SharedRepository/CIL/tools/webserver.sh cirros@${IP5}:webserver.sh
    
	# Copy simple stress tool
    scp -i /etc/octavia/.ssh/octavia_ssh_key -o StrictHostKeyChecking=no ~/devstack/SharedRepository/CIL/tools/webstress.sh cirros@${SIP1}:webstress.sh

screen_process demonode1 "ssh -i /etc/octavia/.ssh/octavia_ssh_key -o StrictHostKeyChecking=no cirros@${IP1} chmod a=rwx 'webserver.sh && ./webserver.sh'"
    screen_process demonode2 "ssh -i /etc/octavia/.ssh/octavia_ssh_key -o StrictHostKeyChecking=no cirros@${IP2} chmod a=rwx 'webserver.sh && ./webserver.sh'"
    screen_process demonode3 "ssh -i /etc/octavia/.ssh/octavia_ssh_key -o StrictHostKeyChecking=no cirros@${IP3} chmod a=rwx 'webserver.sh && ./webserver.sh'"
    screen_process demonode4 "ssh -i /etc/octavia/.ssh/octavia_ssh_key -o StrictHostKeyChecking=no cirros@${IP4} chmod a=rwx 'webserver.sh && ./webserver.sh'"
    screen_process demonode5 "ssh -i /etc/octavia/.ssh/octavia_ssh_key -o StrictHostKeyChecking=no cirros@${IP5} chmod a=rwx 'webserver.sh && ./webserver.sh'"
fi

if is_service_enabled q-lbaasv2; then

source ${TOP_DIR}/openrc admin demo

echo "Creating load-balancer"
    neutron lbaas-loadbalancer-create --name lb1 production-subnet
	sleep $WAIT_AMPHORA_DELAY
    echo "Creating listener"
	neutron lbaas-listener-create --loadbalancer lb1 --protocol HTTP --protocol-port 80 --name listener1
    sleep $WAIT_OCTAVIA_DELAY
	echo "Creating pool"
    neutron lbaas-pool-create --lb-algorithm ROUND_ROBIN --listener listener1 --protocol HTTP --name pool1
    sleep $WAIT_OCTAVIA_DELAY
    echo "Attaching member demonode1"
	neutron lbaas-member-create  --subnet private-subnet --address ${IP1} --protocol-port 80 pool1
    echo "Attaching member demonode2"
	sleep $WAIT_OCTAVIA_DELAY
    neutron lbaas-member-create  --subnet private-subnet --address ${IP2} --protocol-port 80 pool1
    echo "Attaching member demonode3"
	sleep $WAIT_OCTAVIA_DELAY
    neutron lbaas-member-create  --subnet private-subnet --address ${IP3} --protocol-port 80 pool1
    echo "Attaching member demonode4"
	sleep $WAIT_OCTAVIA_DELAY
    neutron lbaas-member-create  --subnet private-subnet --address ${IP4} --protocol-port 80 pool1
    echo "Attaching member demonode5"
	sleep $WAIT_OCTAVIA_DELAY
    neutron lbaas-member-create  --subnet private-subnet --address ${IP5} --protocol-port 80 pool1
	
	VIP=$(neutron lbaas-loadbalancer-show lb1 | grep "vip_address" | awk '{print $4}')
	VIPPORT=$(neutron lbaas-loadbalancer-show lb1 | grep "vip_port_id" | awk '{print $4}')
	SECURITYGROUPID=$(neutron port-show $VIPPORT | grep "\<security_groups\>" | awk '{print $4}')
	FLOATIPID=$(neutron floatingip-create public | grep "\<id\>" | awk '{print $4}')
	FLOATIP=$(neutron floatingip-list| grep "$FLOATIPID" | awk '{print $5}')

nova secgroup-add-rule $SECURITYGROUPID tcp 22 22 0.0.0.0/0
    nova secgroup-add-rule $SECURITYGROUPID tcp 80 80 0.0.0.0/0
    nova secgroup-add-rule $SECURITYGROUPID icmp -1 -1 0.0.0.0/0	
	
	# assosiate floating IP
	echo "Assigning to VIP $VIP floating IP $FLOATIP"
	neutron floatingip-associate $FLOATIPID $VIPPORT
	# route add default gw 192.168.1.254

# Activate stress tool
	echo "Running stresser against vIP $VIP"
	screen_process stresser1 "ssh -i /etc/octavia/.ssh/octavia_ssh_key -o StrictHostKeyChecking=no cirros@${SIP1} chmod a=rwx 'webstress.sh && ./webstress.sh $VIP'"

fi

============================

Revision history for this message

Eran Raichstein (eranra) wrote on 2015-12-22:

#2

SELECT * FROM octavia.member;

# project_id, id, pool_id, subnet_id, ip_address, protocol_port, weight, operating_status, enabled
, 340c12af-82d0-4f07-8a7a-89eb99fa6ae1, 225ad656-2610-4ce5-bf1d-bcedd3cae9ea, a7c9d665-60db-445f-ad86-49da016b52c9, 10.0.0.7, 80, 1, OFFLINE, 1
, 3c19fc1f-7a12-4705-9971-46b2c5fc1ae7, 225ad656-2610-4ce5-bf1d-bcedd3cae9ea, a7c9d665-60db-445f-ad86-49da016b52c9, 10.0.0.6, 80, 1, OFFLINE, 1
, e411f036-a1d6-465d-97a2-06b8074c32da, 225ad656-2610-4ce5-bf1d-bcedd3cae9ea, a7c9d665-60db-445f-ad86-49da016b52c9, 10.0.0.8, 80, 1, OFFLINE, 1

(only 3 entries, so the info was removed from the data base)

Revision history for this message

Eran Raichstein (eranra) wrote on 2015-12-22:

#3

Bug is understood::

get_delete_member_flow original order:
1. DeleteModelObject
2. ListenerUpdate
3. DeleteMemberInDB
4. MarkLBAndListenerActiveInDB

since DeleteMemberInDB is called after ListenerUpdate the information still contains original (before deleted) list of members. I changed the order (i.e. switched task 2 with task 3) so now the DeleteMemberInDB happens before ListenerUpdate and so the information that sent to Amphora is with the updated list of memebers.

New get_delete_member_flow code looks like::

def get_delete_member_flow(self):
"""Create a flow to delete a member

        :returns: The flow for deleting a member
        """
        delete_member_flow = linear_flow.Flow(constants.DELETE_MEMBER_FLOW)
        delete_member_flow.add(model_tasks.
                               DeleteModelObject(rebind={constants.OBJECT:
                                                         constants.MEMBER}))
        delete_member_flow.add(database_tasks.DeleteMemberInDB(
            requires=constants.MEMBER_ID))
        delete_member_flow.add(amphora_driver_tasks.ListenerUpdate(
            requires=[constants.LISTENER, constants.VIP]))
        delete_member_flow.add(database_tasks.
                               MarkLBAndListenerActiveInDB(
                                   requires=[constants.LOADBALANCER,
                                             constants.LISTENER]))

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-12-22: Fix proposed to octavia (master)

#4

Fix proposed to branch: master
Review: https://review.openstack.org/260605

Changed in octavia:
status:	New → In Progress

Revision history for this message

Eran Raichstein (eranra) wrote on 2015-12-22:

#5

Fixed in: https://review.openstack.org/#/c/260605/

Changed in octavia:
status:	In Progress → Confirmed
status:	Confirmed → Fix Committed

Revision history for this message

Eran Raichstein (eranra) wrote on 2015-12-22:

#6

Confirmed, this is now fixed.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-01-13:

#7

Fix proposed to branch: master
Review: https://review.openstack.org/267045

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-01-13: Change abandoned on octavia (master)

#8

Change abandoned by Eran Raichstein (<email address hidden>) on branch: master
Review: https://review.openstack.org/267045
Reason: Mistake in Change-Id

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-01-15: Fix merged to octavia (master)

#9

Reviewed: https://review.openstack.org/260605
Committed: https://git.openstack.org/cgit/openstack/octavia/commit/?id=080f6101fd56ff799c68e420c158458428e5bc0f
Submitter: Jenkins
Branch: master

commit 080f6101fd56ff799c68e420c158458428e5bc0f
Author: EranRaichstein <email address hidden>
Date: Wed Jan 13 17:59:54 2016 +0200

Fix a problem of memebrs not deleted from pool

Get_delete_member_flow original tasks order was:

    1. DeleteModelObject
    2. ListenerUpdate
    3. DeleteMemberInDB
    4. MarkLBAndListenerActiveInDB

    Since memebrs are deleted from Listener only by DeleteMemberInDB,
    ListenerUpdate is being called with all members (including the
    member that should be deleted). The patch changes the order of
    tasks. Calling DeleteMemberInDB before ListenerUpdate (i.e.
    switch task 2 & 3) and thus the memebr is removed from HAProxy
    List and info is updated to Amphorae as requested.

Change call to DeleteMemberInDB to take member and not member_id
change call to ListenerUpdate to use constants.LOADBALANCER, constants.LISTENER

Change-Id: I159831ca13fc8864798972f75d4b0c6e1fcecf26
Closes-Bug: #1514510

Changed in octavia:
status:	Fix Committed → Fix Released

octavia

Delete of last member does not remove it from haproxy config

Bug Description

Other bug subscribers

Remote bug watches