LBAAS: VIP continues to process request even after it is deleted

[Vinod Nair]

Even after deleting the VIP, ha proxy config and even the fip associated with it it is not removed.

root@cs-scale-1:~# neutron lb-vip-list
+--------------------------------------+-------------+----------+----------+----------------+--------+
| id | name | address | protocol | admin_state_up | status |
+--------------------------------------+-------------+----------+----------+----------------+--------+
| bb49c50a-2e5b-47b2-95dd-6e3ff1dd26ac | FE-SSL-VIP1 | 11.1.0.3 | HTTPS | True | ACTIVE |
+--------------------------------------+-------------+----------+----------+----------------+--------+
root@cs-scale-1:~# neutron floatingip-list
+--------------------------------------+------------------+---------------------+--------------------------------------+
| id | fixed_ip_address | floating_ip_address | port_id |
+--------------------------------------+------------------+---------------------+--------------------------------------+
| 08303b98-6f26-4342-bc5d-53ce5a00d463 | | 10.87.126.134 | |
| 0d9b7f93-2a43-4c61-9993-025e869ad6b8 | | 10.87.126.135 | |
| 22accd5a-b427-4eb0-853b-27ecdc49840f | | 10.87.126.136 | |
| 798f0b07-9d85-44c5-8cd3-bba454603744 | | 10.87.126.133 | |
| de1393ca-3732-4d05-a3bb-769cc8f1a0b5 | | 10.87.126.131 | |
| ed3de3d4-1e57-49ac-8ff6-afbad33bfd7b | 11.1.0.3 | 10.87.126.132 | 06c8221e-716f-40da-88b9-debef1ec1e4f |
+--------------------------------------+------------------+---------------------+--------------------------------------+

root@cs-scale-1:~# neutron lb-vip-delete bb49c50a-2e5b-47b2-95dd-6e3ff1dd26ac
Deleted vip: bb49c50a-2e5b-47b2-95dd-6e3ff1dd26ac
root@cs-scale-1:~# neutron lb-vip-list

root@cs-scale-1:~# neutron port-list | grep 11.1.0.3

root@cs-scale-5:/# cat /var/lib/contrail/loadbalancer/3fb291de-31d3-41b7-8591-5389e310a8de/etc/haproxy/haproxy.cfg
global
    daemon
    user nobody
    group nogroup
    stats socket /var/lib/contrail/loadbalancer/3fb291de-31d3-41b7-8591-5389e310a8de/etc/haproxy/haproxy.cfg.sock mode 0666 level user

defaults
    log global
    retries 3
    option redispatch
    timeout connect 5000
    timeout client 50000
    timeout server 50000

listen contrail-config-stats :5937
    mode http
    stats enable
    stats uri /
    stats auth haproxy:contrail123

frontend bb49c50a-2e5b-47b2-95dd-6e3ff1dd26ac
    bind 11.1.0.3:443 ssl crt /store/RSA4096.pem
    mode tcp
    default_backend 3fb291de-31d3-41b7-8591-5389e310a8de

backend 3fb291de-31d3-41b7-8591-5389e310a8de
    mode tcp
    balance roundrobin
    server 1283ae05-42b7-41ee-b572-0a17a063df6e 12.1.0.3:8080 weight 1
    server 522e4895-d0ba-4ecc-8a9e-aa629fa11476 12.1.0.4:8080 weight 1

[Prakash Bailkeri (prakashmb)]

I debugged this further on Vinod¹s setup. I see the issue of RabbitMQ here
again.

The active service mon has NOT received the update/delete of virtual-ip.
This can be seen by running ifmap-view on the cs-scale-2 where SVC mon is
active.

        virtual-ip = FE-SSL-VIP1
  loadbalancer-pool = FE-POOL1
  Virtual-machine-interface = bb49c50a-2e5b-47b2-95dd-6e3ff1dd26ac

I also see an error in contrail-api logs about green let exit due to
rabbit error.

ERROR:cs-scale-3:contrail-api:Config:0:__default__ [SYS_ERR]: VncApiError:
RabbitMQ connection down
WARNING:cs-scale-3:contrail-api:Config:0:__default__ [SYS_NOTICE]:
VncApiError: RabbitMQ connection ESTABLISHED <Connection:
amqp://guest@13.1.0.10:5673// at 0x7f4772b7d810>
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/gevent/greenlet.py", line 327, in
run
    result = self._run(*self.args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/cfgm_common/vnc_kombu.py", line
112, in _connection_watch
    self._reconnect()
  File "/usr/lib/python2.7/dist-packages/cfgm_common/vnc_kombu.py", line
99, in _reconnect
    callbacks=[self._subscribe])
  File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 357, in
__init__
    self.revive(self.channel)
  File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 369, in
revive
    self.declare()
  File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 379, in
declare
    queue.declare()
  File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 505, in
declare
    self.queue_declare(nowait, passive=False)
  File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 531, in
queue_declare
    nowait=nowait)
  File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 1258, in
queue_declare
    (50, 11), # Channel.queue_declare_ok
  File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line
69, in wait
    return self.dispatch_method(method_sig, args, content)
  File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line
87, in dispatch_method
    return amqp_method(self, args)
  File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 241, in
_close
    reply_code, reply_text, (class_id, method_id), ChannelError,
NotFound: Queue.declare: (404) NOT_FOUND - home node
'rabbit@cs-scale-1-ctrl' of durable queue 'vnc_config.cs-scale-3-9100' in
vhost '/' is down or inaccessible
<Greenlet at 0x7f4772bf5870: <bound method
VncServerKombuClient._connection_watch of
<vnc_cfg_api_server.vnc_cfg_ifmap.VncServerKombuClient object at
0x7f4772c9afd0>>> failed with NotFound

This looks similar to the issue I debugged for
https://bugs.launchpad.net/juniperopenstack/+bug/1463642

[Vinod Nair (vinodnair)]

After restarting contrail-config on all the nodes, retested vip deletion and is working fine.