[kuryr-controller] Kuryr deploys Octavia Listeners in disabled state

Bug #1872962 reported by ITD27M01
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kuryr-kubernetes
Fix Released
Undecided
Maysa de Macedo Souza

Bug Description

Hello,

Some octavia listeners are in disabled state after fresh openshift cluster deployment. As a result, cluster doesn't work properly:

~~~
(openstack) [builder@srv8-lacmacis rtc]$ openstack loadbalancer listener list --disable -f value --project vp_rtc | wc -l
18
~~~

Looks like this change introduced some instabillity:
https://github.com/openstack/kuryr-kubernetes/commit/0e581caa2dfd8d2a8d88b0e480e5e5289f0d5454

As a workaround I have enabled the listeners by openstack utility:

~~~
(openstack) [builder@srv8-lacmacis rtc]$ for i in $(openstack --os-cloud vp_rtc loadbalancer listener list --disable -f value -c id);do openstack --os-cloud vp_rtc loadbalancer listener set --enable --wait $i;done
~~~

I have enabled the body logging on the Octavia side. The body is a $request_body from Kuryr controller. During the deployment Kuryr disables the Octavia listeners for some reason:

~~~
> grep 6a6ac363-e2e5-409c-ab6c-5b51bee7fdef /var/log/nginx/octavia_octavia-publicm_9876_access.log
[15/Apr/2020:13:23:39 +0300] - 172.18.90.208:"POST /v2.0/lbaas/pools HTTP/1.1:/v2.0/lbaas/pools" status: 201, body: {\x22pool\x22: {\x22name\x22: \x22openshift-monitoring/prometheus-k8s:TCP:9091\x22, \x22project_id\x22: \x221d1d08dc2a45454d8c7fdd71e1b713ea\x22, \x22listener_id\x22: \x226a6ac363-e2e5-409c-ab6c-5b51bee7fdef\x22, \x22loadbalancer_id\x22: \x225fde6ab6-613e-4bdd-b73e-7e26e63e71b0\x22, \x22protocol\x22: \x22TCP\x22, \x22lb_algorithm\x22: \x22ROUND_ROBIN\x22, \x22tags\x22: [\x22openshiftClusterID=kaas-f588p\x22]}}

[15/Apr/2020:13:23:45 +0300] - 172.18.90.208:"PUT /v2.0/lbaas/listeners/6a6ac363-e2e5-409c-ab6c-5b51bee7fdef HTTP/1.1:/v2.0/lbaas/listeners/6a6ac363-e2e5-409c-ab6c-5b51bee7fdef" status: 200, body: {\x22listener\x22: {\x22allowed_cidrs\x22: [], \x22admin_state_up\x22: false}}

[15/Apr/2020:13:24:11 +0300] - 172.18.90.208:"PUT /v2.0/lbaas/listeners/6a6ac363-e2e5-409c-ab6c-5b51bee7fdef HTTP/1.1:/v2.0/lbaas/listeners/6a6ac363-e2e5-409c-ab6c-5b51bee7fdef" status: 200, body: {\x22listener\x22: {\x22allowed_cidrs\x22: [], \x22admin_state_up\x22: false}}
~~~

> o loadbalancer listener show 6a6ac363-e2e5-409c-ab6c-5b51bee7fdef
+-----------------------------+----------------------------------------------+
| Field | Value |
+-----------------------------+----------------------------------------------+
| admin_state_up | True |
| connection_limit | -1 |
| created_at | 2020-04-15T10:23:32 |
| default_pool_id | f44dc6b3-29f7-4f08-970a-d566a572f4d2 |
| default_tls_container_ref | None |
| description | |
| id | 6a6ac363-e2e5-409c-ab6c-5b51bee7fdef |
| insert_headers | None |
| l7policies | |
| loadbalancers | 5fde6ab6-613e-4bdd-b73e-7e26e63e71b0 |
| name | openshift-monitoring/prometheus-k8s:TCP:9091 |
| operating_status | ONLINE |
| project_id | 1d1d08dc2a45454d8c7fdd71e1b713ea |
| protocol | TCP |
| protocol_port | 9091 |
| provisioning_status | ACTIVE |
| sni_container_refs | [] |
| timeout_client_data | 50000 |
| timeout_member_connect | 5000 |
| timeout_member_data | 50000 |
| timeout_tcp_inspect | 0 |
| updated_at | 2020-04-15T10:24:37 |
| client_ca_tls_container_ref | None |
| client_authentication | NONE |
| client_crl_container_ref | None |
| allowed_cidrs | None |
+-----------------------------+----------------------------------------------+

Changed in kuryr-kubernetes:
assignee: nobody → Maysa de Macedo Souza (maysa)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kuryr-kubernetes (master)

Reviewed: https://review.opendev.org/720817
Committed: https://git.openstack.org/cgit/openstack/kuryr-kubernetes/commit/?id=20f7d24ed1c6b909a1996a6be47a2cb721600570
Submitter: Zuul
Branch: master

commit 20f7d24ed1c6b909a1996a6be47a2cb721600570
Author: Maysa Macedo <email address hidden>
Date: Fri Apr 17 18:55:04 2020 +0000

    Ensure LB state annotation sg matches the SG on the LB

    As soon as the service is created it's possible that the backend pods
    are not yet created resulting in an lbaas_spec annotation with no
    security groups defined, and so security group rules can turn out
    to be removed from the load balancer sg. This commit ensures the
    lbaas_state annotation contains the updated sgs.

    Closes-bug: 1872962
    Change-Id: I296d16a627e39e6534ad9c1dff18b4564624d35d

Changed in kuryr-kubernetes:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.