Comment 8 for bug 1690812

Revision history for this message
Nir Magnezi (nmagnezi) wrote :

Looks like the fix for bug 1681623 did not resolve this bug.

Using loadbalancer_topology = ACTIVE_STANDBY and tested with the following pool configuration:
neutron lbaas-pool-create --lb-algorithm ROUND_ROBIN --listener listener1 --protocol HTTP --session-persistence type=SOURCE_IP
Created a new pool:
+---------------------+------------------------------------------------+
| Field | Value |
+---------------------+------------------------------------------------+
| admin_state_up | True |
| description | |
| healthmonitor_id | |
| id | b084ed49-038b-45dc-9b4b-8a277f60ba5b |
| lb_algorithm | ROUND_ROBIN |
| listeners | {"id": "8ac3b6b3-680e-4a58-b51d-883283a3caf1"} |
| loadbalancers | {"id": "d9800fd6-f010-4540-8d41-ac24ae325cc2"} |
| members | |
| name | |
| protocol | HTTP |
| session_persistence | {"cookie_name": null, "type": "SOURCE_IP"} |
| tenant_id | d67bee545d534850aedfbe77da709c68 |
+---------------------+------------------------------------------------+

The Octavia Worker shows the following exception:
https://paste.fedoraproject.org/paste/vpzv0KnOjaqYdbtf5K9Bng

Digging inside the amphora VMs, I noticed the following

One Amphora managed to spawn haproxy with no errors, looks like the standby amphora.

haproxy.cfg:
============
root@amphora-e643829b-4de3-4ebf-a96c-0bef10389f6f:~# cat /var/lib/octavia/8ac3b6b3-680e-4a58-b51d-883283a3caf1/haproxy.cfg
# Configuration for lb_nir
global
    daemon
    user nobody
    log /dev/log local0
    log /dev/log local1 notice
    stats socket /var/lib/octavia/8ac3b6b3-680e-4a58-b51d-883283a3caf1.sock mode 0666 level user

defaults
    log global
    retries 3
    option redispatch
    timeout connect 5000
    timeout client 50000
    timeout server 50000

peers 8ac3b6b3680e4a58b51d883283a3caf1_peers
    peer yZB0PtEhlFqOwQHLLY3Zj9U_QAg 10.0.0.6:1025
    peer C2BTWaiZ2FW-oF4uOy7c2LeC0mU 10.0.0.14:1025

frontend 8ac3b6b3-680e-4a58-b51d-883283a3caf1
    option httplog
    bind 10.0.0.9:80
    mode http

haproxy log:
============
cat /var/log/haproxy.log
Jul 11 11:11:06 amphora-e643829b-4de3-4ebf-a96c-0bef10389f6f haproxy[1723]: Proxy 8ac3b6b3-680e-4a58-b51d-883283a3caf1 started.
Jul 11 11:11:06 amphora-e643829b-4de3-4ebf-a96c-0bef10389f6f haproxy[1723]: Proxy 8ac3b6b3-680e-4a58-b51d-883283a3caf1 started.

The second amphora fails to spawn haproxy, looks like the active amphora:

haproxy.cfg:
============
root@amphora-6ca8b5ae-3185-48b0-9615-9bf8d5df30f0:~# cat /var/lib/octavia/8ac3b6b3-680e-4a58-b51d-883283a3caf1/haproxy.cfg
# Configuration for lb_nir
global
    daemon
    user nobody
    log /dev/log local0
    log /dev/log local1 notice
    stats socket /var/lib/octavia/8ac3b6b3-680e-4a58-b51d-883283a3caf1.sock mode 0666 level user

defaults
    log global
    retries 3
    option redispatch
    timeout connect 5000
    timeout client 50000
    timeout server 50000

peers 8ac3b6b3680e4a58b51d883283a3caf1_peers
    peer yZB0PtEhlFqOwQHLLY3Zj9U_QAg 10.0.0.6:1025
    peer C2BTWaiZ2FW-oF4uOy7c2LeC0mU 10.0.0.14:1025

frontend 8ac3b6b3-680e-4a58-b51d-883283a3caf1
    option httplog
    bind 10.0.0.9:80
    mode http
    default_backend b084ed49-038b-45dc-9b4b-8a277f60ba5b

backend b084ed49-038b-45dc-9b4b-8a277f60ba5b
    mode http
    balance roundrobin
    stick-table type ip size 10k peers 8ac3b6b3680e4a58b51d883283a3caf1_peers
    stick on src

haproxy log:
============
root@amphora-6ca8b5ae-3185-48b0-9615-9bf8d5df30f0:~# cat /var/log/haproxy.log
Jul 11 11:11:05 amphora-6ca8b5ae-3185-48b0-9615-9bf8d5df30f0 haproxy[1724]: Proxy 8ac3b6b3-680e-4a58-b51d-883283a3caf1 started.
Jul 11 11:11:05 amphora-6ca8b5ae-3185-48b0-9615-9bf8d5df30f0 haproxy[1724]: Proxy 8ac3b6b3-680e-4a58-b51d-883283a3caf1 started.
Jul 11 11:30:03 amphora-6ca8b5ae-3185-48b0-9615-9bf8d5df30f0 haproxy[3079]: [ALERT] 191/113003 (3079) : Proxy 'b084ed49-038b-45dc-9b4b-8a277f60ba5b': unable to find local peer 'amphora-6ca8b5ae-3185-48b0-9615-9bf8d5df30f0' in peers section '8ac3b6b3680e4a58b51d883283a3caf1_peers'.
Jul 11 11:30:03 amphora-6ca8b5ae-3185-48b0-9615-9bf8d5df30f0 haproxy[3079]: [WARNING] 191/113003 (3079) : Removing incomplete section 'peers 8ac3b6b3680e4a58b51d883283a3caf1_peers' (no peer named 'amphora-6ca8b5ae-3185-48b0-9615-9bf8d5df30f0').
Jul 11 11:30:03 amphora-6ca8b5ae-3185-48b0-9615-9bf8d5df30f0 haproxy[3079]: [ALERT] 191/113003 (3079) : Fatal errors found in configuration.

[ALERT] 191/115358 (4580) : Proxy 'b084ed49-038b-45dc-9b4b-8a277f60ba5b': unable to find local peer 'amphora-6ca8b5ae-3185-48b0-9615-9bf8d5df30f0' in peers section '8ac3b6b3680e4a58b51d883283a3caf1_peers'.

It seems like something is not right with they way we configure peers, I just can't put my finger on what exactly is wrong with that haproxy.cfg file.
Attempts to manually invoke haproxy resulted the same error:

root@amphora-6ca8b5ae-3185-48b0-9615-9bf8d5df30f0:/etc/systemd# /usr/sbin/haproxy -f /var/lib/octavia/8ac3b6b3-680e-4a58-b51d-883283a3caf1/haproxy.cfg -f /var/lib/octavia/haproxy-default-user-group.conf -c -q
[ALERT] 191/115358 (4580) : Proxy 'b084ed49-038b-45dc-9b4b-8a277f60ba5b': unable to find local peer 'amphora-6ca8b5ae-3185-48b0-9615-9bf8d5df30f0' in peers section '8ac3b6b3680e4a58b51d883283a3caf1_peers'.
[ALERT] 191/115358 (4580) : Fatal errors found in configuration.

P.S.
====
1. Used the default Ubuntu based amphora.
2. None of this happens with loadbalancer_topology = SINGLE, this is way CI is not failing.