raily failure: ERROR rally.task.engine OpenSSL.SSL.Error: [('SSL routines', 'ssl3_get_record', 'wrong version number')]

[Jason Hobbs]

This is recently new behavior. During rally setup, we get a failure to talk to designate:

2021-01-19 14:48:40.571 61487 ERROR rally.task.engine OpenSSL.SSL.Error: [('SSL routines', 'ssl3_get_record', 'wrong version number')]

Designate was previously working during openstack setup and other tests.

You can see the whole output here:
https://oil-jenkins.canonical.com/artifacts/025a9c0a-fc51-4667-aad4-0cb90ffcf3e4/fce_build_9341/fce_build_9341_console.out

https://solutions.qa.canonical.com/testruns/testRun/025a9c0a-fc51-4667-aad4-0cb90ffcf3e4

[Marian Gasparovic (marosg)]

it hit us 4 times already, setting release blocker tag

[Alex Kavanagh (ajkavanagh)]

This is/was a fascinating problem which resulted in me instrumenting the PeerHARelationAdapter() class in charms.openstack to find out what was going on.

The reason for the problem (in this case) is that the haproxy.cfg is being written with the wrong addresses for the backends. From designate/0 /etc/haproxy/haproxy.cfg:

backend designate-api_admin_10.244.8.143
    balance leastconn
    server designate-0 10.244.8.143:8991 check
    server designate-1 10.246.64.255:8991 check
    server designate-2 10.246.64.251:8991 check

Notice that the addresses for designate-1 and designate-2 are 10.246.64.x and not in 10.244.8.x. This means that 2/3 calls to the designate-api will fail as those addresses are not being listened to on the apache2 servers on those hosts.

The ip addresses for designate/1 on the cluster peer relation are (from designate 0):

root@juju-226d47-0-lxd-3:/var/lib/juju/agents/unit-designate-0/charm# relation-get -r cluster:6 - designate/1
admin-address: 192.168.33.190
egress-subnets: 10.246.64.255/32
ingress-address: 10.246.64.255
internal-address: 192.168.33.190
private-address: 10.246.64.255
public-address: 10.244.8.150
rndc-address: 192.168.33.190

i.e. the charm is using 'private-address' rather than 'public-address' for designate/1 in the haproxy.cfg

So that's what's happening; now to show why:

---

The haproxy.cfg is written from the context supplied (in the designate charm) by the PeerHARelationAdapter() adapter class. With instrumented code we see:

> PeerHARelationAdapter __init__
> add_network_split_addresses
> local_network_split_addresses
 < local_network_split_addresses: cluster hosts: OrderedDict([('10.244.8.143', {'network': '10.244.8.143/255.255.255.0', 'backends': OrderedDict([('designate-0', '10.244.8.143')])}
), ('192.168.33.183', {'network': '192.168.33.183/255.255.255.128', 'backends': OrderedDict([('designate-0', '192.168.33.183')])})])
> local_network_split_addresses
 < local_network_split_addresses: cluster hosts: OrderedDict([('10.244.8.143', {'network': '10.244.8.143/255.255.255.0', 'backends': OrderedDict([('designate-0', '10.244.8.143')])}
), ('192.168.33.183', {'network': '192.168.33.183/255.255.255.128', 'backends': OrderedDict([('designate-0', '192.168.33.183')])})])
> local_network_split_addresses
 < local_network_split_addresses: cluster hosts: OrderedDict([('10.244.8.143', {'network': '10.244.8.143/255.255.255.0', 'backends': OrderedDict([('designate-0', '10.244.8.143')])}
), ('192.168.33.183', {'network': '192.168.33.183/255.255.255.128', 'backends': OrderedDict([('designate-0', '192.168.33.183')])})])
 < add_network_split_addresses: OrderedDict([('10.244.8.143', {'network': '10.244.8.143/255.255.255.0', 'backends': OrderedDict([('designate-0', '10.244.8.143'), ('designate-1', '1
0.244.8.150'), ('designate-2', '10.244.8.146')])}), ('192.168.33.183', {'network': '192.168.33.183/255.255.255.128', 'backends': OrderedDict([('designate-0', '192.168.33.183'), ('de
signate-1', '192.168.33.190'), ('designate-2', '192.168.33.186')])})])
> add_default_addesses
> local_default_addresses
 < local_default_addresses: returns: {'10.244.8.143': {'network': '10.244.8.143/255.255.255.0', 'backends': O...

[Alex Kavanagh (ajkavanagh)]

Review https://review.opendev.org/c/openstack/charms.openstack/+/772208 addresses the problem.

[Alex Kavanagh (ajkavanagh)]

Review https://review.opendev.org/c/openstack/charm-designate/+/772308 resolves the designate issue

[Alex Kavanagh (ajkavanagh)]

Review for designate has been merged so setting to fix committed.