raily failure: ERROR rally.task.engine OpenSSL.SSL.Error: [('SSL routines', 'ssl3_get_record', 'wrong version number')]

Bug #1912505 reported by Jason Hobbs on 2021-01-20
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Designate Charm
Critical
Unassigned
charms.openstack
Critical
Unassigned

Bug Description

This is recently new behavior. During rally setup, we get a failure to talk to designate:

2021-01-19 14:48:40.571 61487 ERROR rally.task.engine OpenSSL.SSL.Error: [('SSL routines', 'ssl3_get_record', 'wrong version number')]

Designate was previously working during openstack setup and other tests.

You can see the whole output here:
https://oil-jenkins.canonical.com/artifacts/025a9c0a-fc51-4667-aad4-0cb90ffcf3e4/fce_build_9341/fce_build_9341_console.out

https://solutions.qa.canonical.com/testruns/testRun/025a9c0a-fc51-4667-aad4-0cb90ffcf3e4

description: updated
Revision history for this message
Marian Gasparovic (marosg) wrote :

it hit us 4 times already, setting release blocker tag

tags: added: cdo-release-blocker
Changed in charm-designate:
status: New → Triaged
importance: Undecided → Critical
assignee: nobody → Alex Kavanagh (ajkavanagh)
Changed in charms.openstack:
status: New → Triaged
importance: Undecided → Critical
assignee: nobody → Alex Kavanagh (ajkavanagh)
Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :
Download full text (4.7 KiB)

This is/was a fascinating problem which resulted in me instrumenting the PeerHARelationAdapter() class in charms.openstack to find out what was going on.

The reason for the problem (in this case) is that the haproxy.cfg is being written with the wrong addresses for the backends. From designate/0 /etc/haproxy/haproxy.cfg:

backend designate-api_admin_10.244.8.143
    balance leastconn
    server designate-0 10.244.8.143:8991 check
    server designate-1 10.246.64.255:8991 check
    server designate-2 10.246.64.251:8991 check

Notice that the addresses for designate-1 and designate-2 are 10.246.64.x and not in 10.244.8.x. This means that 2/3 calls to the designate-api will fail as those addresses are not being listened to on the apache2 servers on those hosts.

The ip addresses for designate/1 on the cluster peer relation are (from designate 0):

root@juju-226d47-0-lxd-3:/var/lib/juju/agents/unit-designate-0/charm# relation-get -r cluster:6 - designate/1
admin-address: 192.168.33.190
egress-subnets: 10.246.64.255/32
ingress-address: 10.246.64.255
internal-address: 192.168.33.190
private-address: 10.246.64.255
public-address: 10.244.8.150
rndc-address: 192.168.33.190

i.e. the charm is using 'private-address' rather than 'public-address' for designate/1 in the haproxy.cfg

So that's what's happening; now to show why:

---

The haproxy.cfg is written from the context supplied (in the designate charm) by the PeerHARelationAdapter() adapter class. With instrumented code we see:

> PeerHARelationAdapter __init__
> add_network_split_addresses
> local_network_split_addresses
 < local_network_split_addresses: cluster hosts: OrderedDict([('10.244.8.143', {'network': '10.244.8.143/255.255.255.0', 'backends': OrderedDict([('designate-0', '10.244.8.143')])}
), ('192.168.33.183', {'network': '192.168.33.183/255.255.255.128', 'backends': OrderedDict([('designate-0', '192.168.33.183')])})])
> local_network_split_addresses
 < local_network_split_addresses: cluster hosts: OrderedDict([('10.244.8.143', {'network': '10.244.8.143/255.255.255.0', 'backends': OrderedDict([('designate-0', '10.244.8.143')])}
), ('192.168.33.183', {'network': '192.168.33.183/255.255.255.128', 'backends': OrderedDict([('designate-0', '192.168.33.183')])})])
> local_network_split_addresses
 < local_network_split_addresses: cluster hosts: OrderedDict([('10.244.8.143', {'network': '10.244.8.143/255.255.255.0', 'backends': OrderedDict([('designate-0', '10.244.8.143')])}
), ('192.168.33.183', {'network': '192.168.33.183/255.255.255.128', 'backends': OrderedDict([('designate-0', '192.168.33.183')])})])
 < add_network_split_addresses: OrderedDict([('10.244.8.143', {'network': '10.244.8.143/255.255.255.0', 'backends': OrderedDict([('designate-0', '10.244.8.143'), ('designate-1', '1
0.244.8.150'), ('designate-2', '10.244.8.146')])}), ('192.168.33.183', {'network': '192.168.33.183/255.255.255.128', 'backends': OrderedDict([('designate-0', '192.168.33.183'), ('de
signate-1', '192.168.33.190'), ('designate-2', '192.168.33.186')])})])
> add_default_addesses
> local_default_addresses
 < local_default_addresses: returns: {'10.244.8.143': {'network': '10.244.8.143/255.255.255.0', 'backends': O...

Read more...

Changed in charm-designate:
milestone: none → 21.01
Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :
Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :
Changed in charms.openstack:
status: Triaged → Fix Committed
Changed in charm-designate:
status: Triaged → In Progress
Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :

Review for designate has been merged so setting to fix committed.

Changed in charm-designate:
status: In Progress → Fix Committed
assignee: Alex Kavanagh (ajkavanagh) → nobody
Changed in charms.openstack:
assignee: Alex Kavanagh (ajkavanagh) → nobody
David Ames (thedac) on 2021-02-10
Changed in charm-designate:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers