charm does not report landscape client certificate validation errors when its connecting to the server

[Dmitrii Shcherbakov]

Tried to continue with a workaround for https://bugs.launchpad.net/landscape-client-charm/+bug/1800667

landscape-client fails to connect to landscape server due to certificate validation errors, meanwhile landscape-client just reports "Need computer-title and juju-info to proceed" which is confusing and incorrect because both are present during hook runtime.

There are two problems here:

1) landscape-haproxy did not propagate a correct certificate to the landscape server for some reason;
2) landscape-client charm does not do proper error reporting.

More information below:

  landscape-client/2* maintenance executing 10.190.92.107 Need computer-title and juju-info to proceed

 sudo grep -P "computer.*?title|account.*?name" /etc/landscape/client.conf
account_name = standalone
computer_title = juju-8d7ebe-5

Landscape is behind haproxy which has a self-signed certificate used for TLS termination:

juju status | grep 10.190.92.180
landscape-haproxy/0* unknown idle 0 10.190.92.180 80/tcp,443/tcp
0 started 10.190.92.180 juju-8d7ebe-0 bionic Running

A certificate used on haproxy for TLS termination:

openssl s_client -connect 10.190.92.180:443 | openssl x509 -noout -text
https://paste.ubuntu.com/p/7xcq8vhwcb/

                Modulus:
                    00:d4:26:a8:74:f9:aa:a9:6e:bb:59:17:60:2b:4f:

A certificate for landscape server on a client machine which has different modulus values:

openssl x509 -in /etc/ssl/certs/landscape_server_ca.crt -text -noout
https://paste.ubuntu.com/p/H7xdmWVqTt/

                Modulus:
                    00:b2:98:eb:75:65:92:8f:0b:af:f8:6a:b4:9a:3d:

A certificate
relation-get ssl-public-key | grep -oP "^base64:\K.*" | base64 -d | openssl x509 -text -noout
https://paste.ubuntu.com/p/6mXMVK2JV3/

                Modulus:
                    00:b2:98:eb:75:65:92:8f:0b:af:f8:6a:b4:9a:3d:

unit-landscape-server-0: 19:11:18 INFO unit.landscape-server/0.juju-log registration:11: No SSL configuration keys found, asking HAproxy to use the 'DEFAULT' certificate.

juju run --unit landscape-server/0 'relation-get -r website:5 ssl_cert landscape-haproxy/0' | base64 -d | openssl x509 -text -noout

https://paste.ubuntu.com/p/F9n5bzMjCc/
                Modulus:
                    00:b2:98:eb:75:65:92:8f:0b:af:f8:6a:b4:9a:3d:

landscape-haproxy/0: /etc/haproxy/haproxy.cfg

frontend landscape-haproxy-0-443
    bind 0.0.0.0:443 ssl crt /var/lib/haproxy/default.pem no-sslv3

less /var/lib/haproxy/default.pem no-sslv3
-----BEGIN CERTIFICATE-----
MIIBxzCCATCgAwIBAgIJAK9vTpiDoy+/MA0GCSqGSIb3DQEBCwUAMBgxFjAUBgNV
BAMMDTEwLjE5MC45Mi4xODAwHhcNMTgxMDMwMTMwNDI2WhcNMjgxMDI3MTMwNDI2
WjAYMRYwFAYDVQQDDA0xMC4xOTAuOTIuMTgwMIGfMA0GCSqGSIb3DQEBAQUAA4GN
ADCBiQKBgQDUJqh0+aqpbrtZF2ArTzk0/VFGBzzYU4F0z+ld7rBAH4ZHmevgrtM0
H8Q2FI8tEjEqqJeVnhKmO3yoyvpRRb2urd4GF+Yu/PovO9odlZhW+3AoJ03SCzJg
LjlAFGPhMgjoLABP4sIycdwytKgBRjgq34IUl481oLQcZlnVTQO/KwIDAQABoxkw
FzAVBgNVHREEDjAMhwQKvly0hwQKvly0MA0GCSqGSIb3DQEBCwUAA4GBADkxWtM/
5SFdib9HAKxFd/FM82deonmgxiwXTHVEzlbNgEG7knBFsiVP8Z61o0M7dTffu1GE
I8El6Ow8kuM9o9LuU/Aa30juXzGZnpJH2rJHRXfWCG+P43EWIMxqnQ+y5USsyfTz
SHC044hWkXNRetZD/c83d+2y9udxeNaNeiYa
-----END CERTIFICATE-----
<redacted-key-goes-here>

The above cert decoded:

cat | openssl x509 -text -noout
https://paste.ubuntu.com/p/pp77SByJKF/

                Modulus:
                    00:d4:26:a8:74:f9:aa:a9:6e:bb:59:17:60:2b:4f:

bundle:
https://paste.ubuntu.com/p/V4qjZSfRwT/

/var/log/landscape/broker.log:

2018-10-30 16:15:06,224 INFO [MainThread] Broker started with config /etc/landscape/client.conf
2018-10-30 16:15:06,333 INFO [MainThread] Starting urgent message exchange with https://10.190.92.180/message-system.
2018-10-30 16:15:06,367 ERROR [PoolThread-twisted.internet.reactor-0] Error contacting the server at https://10.190.92.180/message-system.
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/landscape/lib/fetch.py", line 116, in fetch
    curl.perform()
pycurl.error: (60, 'server certificate verification failed. CAfile: /etc/ssl/certs/landscape_server_ca.crt CRLfile: none')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/landscape/client/broker/transport.py", line 82, in exchange
    message_api)
  File "/usr/lib/python3/dist-packages/landscape/client/broker/transport.py", line 56, in _curl
    headers=headers, cainfo=self._pubkey, curl=curl))
  File "/usr/lib/python3/dist-packages/landscape/lib/fetch.py", line 118, in fetch
    raise PyCurlError(e.args[0], e.args[1])
landscape.lib.fetch.PyCurlError: Error 60: server certificate verification failed. CAfile: /etc/ssl/certs/landscape_server_ca.crt CRLfile: none
2018-10-30 16:15:06,368 ERROR [MainThread] Message exchange failed: server certificate verification failed. CAfile: /etc/ssl/certs/landscape_server_ca.crt CRLfile: none
2018-10-30 16:15:06,368 INFO [MainThread] Message exchange failed.
2018-10-30 16:15:06,368 INFO [MainThread] Message exchange completed in 0.03s.
2018-10-30 16:16:06,369 INFO [MainThread] Starting urgent message exchange with https://10.190.92.180/message-system.
2018-10-30 16:16:06,436 ERROR [PoolThread-twisted.internet.reactor-0] Error contacting the server at https://10.190.92.180/message-system.
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/landscape/lib/fetch.py", line 116, in fetch
    curl.perform()
pycurl.error: (60, 'server certificate verification failed. CAfile: /etc/ssl/certs/landscape_server_ca.crt CRLfile: none')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/landscape/client/broker/transport.py", line 82, in exchange
    message_api)
  File "/usr/lib/python3/dist-packages/landscape/client/broker/transport.py", line 56, in _curl
    headers=headers, cainfo=self._pubkey, curl=curl))
  File "/usr/lib/python3/dist-packages/landscape/lib/fetch.py", line 118, in fetch
    raise PyCurlError(e.args[0], e.args[1])
landscape.lib.fetch.PyCurlError: Error 60: server certificate verification failed. CAfile: /etc/ssl/certs/landscape_server_ca.crt CRLfile: none

2018-10-30 16:16:06,436 ERROR [MainThread] Message exchange failed: server certificate verification failed. CAfile: /etc/ssl/certs/landscape_server_ca.crt CRLfile: none

2018-10-30 16:16:06,437 INFO [MainThread] Message exchange failed.
2018-10-30 16:16:06,437 INFO [MainThread] Message exchange completed in 0.07s.
2018-10-30 16:16:07,434 INFO [MainThread] Broker stopped with config /etc/landscape/client.conf

[Dmitrii Shcherbakov (dmitriis)]

The unit can even go into this state while actually failing on TLS certificate verification:

ubuntu/1* active idle 5 10.190.92.107 ready
  landscape-client/3* active idle 10.190.92.107 System successfully registered

[Alexander Litvinov (alitvinov)]

Proposed patch fixes
2) landscape-client charm does not do proper error reporting.

With mentioned ssl issue the client goes to "Client registration fails" status and line about SSL issue is present in the unit logs.

INFO juju-log Registering landscape-client/42
DEBUG config-changed Stopping landscape-client (via systemctl): landscape-client.service.
INFO juju-log Registration result is ssl-error
INFO juju-log Registration exit_code = 1
INFO juju-log Client config updated, system tried registration but failed

[David O Neill (dmzoneill)]

may help someone resolve landscape-clinet stuck in maintenance mode

JUJU_MODEL=k8s-controller:lma
HAPROXY=landscape-haproxy/0

# Note landscape does not have HA Proxy VIP as in baremetal HA setup.
IP=$( juju status -m $JUJU_MODEL $HAPROXY --format json | jq -r '.machines[]."dns-name"' )

Juju switch lma

juju ssh $HAPROXY "sudo openssl x509 -in /var/lib/haproxy/default.pem > /tmp/landscape.crt; sudo chmod ugo+r /tmp/landscape.crt"
juju scp $HAPROXY:/tmp/landscape.crt /tmp

# Run below for lma and k8s-tenant-1 models:

juju config -m <JUJU_MODEL> landscape-client \
        url="https:/$IP/message-system" \
        ping-url="https://$IP/ping" \
        ssl-public-key="base64:$(cat /tmp/landscape.crt | openssl base64 -e)"