MAAS 2.0 SSL verification error when adding UCSM chassis

[jeffrey leung]

We recently upgraded our MAAS environment to 2.0:

Ubuntu 14.04.4 -> 16.04.1 (hosted on a ESXi 5.5 virtual machine)
MAAS 1.9.3 -> 2.0.0

After the upgrade to MAAS v2.0, we see an SSL certificate verification error when I try to add a UCSM chassis. If I revert back to a clone of the same machine before the upgrade to MAAS 2.0, I am able to add the UCSM chassis without adding the certificate.

maas.log.1:
Sep 8 13:49:20 cspg-MAAS maas.rpc.cluster: [ERROR] Failed to probe and enlist UCS nodes: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)>

rackd.log:
2016-09-08 13:48:37 [HTTPPageGetter,client] Unhandled Error
 Traceback (most recent call last):
   File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 588, in _runCallbacks
     current.result = callback(current.result, *args, **kw)
   File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1184, in gotResult
     _inlineCallbacks(r, g, deferred)
   File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1126, in _inlineCallbacks
     result = result.throwExceptionIntoGenerator(g)
   File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
     return g.throw(self.type, self.value, self.tb)
 --- <exception caught here> ---
   File "/usr/lib/python3/dist-packages/provisioningserver/rpc/clusterservice.py", line 765, in update
     info = yield self._fetch_rpc_info(info_url)
 twisted.web.error.Error: 503 Service Unavailable

I found a similar error on launchpad with adding VMware devices (https://bugs.launchpad.net/ubuntu/+source/maas/+bug/1608639/comments/1) and tried the recommendation to download the certificate from UCSM and add it to MAAS server which allowed me to add the UCSM chassis again.

$ sudo -i
# openssl s_client -connect 192.222.80.1:443 -showcerts < /dev/null
# mkdir /usr/share/ca-certificates/custom
# nano /usr/share/ca-certificates/custom/ucsm.crt
<paste certificate>
# dpkg-reconfigure ca-certificates

Not sure if this may be an issue or if it's working as expected and the certificates will be required from here on out?

[Andres Rodriguez (andreserl)]

Hi Jeffrey,

Thanks for reporting the bug. I just want to note that the support for UCSM between MAAS 2.0 vs 1.9 has not changed, so we will need to investigate what's causing the SSL verification.

That said, however, has there been any change in the firmware of your UCSM chassis ? Such us, when youw ere using 1.9 you had version X of firmware, and after you upgraded to 2.0, you had a version Y of firmware?

Thanks!

[Andres Rodriguez (andreserl)]

Ok, I think I know what the issue is Googling I came across:

http://stackoverflow.com/questions/27835619/ssl-certificate-verify-failed-error

And this highlights:

https://www.python.org/dev/peps/pep-0476/

This seems to be something that changed in Python 3.5 and is causing issues with MAAS. So we would need to look into seeing whether we can change the way how we communicate to the chassis.

[Mike Pontillo (mpontillo)]

I can see why this could be considered a bug in MAAS, but the workaround of adding the certificate to your system as trusted is the correct thing to do. The fact that we connected and *sent a password* using "https" in MAAS 1.9 is the actual bug. It's not "https" if it doesn't validate the certificate.

It's just annoying because in many MAAS environments, customers are using "https" servers provided by device vendors, in cases where the customer doesn't actually care about security, and clicks "connect anyway" every time on a self-signed certificate or similar. (In that case, you might as well connect via http and send your password in cleartext, because someone could easily launch a man-in-the-middle attack against an unverified connection.)

[Adam Collard (adam-collard)]

This bug has not seen any activity in the last 6 months, so it is being automatically closed.

If you are still experiencing this issue, please feel free to re-open.

MAAS Team