Auto tagging fails using private certs

Bug #2067944 reported by Andrew
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
New
Undecided
Unassigned

Bug Description

I added an automatic tag via the web UI with this XPath:

//node[@class="system"]/vendor="Supermicro"

It did nothing. Looking in /var/snap/maas/common/log/rackd.log, I see a traceback reporting a CERTIFICATE_VERIFY_FAILED problem. I'm guessing that might be because we set up MAAS with a private certificate from our internal CA:

$ sudo /var/lib/snapd/snap/bin/maas config-tls enable --yes /var/snap/maas/common/certs/maas.key /var/snap/maas/common/certs/maas.crt

So far everything else seems to be working with these certs - the web interface is working and is reported as secure on internal browsers, and the maas command line worked after this bugfix:

https://bugs.launchpad.net/maas/+bug/2067503

System details:

Rocky Linux release 9.4
snapd-2.62-0.el9.x86_64
maas 3.4.2-14353-g.5a5221d57 35359 3.4/stable canonical✓ -

Full traceback from rackd.log:

2024-06-03 15:26:39 provisioningserver.rpc.common: [critical] Unhandled failure dispatching AMP command. This is probably a bug. Please ensure that this error is handled within application code or declared in the signature of the b'EvaluateTag' command. [jft-maas:pid=2016:cmd=EvaluateTag:ask=f]
        Traceback (most recent call last):
          File "/snap/maas/35359/usr/lib/python3/dist-packages/twisted/internet/asyncioreactor.py", line 271, in _onTimer
            self.runUntilCurrent()
          File "/snap/maas/35359/usr/lib/python3/dist-packages/twisted/internet/base.py", line 991, in runUntilCurrent
            call.func(*call.args, **call.kw)
          File "/snap/maas/35359/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 700, in errback
            self._startRunCallbacks(fail)
          File "/snap/maas/35359/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 763, in _startRunCallbacks
            self._runCallbacks()
        --- <exception caught here> ---
          File "/snap/maas/35359/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 857, in _runCallbacks
            current.result = callback( # type: ignore[misc]
          File "/snap/maas/35359/usr/lib/python3/dist-packages/twisted/protocols/amp.py", line 1138, in checkKnownErrors
            key = error.trap(*command.allErrors)
          File "/snap/maas/35359/usr/lib/python3/dist-packages/twisted/python/failure.py", line 451, in trap
            self.raiseException()
          File "/snap/maas/35359/usr/lib/python3/dist-packages/twisted/python/failure.py", line 475, in raiseException
            raise self.value.with_traceback(self.tb)
          File "/snap/maas/35359/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 244, in inContext
            result = inContext.theWork() # type: ignore[attr-defined]
          File "/snap/maas/35359/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 260, in <lambda>
            inContext.theWork = lambda: context.call( # type: ignore[attr-defined]
          File "/snap/maas/35359/usr/lib/python3/dist-packages/twisted/python/context.py", line 117, in callWithContext
            return self.currentContext().callWithContext(ctx, func, *args, **kw)
          File "/snap/maas/35359/usr/lib/python3/dist-packages/twisted/python/context.py", line 82, in callWithContext
            return func(*args, **kw)
          File "/snap/maas/35359/lib/python3.10/site-packages/provisioningserver/utils/twisted.py", line 203, in wrapper
            result = func(*args, **kwargs)
          File "/snap/maas/35359/lib/python3.10/site-packages/provisioningserver/rpc/tags.py", line 39, in evaluate_tag
            process_node_tags(
          File "/snap/maas/35359/lib/python3.10/site-packages/provisioningserver/tags.py", line 339, in process_node_tags
            process_all(
          File "/snap/maas/35359/lib/python3.10/site-packages/provisioningserver/tags.py", line 303, in process_all
            nodes_matched, nodes_unmatched = classify(
          File "/snap/maas/35359/lib/python3.10/site-packages/provisioningserver/utils/__init__.py", line 75, in classify
            for ident, subject in subjects:
          File "/snap/maas/35359/lib/python3.10/site-packages/provisioningserver/tags.py", line 279, in gen_node_details
            for system_id, details in get_details(batch).items():
          File "/snap/maas/35359/lib/python3.10/site-packages/provisioningserver/tags.py", line 78, in get_details_for_nodes
            data = process_response(client.get(path, op="details"))
          File "/snap/maas/35359/lib/python3.10/site-packages/apiclient/maas_client.py", line 270, in get
            return self.dispatcher.dispatch_query(
          File "/snap/maas/35359/lib/python3.10/site-packages/apiclient/maas_client.py", line 122, in dispatch_query
            res = opener.open(req)
          File "/usr/lib/python3.10/urllib/request.py", line 525, in open
            response = meth(req, response)
          File "/usr/lib/python3.10/urllib/request.py", line 634, in http_response
            response = self.parent.error(
          File "/usr/lib/python3.10/urllib/request.py", line 557, in error
            result = self._call_chain(*args)
          File "/usr/lib/python3.10/urllib/request.py", line 496, in _call_chain
            result = func(*args)
          File "/usr/lib/python3.10/urllib/request.py", line 749, in http_error_302
            return self.parent.open(new, timeout=req.timeout)
          File "/usr/lib/python3.10/urllib/request.py", line 519, in open
            response = self._open(req, data)
          File "/usr/lib/python3.10/urllib/request.py", line 536, in _open
            result = self._call_chain(self.handle_open, protocol, protocol +
          File "/usr/lib/python3.10/urllib/request.py", line 496, in _call_chain
            result = func(*args)
          File "/usr/lib/python3.10/urllib/request.py", line 1391, in https_open
            return self.do_open(http.client.HTTPSConnection, req,
          File "/usr/lib/python3.10/urllib/request.py", line 1351, in do_open
            raise URLError(err)
        urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)>

Revision history for this message
Anton Troyanov (troyanov) wrote :

Hi Andrew,

Unfortunately that is a known bug [0] that is fixed in the upcoming 3.5 release.
Meanwhile I think it should be possible to add CA (of the self-signed cert) to the trusted root to bypass this error.

[0]: https://bugs.launchpad.net/maas/+bug/2054808

Revision history for this message
Andrew (andrew-boatrocker) wrote :

Good to know that it's fixed, thanks.

I tried adding the CA of the private cert with:

$ sudo /var/lib/snapd/snap/bin/maas config-tls enable --yes --cacert /var/snap/maas/common/certs/jft.pem /var/snap/maas/common/certs/maas.key /var/snap/maas/common/certs/maas.crt

...but got the same error in the logs, even after restarting the snap. Maybe we'll try out the dev version if we get a chance, since we're still in testing.

Revision history for this message
Andrew (andrew-boatrocker) wrote :

Or maybe it's working now after all. So that's good.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.