© 2004
Canonical Ltd.
•
Terms of use
•
Data privacy
•
Contact Launchpad Support
•
Blog
•
Careers
•
System status
•
0f7b58d
(Get the code!)
Hi,
I've been going through the logs that were handed off internally[0] where I
found some unexpected failures that I believe could be affecting the behaviour
of Magnum, I will list them and explain what the could mean separately.
1. barbicanclient.
2. pymysql.
3. oslo.messaging.
4. ConnectionReset
5. keystoneauth1.
About (1), this error is present in the logs ~14k times[1], the first occurrence
on January 27th and the last one in the log is on June 19th. The absence of a
healthy Barbican service prevents Magnum from stablishing a connection to k8s
since that's the place where the secrets (e.g. private keys) are stored and read from.
Traceback (most recent call last):
File "/usr/lib/
monitor.
File "/usr/lib/
k8s_api = k8s.create_
File "/usr/lib/
return K8sAPI(context, cluster)
File "/usr/lib/
self.cert_file) = create_
File "/usr/lib/
magnum_
File "/usr/lib/
return operations.
File "/usr/lib/
return self._cert_
File "/usr/lib/
self.
File "/usr/lib/
payload = self._api.
File "/usr/lib/
return self.request(path, 'GET', *args, **kwargs).content
File "/usr/lib/
self.
File "/usr/lib/
raise exceptions.
barbicanclient.
About (2), this is a better understood error in general, it's basically that the
database dropped the client (in this case magnum-conductor process), this can be
due to numerous reasons, more data is needed to understand this, and it's out of
the scope of this bug, you most likely are seeing this issue accross all the
control plane services, if you detect this issue exclusive to Magnum service
please provide us the data you are using to get to that conclusion, so we can
look into why Magmnum may be misbehaving. This error is less common in the
logs[2], although almost all of the occurrences were in June[3]
Traceback (most recent call last):
File "/usr/lib/
self.
File "/usr/lib/
cursor.
File "/usr/lib/
result = self._query(query)
File "/usr/lib/
conn.query(q)
File "/usr/lib/
self.
File "/usr/lib/
result.read()
File "/usr/lib/
first_packet = self.connection
File "/usr/lib/
packet_header = self._read_bytes(4)
File "/usr/lib/
raise err.Operational
pymysql.
About (3), during June the rabbitmq service was having issues and services
couldn't connect to it[4][5], RabbitMQ is critical component for control plane
services (as important as MySQL).
About (4), the ConnectionReset
in the underlying network or large performance degradation where services are
not able to respond to queries on time, hence the client closed the connection
(raising a Timeout Error on the client side) and when the server tries to write
the response in the socket there is no one on the side an fails.
About (5), I don't have enough logs share insights, because we do know that
Keystone and Magnum are correctly configured, since you can launch k8s clusters,
although the main loop fails consistently, I will be looking into Magnum's
source code to understand the consequence of this scheduled task not able to
complete its job.
[0] 8508554a99e603a
[1] $ grep barbicanclient.
14147
[2] $ grep pymysql.
373
[3]$ grep pymysql.
363
[4] $ grep "Connection failed" magnum-
3099
[5] $ grep "Connection failed" magnum-
3068
[6]$ grep "ConnectionRese
305
[7] $ grep "ConnectionRese
301
[8] $ grep "keystoneauth1.
90482