tripleo-ci-centos-9-scenario001-standalone failed during step5 because gnocchi couldn't connect to redis
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
tripleo |
Fix Released
|
Critical
|
Unassigned |
Bug Description
2022-06-16 17:47:41.106170 | fa163e58-
2022-06-16 17:42:12,816 [8] CRITICAL root: Traceback (most recent call last):
File "/usr/lib/
self._num_sacks = int(self.
File "/usr/lib/
return self._client.
File "/usr/lib/
return self.execute_
File "/usr/lib/
conn = self.connection or pool.get_
File "/usr/lib/
connection.
File "/usr/lib/
self.
File "/usr/lib/
auth_response = self.read_
File "/usr/lib/
response = self._parser.
File "/usr/lib/
raw = self._buffer.
File "/usr/lib/
self.
File "/usr/lib/
raise ConnectionError
redis.exception
This container is still passed credentials for Ceph though I'm not sure that's related. If it needs the credentails, nothing about how they are configured changes (our client code to do this still runs during overcloud deployemnt, but we moved server deployment earlier). If there was something wrong with the shared ceph.conf and shared ceph.client.
If it's Ceph related I can help but I'd like to get input from someone who works on gnocchi.
[1]
tags: | added: alert promotion-blocker |
Changed in tripleo: | |
importance: | High → Critical |
Changed in tripleo: | |
status: | Triaged → Fix Released |
There seems to be an issue with redis ant it is never promoted.
https:/ /411deec051d768 e42c33- 554a55c2926ce34 5d8a8f0805ecfe9 93.ssl. cf5.rackcdn. com/846159/ 4/check/ tripleo- ci-centos- 9-scenario001- standalone/ a9a4889/ logs/undercloud /var/log/ extra/pcs. txt 124.73: 5001/tripleomas tercentos9/ openstack- redis:pcmklates t]: redis): Unpromoted standalone
~~~
* Container bundle: redis-bundle [198.72.
* redis-bundle-0 (ocf:heartbeat:
~~~
ceilometer-manage is trying to connect redis because redis is configured as its incoming storage but can't access because of no promoted master.
https:/ /411deec051d768 e42c33- 554a55c2926ce34 5d8a8f0805ecfe9 93.ssl. cf5.rackcdn. com/846159/ 4/check/ tripleo- ci-centos- 9-scenario001- standalone/ a9a4889/ logs/undercloud /var/log/ containers/ haproxy/ haproxy. log standalone. ctlplane. localdomain is DOWN, reason: Layer4 connection problem, info: "Connection refused at step 1 of tcp-check (connect port 6379)", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. 2022:17: 37:45.108] redis redis_be/<NOSRV> -1/-1/0 0 SC 2/1/0/0/0 0/0 2022:17: 37:45.111] redis redis_be/<NOSRV> -1/-1/0 0 SC 2/1/0/0/0 0/0 2022:17: 37:45.113] redis redis_be/<NOSRV> -1/-1/0 0 SC 2/1/0/0/0 0/0 2022:17: 37:45.114] redis redis_be/<NOSRV> -1/-1/0 0 SC 2/1/0/0/0 0/0 2022:17: 37:45.117] redis redis_be/<NOSRV> -1/-1/0 0 SC 2/1/0/0/0 0/0 2022:17: 37:45.120] redis redis_be/<NOSRV> -1/-1/0 0 SC 2/1/0/0/0 0/0 2022:17: 37:45.121] redis redis_be/<NOSRV> -1/-1/0 0 SC 2/1/0/0/0 0/0 2022:17: 37:45.122] redis redis_be/<NOSRV> -1/-1/0 0 SC 2/1/0/0/0 0/0 2022:17: 37:45.123] redis redis_be/<NOSRV> -1/-1/0 0 SC 2/1/0/0/0 0/0 2022:17: 37:45.124] redis redis_be/<NOSRV> -1/-1/0 0 SC 2/1/0/0/0 0/0 2022:17: 37:45.124] redis redis_be/<NOSRV> -1/-1/0 0 SC 2/1/0/0/0 0/0 2022:17: 37:45.125] redis redis_be/<NOSRV> -1/-1/0 0 SC 2/1/0/0/0 0/0 2022:17: 37:45.129] redis redis_be/<NOSRV> -1/-1/0 0 SC 2/1/0/0/0 0/0 2022:17: 37:45.130] redis redis_be/<NOSRV> -1/-1/0 0 SC 2/1/0/0/0 0/0 2022:17: 37:45.135] redis redis_be/<NOSRV> -1/-1/0 0 SC 2/1/0/0...
~~~
Jun 16 17:24:28 standalone haproxy[7]: Server redis_be/
Jun 16 17:24:28 standalone haproxy[7]: backend redis_be has no server available!
...
Jun 16 17:37:45 standalone haproxy[7]: 192.168.24.3:58548 [16/Jun/
Jun 16 17:37:45 standalone haproxy[7]: 192.168.24.3:58550 [16/Jun/
Jun 16 17:37:45 standalone haproxy[7]: 192.168.24.3:58552 [16/Jun/
Jun 16 17:37:45 standalone haproxy[7]: 192.168.24.3:58556 [16/Jun/
Jun 16 17:37:45 standalone haproxy[7]: 192.168.24.3:58558 [16/Jun/
Jun 16 17:37:45 standalone haproxy[7]: 192.168.24.3:58560 [16/Jun/
Jun 16 17:37:45 standalone haproxy[7]: 192.168.24.3:58562 [16/Jun/
Jun 16 17:37:45 standalone haproxy[7]: 192.168.24.3:58564 [16/Jun/
Jun 16 17:37:45 standalone haproxy[7]: 192.168.24.3:58566 [16/Jun/
Jun 16 17:37:45 standalone haproxy[7]: 192.168.24.3:58568 [16/Jun/
Jun 16 17:37:45 standalone haproxy[7]: 192.168.24.3:58570 [16/Jun/
Jun 16 17:37:45 standalone haproxy[7]: 192.168.24.3:58572 [16/Jun/
Jun 16 17:37:45 standalone haproxy[7]: 192.168.24.3:58574 [16/Jun/
Jun 16 17:37:45 standalone haproxy[7]: 192.168.24.3:58576 [16/Jun/
Jun 16 17:37:45 standalone haproxy[7]: 192.168.24.3:58578 [16/Jun/