Deployment fails with :/usr/bin/docker-current: Error response from daemon: grpc: the connection is unavailable
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
tripleo |
Invalid
|
High
|
Unassigned |
Bug Description
The error message is highly likely a red herring pointing out to some other sort of issues, like system under pressure perhaps.
Examples:
dstat shows correlation with high CPU wait numbers (>70%) and an increased memory use (See 19h 48m 44s and further on)
See also the elastic-recheck stats for that error pattern:
total hits: 7
build_branch
85% master
14% stable/rocky
build_change
14% 591540 610728 582301
14% 611447 608354
14% 582735
14% 610087
14% 610491
build_name
14% tripleo-
14% tripleo-
14% tripleo-
14% tripleo-
14% tripleo-
build_node
100% centos-7
build_queue
85% check
14% gate
build_status
71% FAILURE
14% FAILURE SUCCESS
14% SUCCESS FAILURE
build_zuul_url
100% N/A
filename
71% logs/undercloud
28% logs/undercloud
log_url
14% http://
14% http://
14% http://
14% http://
14% http://
node_provider
57% inap-mtl01
14% inap-mtl01 rax-dfw
14% ovh-gra1 rax-iad inap-mtl01
14% rax-iad
port
14% 35486
14% 38788
14% 42428
14% 42552
14% 45124
project
28% openstack/
28% openstack/
14% openstack/
14% openstack/
14% openstack/
severity
71% INFO
28% ERROR
tags
71% logstash.txt console postci multiline _grokparsefailure
28% errors.txt console errors multiline _grokparsefailure
voting
57% 1
28% 0
14% 1 0
zuul_executor
28% ze09.openstack.org
14% ze07.openstack.org ze02.openstack.org ze01.openstack.org
14% ze10.openstack.org ze05.openstack.org
14% ze03.openstack.org
14% ze07.openstack.org
So jobs not always fail with that error. It should be CPU wait (IO) and memory pressure related instead.
Changed in tripleo: | |
importance: | Undecided → High |
milestone: | none → stein-1 |
status: | New → Triaged |
tags: | added: ci |
description: | updated |
description: | updated |
Changed in tripleo: | |
milestone: | stein-1 → stein-2 |
Changed in tripleo: | |
milestone: | stein-2 → stein-3 |
Changed in tripleo: | |
status: | Triaged → Invalid |
Follow up w/ Bogdan