openstack image create --disk-format raw --container-format bare --tag amphora-image --file /tmp/ansible.JI3rrX/amphora.img fails w/ read timeout

[wes hayutin]

train only atm
http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-scenario010-standalone

"openstack image create --disk-format raw --container-format bare --tag amphora-image --file /tmp/ansible.JI3rrX/amphora.img --property hw_architecture=x86_64 --private amphora\\n\", \"delta\": \"0:20:04.453612\", \"end\": \"2020-04-15 16:52:37.137664\", \"msg\": \"non-zero return code\", \"rc\": 1, \"start\": \"2020-04-15 16:32:32.684052\", \"stderr\": \"Error communicating with http://192.168.24.1:9292/v2/images/d8313837-c7a9-4586-8d29-46f7c007ecd5: HTTPConnectionPool(host='192.168.24.1', port=9292): Read timed out. (read timeout=600.0)\", \"stderr_lines\": [\"Error communicating with http://192.168.24.1:9292/v2/images/d8313837-c7a9-4586-8d29-46f7c007ecd5: HTTPConnectionPool(host='192.168.24.1', port=9292): Read timed out. (read timeout=600.0)\"], \"stdout\": \"\", \"stdout_lines\": []}", "",

http://zuul.openstack.org/build/a2228b700521455a8f4e2c9642553d27
http://zuul.openstack.org/build/7d01fc38decc4eb084705624dd77fdff

[wes hayutin (weshayutin)]

hrm.. not much in glance
https://9d2f1bd4e91979fa479a-218ec341821d13471e04f43a25678299.ssl.cf2.rackcdn.com/719368/1/gate/tripleo-ci-centos-7-scenario010-standalone/a2228b7/logs/undercloud/var/log/containers/glance/api.log

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/720287

[OpenStack Infra (hudson-openstack)]

Change abandoned by Emilien Macchi (<email address hidden>) on branch: master
Review: https://review.opendev.org/720287

[Giulio Fidente (gfidente)]

it seems an issue with the ceph cluster refusing writes as it has 96 pgs undersized

I am not sure why that is though because we do set CephPoolDefaultSize to 1 in the job, like we do in scenarios001 and 004 which pass; still investigating

[Giulio Fidente (gfidente)]

we had a problem translating the THT parameters into ceph-ansible group_vars so despite the THT parameter being set, the ceph-ansible group_vars of failing jobs didn't have default_pool_size set [1] which caused the PGs to be undersized and refuse writes -- ultimately making glance timeout

the recent changes do set defaul_pool_size correctly so issue should be resolved [2]

1. https://9d2f1bd4e91979fa479a-218ec341821d13471e04f43a25678299.ssl.cf2.rackcdn.com/719368/1/gate/tripleo-ci-centos-7-scenario010-standalone/a2228b7/logs/undercloud/home/zuul/standalone-ansible-rPlNxf/ceph-ansible/group_vars/all.yml
2. https://55ff20dabee44387be7e-010af429a9a767f80144a0b88859a166.ssl.cf5.rackcdn.com/720052/1/check/tripleo-ci-centos-7-scenario010-standalone/b27b015/logs/undercloud/home/zuul/standalone-ansible-bIGLjC/ceph-ansible/group_vars/all.yml