Brief Description
-----------------
Attempted to create a windows glance image, it failed after ~15 minutes, and after that no pods are running, following error returns from any kubectl cmd:
The connection to the server 192.168.205.2:6443 was refused - did you specify the right host or port
Severity
--------
Major
Steps to Reproduce
------------------
1. copy a windows qcow2 image to active controller
2. Attempt to create a glance image using above windows qcow2 image file
glance --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne image-create --property os_type=windows --container-format bare --file /home/wrsroot/images/win2016.qcow2 --name win_2016 --visibility public --disk-format qcow2
Expected Behavior
------------------
- glance image created successfully
- system is functional after that
Actual Behavior
----------------
- glance image creation failed after 17 minutes with following error
[2019-03-11 20:42:37,816] 387 DEBUG MainThread ssh.expect :: Output:
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | None |
| container_format | bare |
| created_at | 2019-03-11T20:25:18Z |
| disk_format | qcow2 |
| id | 93c65e41-08a7-48c5-b95d-024a3bc0448e |
| min_disk | 0 |
| min_ram | 0 |
| name | win_2016 |
| os_hash_algo | None |
| os_hash_value | None |
| os_hidden | False |
| os_type | windows |
| owner | 9c3337f920d44cb5af0c157af3eb5909 |
| protected | False |
| size | None |
| status | queued |
| tags | [] |
| updated_at | 2019-03-11T20:25:18Z |
| virtual_size | Not available |
| visibility | public |
+------------------+--------------------------------------+
Error finding address for http://glance-api.openstack.svc.cluster.local:9292/v2/images/93c65e41-08a7-48c5-b95d-024a3bc0448e/file: Unable to establish connection to http://glance-api.openstack.svc.cluster.local:9292/v2/images/93c65e41-08a7-48c5-b95d-024a3bc0448e/file: [Errno 110] Connection timed out
- kubectl cmd gets rejected since the failure
controller-1:~$ kubectl get pods
The connection to the server 192.168.205.2:6443 was refused - did you specify the right host or port?
- sudo docker ps returns nothing
controller-1:~$ sudo docker ps
Password:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
Reproducibility
---------------
Reproducible (Saw this issue 2/2 times on regular systems)
System Configuration
--------------------
Two node system, Multi-node system
Branch/Pull Time/Commit
-----------------------
master as of 2019-03-05
Timestamp/Logs
--------------
[2019-03-11 20:25:17,194] 262 DEBUG MainThread ssh.send :: Send 'glance --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne image-create --property os_type=windows --container-format bare --file /home/wrsroot//images/win2016.qcow2 --name win_2016 --visibility public --disk-format qcow2'
Marking as release gating; related to containers.
Assigning to Brent as this needs an architectural decision. This is likely an issue for any large images.