nova image-list failing with SSL enabled on Juno

[Liam Young]

I have a client setup with OS_CACERT set. All endpoints registered in keystone are https. I can query neutron, glance, cinder and keystone but the second and subsequent nova image-list always fails. I can 'fix' it by restarting nova-api-os-compute and one image-list will work but then failure again. In the nova-api-os-compute.log I see:

2015-10-21 11:27:35.518 16818 WARNING keystoneclient.middleware.auth_token [req-64b4bab0-3b47-4326-b2e5-3bd457b3c797 ] Retrying on HTTP connection exception: 'HTTPSConnectionPool' object has no attribute 'insecure'
2015-10-21 11:27:36.021 16818 WARNING keystoneclient.middleware.auth_token [req-64b4bab0-3b47-4326-b2e5-3bd457b3c797 ] Retrying on HTTP connection exception: 'HTTPSConnectionPool' object has no attribute 'insecure'
2015-10-21 11:27:37.024 16818 WARNING keystoneclient.middleware.auth_token [req-64b4bab0-3b47-4326-b2e5-3bd457b3c797 ] Retrying on HTTP connection exception: 'HTTPSConnectionPool' object has no attribute 'insecure'

Reproduce:
$ env | grep OS
OS_REGION_NAME=RegionOne
OS_PASSWORD=openstack
OS_AUTH_URL=https://10.5.55.4:5000/v2.0
OS_USERNAME=admin
OS_TENANT_NAME=admin
OS_AUTH_PROTOCOL=https
OS_CACERT=/tmp/ca_cert.pem

$ neutron net-list
+--------------------------------------+---------+------------------------------------------------------+
| id | name | subnets |
+--------------------------------------+---------+------------------------------------------------------+
| 3acfbdb6-6895-48cf-9603-d09ea7007aa4 | ext_net | e9ddba73-96d0-4036-927e-4b99b8c457ee 10.5.0.0/16 |
| 711f199b-0d8c-4e3a-beff-d8edffff8a1f | private | cd730f70-a032-4784-b4dd-3c20d0335c6f 192.168.21.0/24 |
+--------------------------------------+---------+------------------------------------------------------+

$ glance image-list
+--------------------------------------+---------+-------------+------------------+-----------+--------+
| ID | Name | Disk Format | Container Format | Size | Status |
+--------------------------------------+---------+-------------+------------------+-----------+--------+
| f791dcef-4b13-42b3-aca8-eb56e35a2879 | cirros | qcow2 | bare | 13200896 | active |
| ec738ebf-5c44-4a34-94e2-396d617af079 | precise | qcow2 | bare | 263389696 | active |
+--------------------------------------+---------+-------------+------------------+-----------+--------+

$ nova image-list
+--------------------------------------+---------+--------+--------+
| ID | Name | Status | Server |
+--------------------------------------+---------+--------+--------+
| f791dcef-4b13-42b3-aca8-eb56e35a2879 | cirros | ACTIVE | |
| ec738ebf-5c44-4a34-94e2-396d617af079 | precise | ACTIVE | |
+--------------------------------------+---------+--------+--------+

$ nova image-list
ERROR (ClientException): Unknown Error (HTTP 503)

$ nova image-list
ERROR (ClientException): Unknown Error (HTTP 503)

Other commands still work:

$ neutron net-list
+--------------------------------------+---------+------------------------------------------------------+
| id | name | subnets |
+--------------------------------------+---------+------------------------------------------------------+
| 3acfbdb6-6895-48cf-9603-d09ea7007aa4 | ext_net | e9ddba73-96d0-4036-927e-4b99b8c457ee 10.5.0.0/16 |
| 711f199b-0d8c-4e3a-beff-d8edffff8a1f | private | cd730f70-a032-4784-b4dd-3c20d0335c6f 192.168.21.0/24 |
+--------------------------------------+---------+------------------------------------------------------+

$ glance image-list
+--------------------------------------+---------+-------------+------------------+-----------+--------+
| ID | Name | Disk Format | Container Format | Size | Status |
+--------------------------------------+---------+-------------+------------------+-----------+--------+
| f791dcef-4b13-42b3-aca8-eb56e35a2879 | cirros | qcow2 | bare | 13200896 | active |
| ec738ebf-5c44-4a34-94e2-396d617af079 | precise | qcow2 | bare | 263389696 | active |
+--------------------------------------+---------+-------------+------------------+-----------+--------+

Fix nova commands for one run:
$ service nova-api-os-compute restart
nova-api-os-compute stop/waiting
nova-api-os-compute start/running, process 16910

$ nova image-list +--------------------------------------+---------+--------+--------+
| ID | Name | Status | Server |
+--------------------------------------+---------+--------+--------+
| f791dcef-4b13-42b3-aca8-eb56e35a2879 | cirros | ACTIVE | |
| ec738ebf-5c44-4a34-94e2-396d617af079 | precise | ACTIVE | |
+--------------------------------------+---------+--------+--------+
$ nova image-list
ERROR (Unauthorized): Unauthorized (HTTP 401)

[Liam Young (gnuoy)]

I do not see this behaviour on Icehouse, Kilo or Liberty

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Edward Hope-Morley (hopem)]

This bug is actually in python-glanceclient and should fixed by https://bugs.launchpad.net/python-glanceclient/+bug/1362766 which landed in 0.14.2 (Juno UCA has 0.14.0)

[Edward Hope-Morley (hopem)]

I can confirm that the patch from 1362766 does fix this issue. I will propose the SRU on bug 1362766 so please see that bug for progress.

[Edward Hope-Morley (hopem)]

actually i'll propose SRU to bug 1347150 since it is the original bug raised for the issue.