I. Two other ways to describe this issue:
- Why did Glance return an HTTP 503 [0] when asked to upload a ciros image?
- Glance upload fails and logs "since image size is zero we will be doing resize-before-write for each chunk which will be considerably slower than normal"

II. Q/A from CI logs: 
- Glance logs show that the rbd scheme was enabled [1] 
- Glance logs show Glance creating image with order 23 and size 0 [2] in rbd.py [3]
- There have been cases where the ceph config was the root cause of this error [4] 
- Is the glance-api.conf correct? Yes [6] (but see open question A)
- Was the glance container image mounted with the ceph.conf ? Yes [7]
- Is a normal looking ceph.conf on the subnode (i.e. the container host?) ? Yes [8] 
- Is ceph.client.openstack.keyring on the subnode ? Yes [8]
- Was a change made to support this? Yes, puppet-ceph still genereates the configs and glance container was changed to use them [9]

Two open questions:
A. In the glance conf, rbd_store_ceph_conf has been commented out but worked in the past (for default reasons) might this be affecting us now? 

B. Are the permissions correct of the ceph keyring set so that the glance user can read it?
   - CI logs do not confirm the 644 permissions, but they _should_ be correct... 
   - They have been 644 in the past [10] and need to be so that the container can read the key so why should this change?

Next Steps:
- Attempting to reproduce in my local environment


[0] http://logs.openstack.org/29/490129/4/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container/1d6c57a/logs/undercloud/home/jenkins/overcloud_validate.log.txt.gz#_2017-08-12_13_29_17
[1] http://logs.openstack.org/29/490129/4/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container/1d6c57a/logs/subnode-2/var/log/containers/glance/api.log.txt.gz#_2017-08-12_13_23_40_472
[2] http://logs.openstack.org/29/490129/4/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container/1d6c57a/logs/subnode-2/var/log/containers/glance/api.log.txt.gz#_2017-08-12_13_27_13_144
[3] https://github.com/openstack/glance_store/blob/master/glance_store/_drivers/rbd.py#L460-L461
[4] https://ask.openstack.org/en/question/78493/glance-image-create-ceph-problem/
[5] http://logs.openstack.org/29/490129/4/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container/1d6c57a/logs/subnode-2/var/log/config-data/glance_api/etc/glance/glance-api.conf.txt.gz
[6] glance-api.conf.txt
"""
[glance_store]
stores=http,rbd
default_store=rbd
rbd_store_pool=images
rbd_store_user=openstack
#rbd_store_ceph_conf = /etc/ceph/ceph.conf  # <--- this will default correctly to what's in comment
show_image_direct_url=True
"""
[7] http://logs.openstack.org/29/490129/4/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container/1d6c57a/logs/subnode-2/var/log/extra/docker/containers/glance_api/docker_info.log.txt.gz (note the "/etc/ceph:/var/lib/kolla/config_files/src-ceph:ro")
[8] http://logs.openstack.org/29/490129/4/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container/1d6c57a/logs/subnode-2/etc/ceph/
[9] https://review.openstack.org/#/c/482500
[10] https://github.com/openstack/puppet-ceph/blob/28e8f4525f4448a9f585f7ff4212fa9df58f4464/examples/nodes/client.yaml