connection reset by peer during glance image-create with vcenter backend

[Serg Lystopad]

Environment:
HA with vCenter hypevisor
nova-network vlanmanager
cinder VMwareVcVmdkDriver

node is CentOS 6.5

api: '1.0'
astute_sha: 16b252d93be6aaa73030b8100cf8c5ca6a970a91
auth_required: true
build_id: 2014-12-26_14-25-46
build_number: '58'
feature_groups:
- mirantis
fuellib_sha: fde8ba5e11a1acaf819d402c645c731af450aff0
fuelmain_sha: 81d38d6f2903b5a8b4bee79ca45a54b76c1361b8
nailgun_sha: 5f91157daa6798ff522ca9f6d34e7e135f150a90
ostf_sha: a9afb68710d809570460c29d6c3293219d3624d4
production: docker
release: '6.0'
release_versions:
  2014.2-6.0:
    VERSION:
      api: '1.0'
      astute_sha: 16b252d93be6aaa73030b8100cf8c5ca6a970a91
      build_id: 2014-12-26_14-25-46
      build_number: '58'
      feature_groups:
      - mirantis
      fuellib_sha: fde8ba5e11a1acaf819d402c645c731af450aff0
      fuelmain_sha: 81d38d6f2903b5a8b4bee79ca45a54b76c1361b8
      nailgun_sha: 5f91157daa6798ff522ca9f6d34e7e135f150a90
      ostf_sha: a9afb68710d809570460c29d6c3293219d3624d4
      production: docker
      release: '6.0'

2015-03-24 19:33:14.780 57129 ERROR glance_store._drivers.vmware_datastore [a9eaa1a6-d60c-4f62-8abc-7d2a18e7d4b5 35eec72551634331b0c00b6d1b10fc8f 271797c584e2445d83b82fc57697daaf - - -] Failed to upload content of image 329a5529-e3d7-478f-b63a-ff33e68b9259
2015-03-24 19:33:14.780 57129 TRACE glance_store._drivers.vmware_datastore Traceback (most recent call last):
2015-03-24 19:33:14.780 57129 TRACE glance_store._drivers.vmware_datastore File "/usr/lib/python2.6/site-packages/glance_store/_drivers/vmware_datastore.py", line 346, in add
2015-03-24 19:33:14.780 57129 TRACE glance_store._drivers.vmware_datastore content=image_file)
2015-03-24 19:33:14.780 57129 TRACE glance_store._drivers.vmware_datastore File "/usr/lib/python2.6/site-packages/glance_store/_drivers/vmware_datastore.py", line 489, in _get_http_conn
2015-03-24 19:33:14.780 57129 TRACE glance_store._drivers.vmware_datastore conn.request(method, url, content, headers)
2015-03-24 19:33:14.780 57129 TRACE glance_store._drivers.vmware_datastore File "/usr/lib64/python2.6/httplib.py", line 914, in request
2015-03-24 19:33:14.780 57129 TRACE glance_store._drivers.vmware_datastore self._send_request(method, url, body, headers)
2015-03-24 19:33:14.780 57129 TRACE glance_store._drivers.vmware_datastore File "/usr/lib64/python2.6/httplib.py", line 954, in _send_request
2015-03-24 19:33:14.780 57129 TRACE glance_store._drivers.vmware_datastore self.send(body)
2015-03-24 19:33:14.780 57129 TRACE glance_store._drivers.vmware_datastore File "/usr/lib64/python2.6/httplib.py", line 756, in send
2015-03-24 19:33:14.780 57129 TRACE glance_store._drivers.vmware_datastore self.sock.sendall(data)
2015-03-24 19:33:14.780 57129 TRACE glance_store._drivers.vmware_datastore File "/usr/lib/python2.6/site-packages/eventlet/green/ssl.py", line 137, in sendall
2015-03-24 19:33:14.780 57129 TRACE glance_store._drivers.vmware_datastore v = self.send(data[count:])
2015-03-24 19:33:14.780 57129 TRACE glance_store._drivers.vmware_datastore File "/usr/lib/python2.6/site-packages/eventlet/green/ssl.py", line 113, in send
2015-03-24 19:33:14.780 57129 TRACE glance_store._drivers.vmware_datastore super(GreenSSLSocket, self).send, data, flags)
2015-03-24 19:33:14.780 57129 TRACE glance_store._drivers.vmware_datastore File "/usr/lib/python2.6/site-packages/eventlet/green/ssl.py", line 80, in _call_trampolining
2015-03-24 19:33:14.780 57129 TRACE glance_store._drivers.vmware_datastore return func(*a, **kw)
2015-03-24 19:33:14.780 57129 TRACE glance_store._drivers.vmware_datastore File "/usr/lib64/python2.6/ssl.py", line 174, in send
2015-03-24 19:33:14.780 57129 TRACE glance_store._drivers.vmware_datastore v = self._sslobj.write(data)
2015-03-24 19:33:14.780 57129 TRACE glance_store._drivers.vmware_datastore error: [Errno 104] Connection reset by peer

[Serg Lystopad (slystopad)]

is glance sensible to selfsigned certificates on vcenter server?

[Serg Lystopad (slystopad)]

cirros.vmdk successfully uploaded as part of MOS deployment,
also i successfully uploaded qcow2 image to glance,
but after maybe and hour I seesame error, and cant upload image anymore

[Serg Lystopad (slystopad)]

looks like https://bugs.launchpad.net/glance/+bug/1402354

[Serg Lystopad (slystopad)]

i've restarted glance-api and started image-create. It is in progress (image is 72G)

[Serg Lystopad (slystopad)]

I got this from slient side
http://paste.openstack.org/show/196384/

On glance-api side errors are:
http://paste.openstack.org/show/196385/

[Serg Lystopad (slystopad)]

I can see image file on glance backend, size of file is correct (so 100% was uploaded)

[Serg Lystopad (slystopad)]

glance registry fails in finding token
http://paste.openstack.org/show/196386/

Did token expired while image uploading was in progress?

[Serg Lystopad (slystopad)]

I've just tried to upload TestVMcirros vmdk (after some idle period) and it fails like in original message
http://paste.openstack.org/show/196392/

As i can see glance-api restart fixes this for some period
http://paste.openstack.org/show/196394/

So

glance-api start -> issue isn't here -> idle -> issue is here -> glance-api restart -> issue is here ....

[Serg Lystopad (slystopad)]

> glance registry fails in finding token
> http://paste.openstack.org/show/196386/
>
> Did token expired while image uploading was in progress?

I've configured in keystone.conf:
expiration=14400

and started image-create.

Image was created as expected:
http://paste.openstack.org/show/196604/

[Sabari Murugesan (smurugesan)]

This was a known issue before glance_store v0.1.11. What version of glance_store are you running with ? If it's <0.1.11 can you please try the patch from https://review.openstack.org/#/c/141665/ or upgrade to it ?

[Serg Lystopad (slystopad)]

glance_store package is v0.1.7

[Sabari Murugesan (smurugesan)]

@sergiy: Could you upgrade to the newer glance_store or patch with the fix in https://review.openstack.org/#/c/141665/ and verify ?

[Serg Lystopad (slystopad)]

I've already manually upgraded glance_store to 0.1.12 (git clone) and upgraded it's dependencies (pip install).
Seems like this fixes an issue. There are traces in logs about unauthenticated sessions to vcenter and there are records about reconnections.

pip install enum34
pip install oslo.utils==1.2.0
pip install --upgrade oslo.config
pip install --upgrade oslo.vmware
This also has upgraded oslo.concurrency to 1.8.0

# pip search oslo.vmware
oslo.vmware - Oslo VMware library for OpenStack projects
  INSTALLED: 0.11.1 (latest)

# pip search oslo.concurrency
oslo.concurrency - oslo.concurrency library
  INSTALLED: 1.8.0 (latest)

I'm not sure that this upgrade doesn't break other functionality

[Mike Fedosin (mfedosin)]

Yep, I made a patch for 6.0 branch with backported fix from 0.1.12 https://review.fuel-infra.org/#/c/5169/

It has some issues with unit tests, but I don't think it's critical, because locally all tests have passed well.

I'm going to make the same patches for 6.0.1 and 6.1.

[OSCI Robot (oscirobot)]

RPM package glance_store has been built for project openstack/glance_store
Package version == 0.1.7, package release == fuel6.0.mira2.git.c396c8b.15269c0

Changeset: https://review.fuel-infra.org/5169
project: openstack/glance_store
branch: openstack-ci/fuel-6.0/2014.2
author: Mikhail Fedosin
committer: Mikhail Fedosin
subject: Check VMware session before uploading image
status: patchset-created

Files placed on repository:
python-glance-store-0.1.7-fuel6.0.mira2.git.c396c8b.15269c0.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0-stable-5169/centos

[OSCI Robot (oscirobot)]

DEB package glance_store has been built for project openstack/glance_store
Package version == 0.1.7, package release == fuel6.0~mira2+git.c396c8b.15269c0

Changeset: https://review.fuel-infra.org/5169
project: openstack/glance_store
branch: openstack-ci/fuel-6.0/2014.2
author: Mikhail Fedosin
committer: Mikhail Fedosin
subject: Check VMware session before uploading image
status: patchset-created

Files placed on repository:
python-glance-store-doc_0.1.7-fuel6.0~mira2+git.c396c8b.15269c0_all.deb
python-glance-store_0.1.7-fuel6.0~mira2+git.c396c8b.15269c0_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0-stable-5169/ubuntu

[OSCI Robot (oscirobot)]

RPM package glance_store has been built for project openstack/glance_store
Package version == 0.1.7, package release == fuel6.0.1.mira3.git.cc7635c.15269c0

Changeset: https://review.fuel-infra.org/5182
project: openstack/glance_store
branch: openstack-ci/fuel-6.0.1/2014.2
author: Mikhail Fedosin
committer: Mikhail Fedosin
subject: Check VMware session before uploading image
status: patchset-created

Files placed on repository:
python-glance-store-0.1.7-fuel6.0.1.mira3.git.cc7635c.15269c0.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0.1-stable-5182/centos

[OSCI Robot (oscirobot)]

DEB package glance_store has been built for project openstack/glance_store
Package version == 0.1.7, package release == fuel6.0.1~mira3+git.cc7635c.15269c0

Changeset: https://review.fuel-infra.org/5182
project: openstack/glance_store
branch: openstack-ci/fuel-6.0.1/2014.2
author: Mikhail Fedosin
committer: Mikhail Fedosin
subject: Check VMware session before uploading image
status: patchset-created

Files placed on repository:
python-glance-store-doc_0.1.7-fuel6.0.1~mira3+git.cc7635c.15269c0_all.deb
python-glance-store_0.1.7-fuel6.0.1~mira3+git.cc7635c.15269c0_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0.1-stable-5182/ubuntu

[OSCI Robot (oscirobot)]

RPM package glance_store has been built for project openstack/glance_store
Package version == 0.1.7, package release == fuel6.0.mira3.git.d594987.15269c0

Changeset: https://review.fuel-infra.org/5184
project: openstack/glance_store
branch: openstack-ci/fuel-6.0-updates/2014.2
author: Mikhail Fedosin
committer: Mikhail Fedosin
subject: Check VMware session before uploading image
status: patchset-created

Files placed on repository:
python-glance-store-0.1.7-fuel6.0.mira3.git.d594987.15269c0.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0-updates-stable-5184/centos

[OSCI Robot (oscirobot)]

DEB package glance_store has been built for project openstack/glance_store
Package version == 0.1.7, package release == fuel6.0~mira3+git.d594987.15269c0

Changeset: https://review.fuel-infra.org/5184
project: openstack/glance_store
branch: openstack-ci/fuel-6.0-updates/2014.2
author: Mikhail Fedosin
committer: Mikhail Fedosin
subject: Check VMware session before uploading image
status: patchset-created

Files placed on repository:
python-glance-store-doc_0.1.7-fuel6.0~mira3+git.d594987.15269c0_all.deb
python-glance-store_0.1.7-fuel6.0~mira3+git.d594987.15269c0_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0-updates-stable-5184/ubuntu

[OSCI Robot (oscirobot)]

RPM package glance_store has been built for project openstack/glance_store
Package version == 0.1.7, package release == fuel6.0.mira3.git.f956d2d.15269c0

Changeset: https://review.fuel-infra.org/5184
project: openstack/glance_store
branch: openstack-ci/fuel-6.0-updates/2014.2
author: Mikhail Fedosin
committer: Mikhail Fedosin
subject: Check VMware session before uploading image
status: patchset-created

Files placed on repository:
python-glance-store-0.1.7-fuel6.0.mira3.git.f956d2d.15269c0.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0-updates-stable-5184/centos

[OSCI Robot (oscirobot)]

DEB package glance_store has been built for project openstack/glance_store
Package version == 0.1.7, package release == fuel6.0~mira3+git.f956d2d.15269c0

Changeset: https://review.fuel-infra.org/5184
project: openstack/glance_store
branch: openstack-ci/fuel-6.0-updates/2014.2
author: Mikhail Fedosin
committer: Mikhail Fedosin
subject: Check VMware session before uploading image
status: patchset-created

Files placed on repository:
python-glance-store-doc_0.1.7-fuel6.0~mira3+git.f956d2d.15269c0_all.deb
python-glance-store_0.1.7-fuel6.0~mira3+git.f956d2d.15269c0_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0-updates-stable-5184/ubuntu

[OSCI Robot (oscirobot)]

RPM package glance_store has been built for project openstack/glance_store
Package version == 0.1.7, package release == fuel6.0.1.mira3.git.b9a9ee7.15269c0

Changeset: https://review.fuel-infra.org/5182
project: openstack/glance_store
branch: openstack-ci/fuel-6.0.1/2014.2
author: Mikhail Fedosin
committer: Mikhail Fedosin
subject: Check VMware session before uploading image
status: patchset-created

Files placed on repository:
python-glance-store-0.1.7-fuel6.0.1.mira3.git.b9a9ee7.15269c0.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0.1-stable-5182/centos

[OSCI Robot (oscirobot)]

DEB package glance_store has been built for project openstack/glance_store
Package version == 0.1.7, package release == fuel6.0.1~mira3+git.b9a9ee7.15269c0

Changeset: https://review.fuel-infra.org/5182
project: openstack/glance_store
branch: openstack-ci/fuel-6.0.1/2014.2
author: Mikhail Fedosin
committer: Mikhail Fedosin
subject: Check VMware session before uploading image
status: patchset-created

Files placed on repository:
python-glance-store-doc_0.1.7-fuel6.0.1~mira3+git.b9a9ee7.15269c0_all.deb
python-glance-store_0.1.7-fuel6.0.1~mira3+git.b9a9ee7.15269c0_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0.1-stable-5182/ubuntu

[Serg Lystopad (slystopad)]

Patch works fine.
Package python-glance-store-0.1.7-fuel6.0.mira2.git.c396c8b.15269c0.noarch.rpm fixes the issue.
Thanks

[OSCI Robot (oscirobot)]

RPM package glance_store has been built for project openstack/glance_store
Package version == 0.1.7, package release == fuel6.0.mira3.git.6b86b3d.15269c0

Changeset: https://review.fuel-infra.org/5184
project: openstack/glance_store
branch: openstack-ci/fuel-6.0-updates/2014.2
author: Mikhail Fedosin
committer: Alex Khivin
subject: Check VMware session before uploading image
status: patchset-created

Files placed on repository:
python-glance-store-0.1.7-fuel6.0.mira3.git.6b86b3d.15269c0.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0-updates-stable-5184/centos

[OSCI Robot (oscirobot)]

DEB package glance_store has been built for project openstack/glance_store
Package version == 0.1.7, package release == fuel6.0~mira3+git.6b86b3d.15269c0

Changeset: https://review.fuel-infra.org/5184
project: openstack/glance_store
branch: openstack-ci/fuel-6.0-updates/2014.2
author: Mikhail Fedosin
committer: Alex Khivin
subject: Check VMware session before uploading image
status: patchset-created

Files placed on repository:
python-glance-store-doc_0.1.7-fuel6.0~mira3+git.6b86b3d.15269c0_all.deb
python-glance-store_0.1.7-fuel6.0~mira3+git.6b86b3d.15269c0_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0-updates-stable-5184/ubuntu

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Mike Fedosin <email address hidden>
Review: https://review.fuel-infra.org/8116

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Mike Fedosin <email address hidden> on branch: openstack-ci/fuel-7.0/2015.1.0
Review: https://review.fuel-infra.org/8116