Fuel for OpenStack

[fuel-library] Glance with vmware backend - certificate verify failed

  1. Series newton
  2. Bug #1559067
Bug #1559067 reported by Artem Savinov
18
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Alexander Arzhanov
Fuel for OpenStack 10.0
Mitaka
Fix Released
High
Alexander Arzhanov
Fuel for OpenStack 9.0
Newton
Fix Committed
High
Alexander Arzhanov
Fuel for OpenStack 10.0

Bug Description

Service glance-api not start .

Steps to reproduce:
Deploy Feul 9 #78 with nsxv-fuel- plugin and enable "VMware vCenter/ESXi datastore for images (Glance)" from Settings tab-> Storage. On Settings tab-> Security -> Public TLS options not enabled.

Expected results: glance work fine.

Actual result: glance-api service not start, with error:

2016-03-18 10:42:45.331 18566 CRITICAL glance [-] SSLError: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
2016-03-18 10:42:45.331 18566 ERROR glance Traceback (most recent call last):
2016-03-18 10:42:45.331 18566 ERROR glance File "/usr/bin/glance-api", line 10, in <module>
2016-03-18 10:42:45.331 18566 ERROR glance sys.exit(main())
2016-03-18 10:42:45.331 18566 ERROR glance File "/usr/lib/python2.7/dist-packages/glance/cmd/api.py", line 91, in main
2016-03-18 10:42:45.331 18566 ERROR glance server.start(config.load_paste_app('glance-api'), default_port=9292)
2016-03-18 10:42:45.331 18566 ERROR glance File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 311, in start
2016-03-18 10:42:45.331 18566 ERROR glance self.configure()
2016-03-18 10:42:45.331 18566 ERROR glance File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 387, in configure
2016-03-18 10:42:45.331 18566 ERROR glance initialize_glance_store()
2016-03-18 10:42:45.331 18566 ERROR glance File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 233, in initialize_glance_store
2016-03-18 10:42:45.331 18566 ERROR glance glance_store.create_stores(CONF)
2016-03-18 10:42:45.331 18566 ERROR glance File "/usr/lib/python2.7/dist-packages/glance_store/backend.py", line 189, in create_stores
2016-03-18 10:42:45.331 18566 ERROR glance store_instance.configure(re_raise_bsc=False)
2016-03-18 10:42:45.331 18566 ERROR glance File "/usr/lib/python2.7/dist-packages/glance_store/_drivers/vmware_datastore.py", line 282, in configure
2016-03-18 10:42:45.331 18566 ERROR glance self.session = self.reset_session()
2016-03-18 10:42:45.331 18566 ERROR glance File "/usr/lib/python2.7/dist-packages/glance_store/_drivers/vmware_datastore.py", line 249, in reset_session
2016-03-18 10:42:45.331 18566 ERROR glance insecure=self.api_insecure)
2016-03-18 10:42:45.331 18566 ERROR glance File "/usr/lib/python2.7/dist-packages/oslo_vmware/api.py", line 184, in __init__
2016-03-18 10:42:45.331 18566 ERROR glance self._create_session()
2016-03-18 10:42:45.331 18566 ERROR glance File "/usr/lib/python2.7/dist-packages/oslo_vmware/api.py", line 123, in func
2016-03-18 10:42:45.331 18566 ERROR glance return evt.wait()
2016-03-18 10:42:45.331 18566 ERROR glance File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
2016-03-18 10:42:45.331 18566 ERROR glance return hubs.get_hub().switch()
2016-03-18 10:42:45.331 18566 ERROR glance File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
2016-03-18 10:42:45.331 18566 ERROR glance return self.greenlet.switch()
2016-03-18 10:42:45.331 18566 ERROR glance File "/usr/lib/python2.7/dist-packages/oslo_vmware/common/loopingcall.py", line 124, in _inner
2016-03-18 10:42:45.331 18566 ERROR glance idle = self.f(*self.args, **self.kw)
2016-03-18 10:42:45.331 18566 ERROR glance File "/usr/lib/python2.7/dist-packages/oslo_vmware/api.py", line 96, in _func
2016-03-18 10:42:45.331 18566 ERROR glance result = f(*args, **kwargs)
2016-03-18 10:42:45.331 18566 ERROR glance File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
2016-03-18 10:42:45.331 18566 ERROR glance return f(*args, **kwargs)
2016-03-18 10:42:45.331 18566 ERROR glance File "/usr/lib/python2.7/dist-packages/oslo_vmware/api.py", line 231, in _create_session
2016-03-18 10:42:45.331 18566 ERROR glance session_manager = self.vim.service_content.sessionManager
2016-03-18 10:42:45.331 18566 ERROR glance File "/usr/lib/python2.7/dist-packages/oslo_vmware/api.py", line 200, in vim
2016-03-18 10:42:45.331 18566 ERROR glance pool_maxsize=self._pool_size)
2016-03-18 10:42:45.331 18566 ERROR glance File "/usr/lib/python2.7/dist-packages/oslo_vmware/vim.py", line 44, in __init__
2016-03-18 10:42:45.331 18566 ERROR glance pool_maxsize)
2016-03-18 10:42:45.331 18566 ERROR glance File "/usr/lib/python2.7/dist-packages/oslo_vmware/service.py", line 201, in __init__
2016-03-18 10:42:45.331 18566 ERROR glance cache=_CACHE)
2016-03-18 10:42:45.331 18566 ERROR glance File "/usr/lib/python2.7/dist-packages/suds/client.py", line 120, in __init__
2016-03-18 10:42:45.331 18566 ERROR glance self.wsdl = reader.open(url)
2016-03-18 10:42:45.331 18566 ERROR glance File "/usr/lib/python2.7/dist-packages/suds/reader.py", line 104, in open
2016-03-18 10:42:45.331 18566 ERROR glance wsdl = self.fn(url, self.options)
2016-03-18 10:42:45.331 18566 ERROR glance File "/usr/lib/python2.7/dist-packages/suds/wsdl.py", line 164, in __init__
2016-03-18 10:42:45.331 18566 ERROR glance d = reader.open(url)
2016-03-18 10:42:45.331 18566 ERROR glance File "/usr/lib/python2.7/dist-packages/suds/reader.py", line 148, in open
2016-03-18 10:42:45.331 18566 ERROR glance xml = self.__fetch(url)
2016-03-18 10:42:45.331 18566 ERROR glance File "/usr/lib/python2.7/dist-packages/suds/reader.py", line 188, in __fetch
2016-03-18 10:42:45.331 18566 ERROR glance fp = self.options.transport.open(request)
2016-03-18 10:42:45.331 18566 ERROR glance File "/usr/lib/python2.7/dist-packages/oslo_vmware/service.py", line 148, in open
2016-03-18 10:42:45.331 18566 ERROR glance resp = self.session.get(request.url, verify=self.verify)
2016-03-18 10:42:45.331 18566 ERROR glance File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 480, in get
2016-03-18 10:42:45.331 18566 ERROR glance return self.request('GET', url, **kwargs)
2016-03-18 10:42:45.331 18566 ERROR glance File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 468, in request
2016-03-18 10:42:45.331 18566 ERROR glance resp = self.send(prep, **send_kwargs)
2016-03-18 10:42:45.331 18566 ERROR glance File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 576, in send
2016-03-18 10:42:45.331 18566 ERROR glance r = adapter.send(request, **kwargs)
2016-03-18 10:42:45.331 18566 ERROR glance File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 447, in send
2016-03-18 10:42:45.331 18566 ERROR glance raise SSLError(e, request=request)
2016-03-18 10:42:45.331 18566 ERROR glance SSLError: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

in /etc/glance/glance-api.conf get option vmware_api_insecure=False

Workaround:
set vmware_api_insecure=True in /etc/glance/glance-api.conf

See original description
Artem Savinov (asavinov)
Changed in fuel:
milestone: none → 9.0
Artem Savinov (asavinov)
description: updated
Alexander Saprykin (cutwater)
tags: added: area-partners
Changed in fuel:
assignee: nobody → Fuel Partner Integration Team (fuel-partner)
importance: Undecided → High
status: New → Confirmed
Alexander Arzhanov (aarzhanov)
Changed in fuel:
assignee: Fuel Partner Integration Team (fuel-partner) → Partner Centric Engineering (fuel-partner-engineering)
Artem Savinov (asavinov)
Changed in fuel:
assignee: Partner Centric Engineering (fuel-partner-engineering) → Artem Savinov (asavinov)
Artem Savinov (asavinov)
Changed in fuel:
status: Confirmed → In Progress
Artem Savinov (asavinov)
summary: - Glance with vmware backend - certificate verify failed
+ [fuel-library] Glance with vmware backend - certificate verify failed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/296508

OpenStack Infra (hudson-openstack)
Changed in fuel:
assignee: Artem Savinov (asavinov) → Alex Schultz (alex-schultz)
Alex Schultz (alex-schultz)
Changed in fuel:
assignee: Alex Schultz (alex-schultz) → Artem Savinov (asavinov)
OpenStack Infra (hudson-openstack)
Changed in fuel:
assignee: Artem Savinov (asavinov) → Igor Zinovik (izinovik)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (master)

Fix proposed to branch: master
Review: https://review.openstack.org/301000

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/301001

OpenStack Infra (hudson-openstack)
Changed in fuel:
assignee: Igor Zinovik (izinovik) → Andriy Popovych (popovych-andrey)
Alexander Arzhanov (aarzhanov)
Changed in fuel:
assignee: Andriy Popovych (popovych-andrey) → Alexander Arzhanov (aarzhanov)
Revision history for this message
Alexander Arzhanov (aarzhanov) wrote :

Another review : https://review.openstack.org/#/c/300495/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (master)

Change abandoned by Artem Savinov (<email address hidden>) on branch: master
Review: https://review.openstack.org/296508
Reason: https://review.openstack.org/#/c/301001/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-ui (master)

Fix proposed to branch: master
Review: https://review.openstack.org/305880

Changed in fuel:
assignee: Alexander Arzhanov (aarzhanov) → Anton Zemlyanov (azemlyanov)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-web (master)

Reviewed: https://review.openstack.org/301000
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=59c8b8c7909de07bcbb97ec6c9ae509f98bdaff2
Submitter: Jenkins
Branch: master

commit 59c8b8c7909de07bcbb97ec6c9ae509f98bdaff2
Author: Igor Zinovik <email address hidden>
Date: Mon Apr 4 10:08:25 2016 +0300

    Allow user upload CA bundle file for Glance with VMware backend

    Add field that allows user to upload CA certificate that emitted
    vCenters TLS/SSL certificate. It allow Glance with VMware backend verify
    connection to vCenter server (verification is turned on by default).
    If user did not uploaded certificate then, glance will be configured
    to skip verification step during connection to vCenter.

    Partial-bug: #1559067
    DocImpact: Document how to use 'CA file' field on VMware tab.
    Change-Id: I810bc000e54b941018a1190acb26d33150b74ce0

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/310435

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-web (stable/mitaka)

Reviewed: https://review.openstack.org/310435
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=80fafcd4eb010bee394c775fabf19849b66a70df
Submitter: Jenkins
Branch: stable/mitaka

commit 80fafcd4eb010bee394c775fabf19849b66a70df
Author: Igor Zinovik <email address hidden>
Date: Mon Apr 4 10:08:25 2016 +0300

    Allow user upload CA bundle file for Glance with VMware backend

    Add field that allows user to upload CA certificate that emitted
    vCenters TLS/SSL certificate. It allow Glance with VMware backend verify
    connection to vCenter server (verification is turned on by default).
    If user did not uploaded certificate then, glance will be configured
    to skip verification step during connection to vCenter.

    Partial-bug: #1559067
    DocImpact: Document how to use 'CA file' field on VMware tab.
    Change-Id: I810bc000e54b941018a1190acb26d33150b74ce0
    (cherry picked from commit 59c8b8c7909de07bcbb97ec6c9ae509f98bdaff2)

tags: added: in-stable-mitaka
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-ui (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/314500

OpenStack Infra (hudson-openstack)
Changed in fuel:
assignee: Igor Zinovik (izinovik) → Anton Zemlyanov (azemlyanov)
OpenStack Infra (hudson-openstack)
Changed in fuel:
assignee: Anton Zemlyanov (azemlyanov) → Igor Zinovik (izinovik)
OpenStack Infra (hudson-openstack)
Changed in fuel:
assignee: Igor Zinovik (izinovik) → Anton Zemlyanov (azemlyanov)
OpenStack Infra (hudson-openstack)
Changed in fuel:
assignee: Anton Zemlyanov (azemlyanov) → Igor Zinovik (izinovik)
OpenStack Infra (hudson-openstack)
Changed in fuel:
assignee: Igor Zinovik (izinovik) → Anton Zemlyanov (azemlyanov)
OpenStack Infra (hudson-openstack)
Changed in fuel:
assignee: Anton Zemlyanov (azemlyanov) → Igor Zinovik (izinovik)
OpenStack Infra (hudson-openstack)
Changed in fuel:
assignee: Igor Zinovik (izinovik) → Anton Zemlyanov (azemlyanov)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/301001
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=ac2801b8ba9a73542e14003ed6069e2bb0d1683f
Submitter: Jenkins
Branch: master

commit ac2801b8ba9a73542e14003ed6069e2bb0d1683f
Author: Igor Zinovik <email address hidden>
Date: Mon Apr 4 10:39:15 2016 +0300

    Configure Glance SSL verification settings (VMware backend)

    Fetch CA certificate bundle and pass it into puppet-glance module
    (glance::backend::vsphere class).

    Default behaviour of class is to turn off verification (we pass `undef'
    value to ::glance::backend::vsphere class), if `vcenter_ca_file' is
    provided certificate is verified.

    Support of CA certificate setting was introduced in change request
    of puppet-glance module Icef5c35ad1128df465da548dd880a0dfeeadb5e1

    CA bundle file gets uploaded to `/etc/glance/'.

    Provide noop test case.

    Closes-bug: #1559067
    Change-Id: Id0976706221dbac60c424ba9eb865d84411fc6d0

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/316886

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/mitaka)

Reviewed: https://review.openstack.org/316886
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=9121d6b6377caa1ac832ab30627bab4c4990e203
Submitter: Jenkins
Branch: stable/mitaka

commit 9121d6b6377caa1ac832ab30627bab4c4990e203
Author: Igor Zinovik <email address hidden>
Date: Mon Apr 4 10:39:15 2016 +0300

    Configure Glance SSL verification settings (VMware backend)

    Fetch CA certificate bundle and pass it into puppet-glance module
    (glance::backend::vsphere class).

    Default behaviour of class is to turn off verification (we pass `undef'
    value to ::glance::backend::vsphere class), if `vcenter_ca_file' is
    provided certificate is verified.

    Support of CA certificate setting was introduced in change request
    of puppet-glance module Icef5c35ad1128df465da548dd880a0dfeeadb5e1

    CA bundle file gets uploaded to `/etc/glance/'.

    Provide noop test case.

    Closes-bug: #1559067
    Change-Id: Id0976706221dbac60c424ba9eb865d84411fc6d0
    (cherry picked from commit ac2801b8ba9a73542e14003ed6069e2bb0d1683f)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-ui (master)

Reviewed: https://review.openstack.org/305880
Committed: https://git.openstack.org/cgit/openstack/fuel-ui/commit/?id=97e891fa50828b991a4a26261dd15ed9a8345597
Submitter: Jenkins
Branch: master

commit 97e891fa50828b991a4a26261dd15ed9a8345597
Author: Anton Zemlyanov <email address hidden>
Date: Thu Apr 14 17:10:25 2016 +0300

    Added certificate upload support for Glance in VmWare tab

    Added file upload support that allows certificate upload on
    the VmWare tab.

    Change-Id: I3dcb73f588ed968dd41f5dc2f94ce7538154a9d0
    Partial-Bug: 1559067

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-ui (stable/mitaka)

Reviewed: https://review.openstack.org/314500
Committed: https://git.openstack.org/cgit/openstack/fuel-ui/commit/?id=3041d69527c001daa9123356ef06d6cb40af3567
Submitter: Jenkins
Branch: stable/mitaka

commit 3041d69527c001daa9123356ef06d6cb40af3567
Author: Anton Zemlyanov <email address hidden>
Date: Thu Apr 14 17:10:25 2016 +0300

    Added certificate upload support for Glance in VmWare tab

    Added file upload support that allows certificate upload on
    the VmWare tab.

    Change-Id: I3dcb73f588ed968dd41f5dc2f94ce7538154a9d0
    Partial-Bug: 1559067

Revision history for this message
Ilya Bumarskov (ibumarskov) wrote :

I still can reproduce this bug on Fuel 9.0 iso#372:

Deployment has failed. All nodes are finished. Failed tasks: Task[glance/1], Task[vmware-dvs-neutron-agent-install/1], Task[glance/2], Task[glance/5] Stopping the deployment process!

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/318553

Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/318553
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=1ae0afc43eec049cb13bc117694995e28cdc41b3
Submitter: Jenkins
Branch: master

commit 1ae0afc43eec049cb13bc117694995e28cdc41b3
Author: Alexander Arzhanov <email address hidden>
Date: Thu May 19 13:14:29 2016 +0300

    Fix get incorrect argument in hash (Glance VMware)

    vc_ca_file not contain any data if user not specify CA certificate
    bundle.

    Change-Id: I98b1f6411071586060f80a363952502b7ff3e08e
    Closes-bug: #1559067

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/318615

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/mitaka)

Reviewed: https://review.openstack.org/318615
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=c4ec151cc0a49ab2433297bdf4fad82c35b97118
Submitter: Jenkins
Branch: stable/mitaka

commit c4ec151cc0a49ab2433297bdf4fad82c35b97118
Author: Alexander Arzhanov <email address hidden>
Date: Thu May 19 13:14:29 2016 +0300

    Fix get incorrect argument in hash (Glance VMware)

    vc_ca_file not contain any data if user not specify CA certificate
    bundle.

    Change-Id: I98b1f6411071586060f80a363952502b7ff3e08e
    Closes-bug: #1559067
    (cherry picked from commit 1ae0afc43eec049cb13bc117694995e28cdc41b3)

Revision history for this message
Ilya Bumarskov (ibumarskov) wrote :

Verified on Fuel mos-iso#449 with uploaded certificate for glance backend (vCenter certificate)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.