Cirros Image creation failures

Bug #1538642 reported by Hugh Saunders
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack-Ansible
Fix Released
Low
Jimmy McCrory

Bug Description

Sometimes the cirros image fails to upload to glance. Its not clear whether this is a problem with glance, the backend glance is using, or the server from which the image is being pulled. Also this task is not retried.

Example failure:

===========================
**15:48:22** TASK: [os_tempest | Ensure cirros image] **************************************
**15:48:22** failed: [jrpcaio-e65_utility_container-f235d0d1] => {"failed": true, "parsed": false}
**15:48:22** OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014
**15:48:22** debug1: Reading configuration data /etc/ssh/ssh_config
**15:48:22** debug1: /etc/ssh/ssh_config line 19: Applying options for *
**15:48:22** debug1: auto-mux: Trying existing master
**15:48:22** debug1: mux_client_request_session: master session id: 2
**15:48:22** Traceback (most recent call last):
**15:48:22** File "<stdin>", line 1845, in <module>
**15:48:22** File "<stdin>", line 229, in main
**15:48:22** File "<stdin>", line 149, in route
**15:48:22** File "<stdin>", line 202, in create_image
**15:48:22** File "/usr/local/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 319, in create
**15:48:22** data=image_data)
**15:48:22** File "/usr/local/lib/python2.7/dist-packages/glanceclient/common/http.py", line 265, in post
**15:48:22** return self._request('POST', url, **kwargs)
**15:48:22** File "/usr/local/lib/python2.7/dist-packages/glanceclient/common/http.py", line 230, in _request
**15:48:22** raise exc.from_response(resp, resp.text)
**15:48:22** glanceclient.exc.HTTPException: 504 Gateway Time-out: The server didn't respond in time. (HTTP N/A)
===========================

In order to pin point the problems, I propose that we split the cirros image creation task into two tasks:
1) Download the cirros image from a public repo
2) Upload the image to glance

Changed in openstack-ansible:
assignee: nobody → Hugh Saunders (hughsaunders)
status: New → In Progress
Revision history for this message
Hugh Saunders (hughsaunders) wrote :

This isn't as straightforward as I'd thought as the OSA glance library doesn't support uploading a local file, and adding that functionality is not made easier by the python glance client api.

Changed in openstack-ansible:
assignee: Hugh Saunders (hughsaunders) → nobody
status: In Progress → Triaged
importance: Undecided → Low
Changed in openstack-ansible:
assignee: nobody → Jimmy McCrory (jimmy-mccrory)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to openstack-ansible-os_magnum (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/396381

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to openstack-ansible-plugins (master)

Fix proposed to branch: master
Review: https://review.openstack.org/396445

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to openstack-ansible-os_tempest (master)

Reviewed: https://review.openstack.org/396330
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_tempest/commit/?id=d0fe1bd253e51843d691fcd9659919aad43b2ba3
Submitter: Jenkins
Branch: master

commit d0fe1bd253e51843d691fcd9659919aad43b2ba3
Author: Jimmy McCrory <email address hidden>
Date: Thu Nov 10 09:32:00 2016 -0800

    Use upstream openstack image module

    Replace the use of OSA's glance library with the upstream Ansible
    os_image module.

    Partial-Bug: 1538642
    Change-Id: Ic98613c37069ccb2ec1bc9a7644c922cc5b37e22

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to openstack-ansible-os_magnum (master)

Reviewed: https://review.openstack.org/396381
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_magnum/commit/?id=2e096ecd5478576f7c5f3f9f18fe5e392aeb6028
Submitter: Jenkins
Branch: master

commit 2e096ecd5478576f7c5f3f9f18fe5e392aeb6028
Author: Jimmy McCrory <email address hidden>
Date: Thu Nov 10 11:47:45 2016 -0800

    Use upstream openstack image module

    Replace the use of OSA's glance library with the upstream Ansible
    os_image module. This should be much more reliable for larger images,
    such as magnum's. Include the in-role task in testing and remove the
    additional testing tasks that were performing the image upload outside
    of the role.

    Related-Bug: 1538642
    Change-Id: I017303e9a731ebc60315ea1784ba184a8294b08e

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to openstack-ansible-plugins (master)

Reviewed: https://review.openstack.org/396445
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-plugins/commit/?id=0af898835a358a1f5315f50c889c992618aad86f
Submitter: Jenkins
Branch: master

commit 0af898835a358a1f5315f50c889c992618aad86f
Author: Jimmy McCrory <email address hidden>
Date: Thu Nov 10 16:16:06 2016 -0800

    Remove glance library

    With the use of this library replaced by upstream Ansible modules it's
    no longer necessary to carry it.

    Closes-Bug: 1538642

    Change-Id: I55591132180eec99ce9b9bc38caa996b01d18949
    Depends-On: Ic98613c37069ccb2ec1bc9a7644c922cc5b37e22
    Depends-On: I017303e9a731ebc60315ea1784ba184a8294b08e
    Depends-On: If0e36c61307056c8786c67b60fd56b03f2ad7bda

Changed in openstack-ansible:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/openstack-ansible-plugins 15.0.0.0b1

This issue was fixed in the openstack/openstack-ansible-plugins 15.0.0.0b1 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.