[NetApp NFS] Cinder downloads image from Glance twice

Bug #1632333 reported by Serg Lystopad
28
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
Undecided
Goutham Pacha Ravi
Mirantis OpenStack
Won't Fix
Medium
Ivan Kolodyazhny
10.0.x
Won't Fix
Medium
MOS Cinder
8.0.x
Won't Fix
Medium
MOS Cinder

Bug Description

Detailed bug description:
Both Cinder and Glance use NFS as backend. For such scenarios Cinder supports 'NetApp NFS Copy Offload client'(http://docs.openstack.org/juno/config-reference/content/ontap-cluster-nfs.html) to copy images from Glance to Cinder using NFS (unlike regular HTTP downloading from Glance API).
The issue is when offloading tool is configured Cinder downloads image from Glance via HTTP before using offloading tool. So effectively it does the job twice: first it downloads image from Glance using HTTP (converts it ....) and then it downloads image second time using offloading tool.

Steps to reproduce:
Configure NetApp NFS backend for Cinder, specify 2 (or more) NFS shares for NFS backend.
Configure NFS backend for Glance (file backend pointing to NFS mount point).
Create volume `cinder create --image 59cd2153-f892-41e0-b8fa-3bd9cdde16af 40`
Check cinder-volume log:
2016-10-11 11:26:41.819 17653 DEBUG glanceclient.common.http [req-f987f983-ee2e-4886-9f35-945ee9f554b9 8555d82365964315840708d6efdafeae 0ff102a3698842fa89dacf049cf84e40
- - -] curl -g -i -X GET -H 'Accept-Encoding: gzip, deflate' -H 'Accept: */*' -H 'User-Agent: python-glanceclient' -H 'Connection: keep-alive' -H 'X-Auth-Token: {SHA1}52
5561ab8f9c94662aced0187f7b58ce29dc0fa5' -H 'Content-Type: application/octet-stream' http://172.18.54.101:9292/v2/images/59cd2153-f892-41e0-b8fa-3bd9cdde16af/file log_cur
l_request /usr/lib/python2.7/dist-packages/glanceclient/common/http.py:185
2016-10-11 11:26:41.931 17653 DEBUG glanceclient.common.http [req-f987f983-ee2e-4886-9f35-945ee9f554b9 8555d82365964315840708d6efdafeae 0ff102a3698842fa89dacf049cf84e40
- - -]
HTTP/1.1 200 OK
content-length: 8823242752
content-md5: e7f6e7d7d38423a705394ad72fdb823c
connection: close
date: Tue, 11 Oct 2016 11:26:41 GMT
content-type: application/octet-stream
x-openstack-request-id: req-1fe0c5cb-1467-4fb8-bfc7-ee91e278a252
 log_http_response /usr/lib/python2.7/dist-packages/glanceclient/common/http.py:199
...
2016-10-11 11:28:32.317 17653 DEBUG cinder.image.image_utils [req-f987f983-ee2e-4886-9f35-945ee9f554b9 8555d82365964315840708d6efdafeae 0ff102a3698842fa89dacf049cf84e40 - - -] Image fetch details: dest /var/lib/cinder/conversion/tmpUgLYML, size 8414.00 MB, duration 110.50 sec fetch /usr/lib/python2.7/dist-packages/cinder/image/image_utils.py:177

and later

2016-10-11 11:28:32.872 17653 DEBUG cinder.volume.flows.manager.create_volume [req-c4980bb2-c286-4739-b0bc-9b87770e225f - - - - -] Attempting download of 59cd2153-f892-41e0-b8fa-3bd9cdde16af ((u'file:///mnt/glance/images/59cd2153-f892-41e0-b8fa-3bd9cdde16af', [{u'url': u'file:///mnt/glance/images/59cd2153-f892-41e0-b8fa-3bd9cdde16af', u'metadata': {u'mountpoint': u'/mnt/glance/images', u'type': u'nfs', u'id': u'NetAppNFS', u'share_location': u'nfs://172.18.52.240/vol_glance/images'}}])) to volume 915ec5e6-73f8-44c5-975c-b5bb4652db96. _copy_image_to_volume /usr/lib/python2.7/dist-packages/cinder/volume/flows/manager/create_volume.py:517
2016-10-11 11:28:32.875 17653 DEBUG cinder.volume.drivers.netapp.dataontap.nfs_cmode [req-c4980bb2-c286-4739-b0bc-9b87770e225f - - - - -] No result found in image cache copy_image_to_volume /usr/lib/python2.7/dist-packages/cinder/volume/drivers/netapp/dataontap/nfs_cmode.py:426
2016-10-11 11:28:32.875 17653 DEBUG cinder.volume.drivers.netapp.dataontap.nfs_cmode [req-c4980bb2-c286-4739-b0bc-9b87770e225f - - - - -] Trying copy from image service using copy offload. _copy_from_img_service /usr/lib/python2.7/dist-packages/cinder/volume/drivers/netapp/dataontap/nfs_cmode.py:502

Example log file https://drive.google.com/open?id=0BzqvkqZNKRGdYWQ1eVhFNkpObGs

Expected results:
Cinder downloads image with ID 59cd2153-f892-41e0-b8fa-3bd9cdde16af dirrectly (via NFS using offloading tool)

Actual result:
Although 'offloading tool' is configured for Cinder and Glance backend is NFS, Cinder downloads image from Glance using HTTP client before using 'offloading tool' (so actually it does the job twice)

Description of the environment
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "570"
  build_id: "570"
  fuel-nailgun_sha: "558ca91a854cf29e395940c232911ffb851899c1"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "c2a335b5b725f1b994f78d4c78723d29fa44685a"
  fuel-ostf_sha: "3bc76a63a9e7d195ff34eadc29552f4235fa6c52"
  fuel-mirror_sha: "fb45b80d7bee5899d931f926e5c9512e2b442749"
  fuelmenu_sha: "78ffc73065a9674b707c081d128cb7eea611474f"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "d605bcbabf315382d56d0ce8143458be67c53434"

description: updated
Ivan Kolodyazhny (e0ne)
Changed in mos:
assignee: nobody → Ivan Kolodyazhny (e0ne)
Revision history for this message
Ivan Kolodyazhny (e0ne) wrote :
Changed in mos:
milestone: none → 8.0-mu-4
Revision history for this message
Goutham Pacha Ravi (gouthamr) wrote :

Going through the changes that were made in Liberty, I fail to understand why the image is being downloaded by Cinder before calling the interface _copy_image_to_volume(). The only use case seems to be to determine the virtual size, which is only used in case the image volume cache is being used. The NetApp NFS driver had its own caching logic since Grizzly.

https://github.com/openstack/cinder/blob/15c13f8/cinder/volume/flows/manager/create_volume.py#L736

tags: added: area-cinder
Revision history for this message
Bishoy (bishoysamy) wrote :

Could you try:

cinder type-create nfs
cinder type-key nfs set storage_protocol=nfs

then create volumes with type NFS.

Revision history for this message
Serg Lystopad (slystopad) wrote :

I've tied to create volume with type below:
# cinder extra-specs-list | grep nfs
| 0807e887-9ef5-4444-b21e-ddfbe5e0e656 | nfs | {u'volume_backend_name': u'cinder_netapp', u'storage_protocol': u'nfs'} |

It doesn't help to avoid downloading glance image via HTTP.

Revision history for this message
Bishoy (bishoysamy) wrote :

looking at this, looks bad.
duration 110.50 sec fetch /usr/lib/python2.7/dist-packages/cinder/image/image_utils.py:177

I will ping NetApp Engineering regarding this,

Thanks

Changed in cinder:
assignee: nobody → Goutham Pacha Ravi (gouthamr)
Bishoy (bishoysamy)
Changed in cinder:
status: New → Confirmed
summary: - [netapp] Cinder downloads image from Glance twice
+ Cinder downloads image from Glance twice
tags: added: netapp
tags: added: ct1
Ivan Kolodyazhny (e0ne)
summary: - Cinder downloads image from Glance twice
+ [NetApp NFS] Cinder downloads image from Glance twice
Revision history for this message
Ivan Kolodyazhny (e0ne) wrote :

Customer applied workaround for 9.x.

Revision history for this message
Bishoy (bishoysamy) wrote :

@Ivan,

What is the workaround for this?

Revision history for this message
Ivan Kolodyazhny (e0ne) wrote :

Bishoy, there is proposed fix from NetApp team for it: https://review.openstack.org/#/c/409902/

Changed in mos:
status: Confirmed → Won't Fix
no longer affects: mos/9.x
Changed in mos:
milestone: 9.2-mu-1 → 9.x-updates
Revision history for this message
Sean McGinnis (sean-mcginnis) wrote : Bug Assignee Expired

Unassigning due to no activity for > 6 months.

Changed in cinder:
assignee: Goutham Pacha Ravi (gouthamr) → nobody
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/527762

Changed in cinder:
assignee: nobody → Adriano (rosso-af)
status: Confirmed → In Progress
Changed in cinder:
assignee: Adriano (rosso-af) → Goutham Pacha Ravi (gouthamr)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.openstack.org/527762
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=c27173bad69da4889a5237cf2becc14bb6fc578a
Submitter: Zuul
Branch: master

commit c27173bad69da4889a5237cf2becc14bb6fc578a
Author: Adriano Rosso <email address hidden>
Date: Wed Nov 8 15:51:36 2017 -0200

    NetApp ONTAP: Copy offload bugfix

    When 'NetApp NFS Copy Offload' tool is configured to download Glance
    images, these images are downloaded twice because the tool is doing
    the job after Cinder has already done it.
    This patch fixes the bug by executing the copy offload tool inside the
    clone_image function instead of using the copy_image_to_volume.

    Closes-bug: #1632333

    Change-Id: I5c6ad150543213acfd0c78dbbdb1dc1584d22d26

Changed in cinder:
status: In Progress → Fix Released
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/cinder (mcp/1.0/mitaka)

Fix proposed to branch: mcp/1.0/mitaka
Change author: Adriano Rosso <email address hidden>
Review: https://review.fuel-infra.org/37484

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 12.0.0.0b3

This issue was fixed in the openstack/cinder 12.0.0.0b3 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/542770

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (driverfixes/newton)

Fix proposed to branch: driverfixes/newton
Review: https://review.openstack.org/542771

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (driverfixes/ocata)

Fix proposed to branch: driverfixes/ocata
Review: https://review.openstack.org/542773

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (driverfixes/mitaka)

Fix proposed to branch: driverfixes/mitaka
Review: https://review.openstack.org/542775

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on cinder (driverfixes/mitaka)

Change abandoned by Erlon R. Cruz (<email address hidden>) on branch: driverfixes/mitaka
Review: https://review.openstack.org/542775
Reason: Wont backport top Mitaka. We dont have how to test this.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/pike)

Reviewed: https://review.openstack.org/542770
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=82a13da48e7451c2f7813bf2de6990625c05624c
Submitter: Zuul
Branch: stable/pike

commit 82a13da48e7451c2f7813bf2de6990625c05624c
Author: Adriano Rosso <email address hidden>
Date: Wed Nov 8 15:51:36 2017 -0200

    NetApp ONTAP: Copy offload bugfix

    When 'NetApp NFS Copy Offload' tool is configured to download Glance
    images, these images are downloaded twice because the tool is doing
    the job after Cinder has already done it.
    This patch fixes the bug by executing the copy offload tool inside the
    clone_image function instead of using the copy_image_to_volume.

    Closes-bug: #1632333

    Change-Id: I5c6ad150543213acfd0c78dbbdb1dc1584d22d26
    (cherry-picked from commit c27173bad69da4889a5237cf2becc14bb6fc578a)

tags: added: in-stable-pike
tags: added: in-driverfixes-ocata
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (driverfixes/ocata)

Reviewed: https://review.openstack.org/542773
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=d090efe57bc2988e1573f4c36c86d1faea03fda3
Submitter: Zuul
Branch: driverfixes/ocata

commit d090efe57bc2988e1573f4c36c86d1faea03fda3
Author: Adriano Rosso <email address hidden>
Date: Wed Nov 8 15:51:36 2017 -0200

    NetApp ONTAP: Copy offload bugfix

    When 'NetApp NFS Copy Offload' tool is configured to download Glance
    images, these images are downloaded twice because the tool is doing
    the job after Cinder has already done it.
    This patch fixes the bug by executing the copy offload tool inside the
    clone_image function instead of using the copy_image_to_volume.

    Closes-bug: #1632333
    Depends-on: https://review.openstack.org/542770

    Change-Id: I5c6ad150543213acfd0c78dbbdb1dc1584d22d26
    (cherry-picked from commit c27173bad69da4889a5237cf2becc14bb6fc578a)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (driverfixes/newton)

Reviewed: https://review.openstack.org/542771
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=5840e0e654e4c87e3d673b2a25c199e84a4af80c
Submitter: Zuul
Branch: driverfixes/newton

commit 5840e0e654e4c87e3d673b2a25c199e84a4af80c
Author: Adriano Rosso <email address hidden>
Date: Wed Nov 8 15:51:36 2017 -0200

    NetApp ONTAP: Copy offload bugfix

    When 'NetApp NFS Copy Offload' tool is configured to download Glance
    images, these images are downloaded twice because the tool is doing
    the job after Cinder has already done it.
    This patch fixes the bug by executing the copy offload tool inside the
    clone_image function instead of using the copy_image_to_volume.

    Closes-bug: #1632333
    Depends-on: https://review.openstack.org/542773

    Change-Id: I5c6ad150543213acfd0c78dbbdb1dc1584d22d26
    (cherry-picked from commit c27173bad69da4889a5237cf2becc14bb6fc578a)

tags: added: in-driverfixes-newton
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/560616

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/ocata)

Reviewed: https://review.openstack.org/560616
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=6d59c490c262d7634af5b2e03149c9f028f4d81c
Submitter: Zuul
Branch: stable/ocata

commit 6d59c490c262d7634af5b2e03149c9f028f4d81c
Author: Adriano Rosso <email address hidden>
Date: Wed Nov 8 15:51:36 2017 -0200

    NetApp ONTAP: Copy offload bugfix

    When 'NetApp NFS Copy Offload' tool is configured to download Glance
    images, these images are downloaded twice because the tool is doing
    the job after Cinder has already done it.
    This patch fixes the bug by executing the copy offload tool inside the
    clone_image function instead of using the copy_image_to_volume.

    Closes-bug: #1632333
    Depends-on: https://review.openstack.org/542770

    Change-Id: I5c6ad150543213acfd0c78dbbdb1dc1584d22d26
    (cherry-picked from commit c27173bad69da4889a5237cf2becc14bb6fc578a)

tags: added: in-stable-ocata
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 10.0.7

This issue was fixed in the openstack/cinder 10.0.7 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 11.1.1

This issue was fixed in the openstack/cinder 11.1.1 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.