During a conversion process cinder can overfill a controller root file system

Bug #1651411 reported by Anatolii Neliubin
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Status tracked in 10.0.x
10.0.x
In Progress
High
Ivan Kolodyazhny

Bug Description

Detailed bug description:
When I create a new volume from a very large QCOW image (virtual size) the root file system of a controller might be overfilled because the cinder downloads the full size image to the /var/lib/cinder/conversion/ directory
Steps to reproduce:
Install one controller and two ceph/compute nodes. The controller has to have the minimal possible size of a root filesystem (20GB). Ceph size should be enough to keep about 30GB. Create a QCOW2 image with large virtual size and upload it to Glance:
root@node-2:/var/lib/nova/tmp# qemu-img info ubuntu_qcow_20GB.qcow2
image: ubuntu_qcow_20GB.qcow2
file format: qcow2
virtual size: 20G (21474836480 bytes)
disk size: 835M
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

Then try to create a new volume from this image:
root@node-1:~# cinder create --image-id 4a5c016b-dff0-4e52-b640-bd63528911a6 --name volume_from_image_20GB 20
During the conversion Cinder will download the whole image to the /var/lib/cinder/conversion/ directory and it might overfill the root directory of a controller node
Expected results:
Root filesystem should not be overfilled
Reproducibility:
Fuel 9.1
Workaround:
1. To keep images in Raw format in Ceph so that Cinder does not download them but perform snapshots in Ceph storage
2. To forcefully separate /var from the root file system
Impact:
One of controllers might stop because its root file system is overfilled
Description of the environment:
MOS 9.1

Changed in mos:
importance: Undecided → High
milestone: none → 9.3
Changed in mos:
milestone: 9.3 → 9.2
assignee: nobody → MOS Cinder (mos-cinder)
status: New → Confirmed
tags: added: area-cinder
Ivan Kolodyazhny (e0ne)
Changed in mos:
assignee: MOS Cinder (mos-cinder) → Ivan Kolodyazhny (e0ne)
Revision history for this message
Ivan Kolodyazhny (e0ne) wrote :
Revision history for this message
Anatolii Neliubin (aneliubin) wrote :

A workaround solution is used until a fix is available.

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/cinder (9.0/mitaka)

Fix proposed to branch: 9.0/mitaka
Change author: Ivan Kolodyazhny <email address hidden>
Review: https://review.fuel-infra.org/29661

Ivan Kolodyazhny (e0ne)
Changed in mos:
assignee: Ivan Kolodyazhny (e0ne) → Maksym Shalamov (mshalamov)
Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

@Ivan, why the bug was assigned to QA engineer?

It should be assigned to the engineer who proposed the fix.

Changed in mos:
assignee: Maksym Shalamov (mshalamov) → Ivan Kolodyazhny (e0ne)
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to openstack/cinder (9.0/mitaka)

Reviewed: https://review.fuel-infra.org/29661
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0/mitaka

Commit: 318f424bb831e98f6af0257feced139d303bee9f
Author: Ivan Kolodyazhny <email address hidden>
Date: Thu Jan 5 14:46:20 2017

Check if volume node has enough space for image operations

Add check if free space is more than image size before downloading and
convertion.

Change-Id: I66f9e50cc8b7135d58822e13f62dea91ccb37c23
Closes-Bug: #1651411

Changed in mos:
status: Confirmed → Fix Committed
tags: added: on-verification
Revision history for this message
Ekaterina Shutova (eshutova) wrote :

Verified on 9.2 snapshot #746.

tags: removed: on-verification
Changed in mos:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.