Periodic tasks incurs IO performance penalty with LVM imagebackend

Bug #1130697 reported by Qiu Yu
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Undecided
Qiu Yu

Bug Description

Using LVM as image backend, periodic tasks (update_available_resource, _report_driver_status) can eat up to 10% I/O util rate, which could be a serious performance burden in production.

# iostat -x 1 /dev/sda
avg-cpu: %user %nice %system %iowait %steal %idle
           3.01 0.00 2.59 0.50 0.00 93.90

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 1.00 2931.00 9.00 23982.00 88.00 8.19 0.11 0.04 0.04 10.70

# dstat -tcd --top-io --top-cpu
----system---- ----total-cpu-usage---- -dsk/total- ----most-expensive---- -most-expensive-
  date/time |usr sys idl wai hiq siq| read writ| i/o process | cpu process
20-02 17:24:32| 3 3 94 0 0 0| 0 0 |SCREEN 2717k 8182B|nova-compute 0.2
20-02 17:24:33| 5 4 91 1 0 0| 11M 32k|nova-comput 15M 7661B|libvirtd 0.6
20-02 17:24:34| 9 4 87 0 0 0|4322k 4096B|nova-comput5333k 7772B|libvirtd 8.2
20-02 17:24:35| 10 3 88 0 0 0| 0 0 |libvirtd 12k 225k|libvirtd 8.3
20-02 17:24:36| 10 4 86 0 0 0| 0 0 |nova-comput 269k 15k|libvirtd 8.4
20-02 17:24:37| 8 3 88 0 0 0|3997k 8192B|nova-comput5583k 25k|libvirtd 2.2
20-02 17:24:38| 3 3 94 0 0 0|3997k 4096B|nova-comput5201k 6398B|nova-compute 0.2

Tags: performance
Qiu Yu (unicell)
Changed in nova:
assignee: nobody → unicell (unicell)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/22432

Changed in nova:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/22432
Committed: http://github.com/openstack/nova/commit/8f37626f72ce9865bb848170f46c29123014ba55
Submitter: Jenkins
Branch: master

commit 8f37626f72ce9865bb848170f46c29123014ba55
Author: unicell <email address hidden>
Date: Wed Feb 20 23:09:19 2013 +0800

    Improve I/O performance for periodic tasks

    Unfortunately, periodic tasks to update local resource and
    driver status come with performance penalties, especially on LVM
    image backend type.

    This is because low level implementation relies 'vgs' command to
    query LVM info, which itself has a I/O performance price and
    extravagant use of local_gb_xxx query make things even worse.

    This commit centralize such query in one single call
    (volume_group_info) and reduce invocation times in periodic tasks
    accordingly.

    Partially resolves LP #1130697

    Change-Id: I165f96dd3e4fdad770b8a9643f5a038d353312ff

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → grizzly-3
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: grizzly-3 → 2013.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.