live block migration of instance with vfat config drive fails
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
High
|
Lee Yarwood | |||
Ocata |
High
|
Matt Riedemann | |||
Pike |
High
|
Matt Riedemann | |||
Queens |
High
|
Matt Riedemann | |||
Ubuntu Cloud Archive |
High
|
Unassigned | |||
Ocata |
High
|
Unassigned | |||
Pike |
High
|
Unassigned | |||
Queens |
High
|
Unassigned | |||
Rocky |
High
|
Unassigned | |||
nova (Ubuntu) |
High
|
Unassigned | |||
Bionic |
High
|
Unassigned | |||
Cosmic |
High
|
Unassigned |
Bug Description
It seems like the following commit broke block live migrations of instances with vfat config drive -- https:/
The exception happening is:
May 10 17:01:48 cmp02 nova-compute[4443]: Traceback (most recent call last):
May 10 17:01:48 cmp02 nova-compute[4443]: File "/usr/lib/
May 10 17:01:48 cmp02 nova-compute[4443]: listener.cb(fileno)
May 10 17:01:48 cmp02 nova-compute[4443]: File "/usr/lib/
May 10 17:01:48 cmp02 nova-compute[4443]: result = function(*args, **kwargs)
May 10 17:01:48 cmp02 nova-compute[4443]: File "/usr/lib/
May 10 17:01:48 cmp02 nova-compute[4443]: return func(*args, **kwargs)
May 10 17:01:48 cmp02 nova-compute[4443]: File "/usr/lib/
May 10 17:01:48 cmp02 nova-compute[4443]: LOG.error("Live Migration failure: %s", e, instance=instance)
May 10 17:01:48 cmp02 nova-compute[4443]: File "/usr/lib/
May 10 17:01:48 cmp02 nova-compute[4443]: self.force_
May 10 17:01:48 cmp02 nova-compute[4443]: File "/usr/lib/
May 10 17:01:48 cmp02 nova-compute[4443]: six.reraise(
May 10 17:01:48 cmp02 nova-compute[4443]: File "/usr/lib/
May 10 17:01:48 cmp02 nova-compute[4443]: bandwidth=
May 10 17:01:48 cmp02 nova-compute[4443]: File "/usr/lib/
May 10 17:01:48 cmp02 nova-compute[4443]: destination, params=params, flags=flags)
May 10 17:01:48 cmp02 nova-compute[4443]: File "/usr/lib/
May 10 17:01:48 cmp02 nova-compute[4443]: result = proxy_call(
May 10 17:01:48 cmp02 nova-compute[4443]: File "/usr/lib/
May 10 17:01:48 cmp02 nova-compute[4443]: rv = execute(f, *args, **kwargs)
May 10 17:01:48 cmp02 nova-compute[4443]: File "/usr/lib/
May 10 17:01:48 cmp02 nova-compute[4443]: six.reraise(c, e, tb)
May 10 17:01:48 cmp02 nova-compute[4443]: File "/usr/lib/
May 10 17:01:48 cmp02 nova-compute[4443]: rv = meth(*args, **kwargs)
May 10 17:01:48 cmp02 nova-compute[4443]: File "/usr/lib/
May 10 17:01:48 cmp02 nova-compute[4443]: if ret == -1: raise libvirtError ('virDomainMigr
May 10 17:01:48 cmp02 nova-compute[4443]: libvirtError: operation failed: migration of disk vdb failed: Input/output error
May 10 17:01:48 cmp02 nova-compute[4443]: Removing descriptor: 29
(vdb is a vfat config drive)
after turning on the info logging in qemu seen the following in the syslog:
May 10 17:01:48 cmp02 libvirtd[10318]: 2018-05-10 17:01:48.204+0000: 10323: info : qemuMonitorSend
May 10 17:01:48 cmp02 libvirtd[10318]: fd=-1
May 10 17:01:48 cmp02 libvirtd[10318]: 2018-05-10 17:01:48.204+0000: 10318: info : qemuMonitorIOWr
May 10 17:01:48 cmp02 libvirtd[10318]: len=228 ret=228 errno=0
May 10 17:01:48 cmp02 libvirtd[10318]: 2018-05-10 17:01:48.296+0000: 10323: info : qemuMonitorSend
May 10 17:01:48 cmp02 libvirtd[10318]: fd=-1
May 10 17:01:48 cmp02 libvirtd[10318]: 2018-05-10 17:01:48.296+0000: 10318: info : qemuMonitorIOWr
May 10 17:01:48 cmp02 libvirtd[10318]: len=228 ret=228 errno=0
May 10 17:01:48 cmp02 libvirtd[10318]: 2018-05-10 17:01:48.388+0000: 10318: info : qemuMonitorJSON
May 10 17:01:48 cmp02 libvirtd[10318]: 2018-05-10 17:01:48.388+0000: 10318: info : qemuMonitorJSON
May 10 17:01:48 cmp02 libvirtd[10318]: 2018-05-10 17:01:48.392+0000: 10318: info : qemuMonitorJSON
May 10 17:01:48 cmp02 libvirtd[10318]: 2018-05-10 17:01:48.393+0000: 10318: info : qemuMonitorJSON
May 10 17:01:48 cmp02 libvirtd[10318]: 2018-05-10 17:01:48.395+0000: 10323: error : qemuMigrationDr
May 10 17:01:48 cmp02 libvirtd[10318]: 2018-05-10 17:01:48.395+0000: 10323: info : qemuMonitorSend
May 10 17:01:48 cmp02 libvirtd[10318]: fd=-1
May 10 17:01:48 cmp02 libvirtd[10318]: 2018-05-10 17:01:48.395+0000: 10318: info : qemuMonitorIOWr
May 10 17:01:48 cmp02 libvirtd[10318]: len=95 ret=95 errno=0
May 10 17:01:48 cmp02 libvirtd[10318]: 2018-05-10 17:01:48.398+0000: 10323: info : qemuMonitorSend
May 10 17:01:48 cmp02 libvirtd[10318]: fd=-1
May 10 17:01:48 cmp02 libvirtd[10318]: 2018-05-10 17:01:48.398+0000: 10318: info : qemuMonitorIOWr
May 10 17:01:48 cmp02 libvirtd[10318]: len=95 ret=95 errno=0
May 10 17:01:48 cmp02 libvirtd[10318]: 2018-05-10 17:01:48.402+0000: 10318: info : qemuMonitorJSON
May 10 17:01:48 cmp02 libvirtd[10318]: 2018-05-10 17:01:48.403+0000: 10318: info : qemuMonitorJSON
May 10 17:01:48 cmp02 libvirtd[10318]: 2018-05-10 17:01:48.414+0000: 10318: info : qemuMonitorJSON
May 10 17:01:48 cmp02 libvirtd[10318]: "wr_operations": 0, "rd_merged": 0, "rd_bytes": 0, "invalid_
May 10 17:01:48 cmp02 libvirtd[10318]: 2018-05-10 17:01:48.823+0000: 10323: error : virNetClientPro
May 10 17:01:48 cmp02 libvirtd[10318]: 2018-05-10 17:01:48.845+0000: 10318: error : virNetSocketRea
It seems like qemu's drive mirror expects the full disk size to be present on dest compute, not only what is allocated.
Revert of the patch solved the issue.
description: | updated |
summary: |
- migration of instance with vfat config drive fails + live block migration of instance with vfat config drive fails |
Changed in nova: | |
assignee: | nobody → Vladyslav Drok (vdrok) |
status: | New → In Progress |
Changed in nova: | |
importance: | Undecided → High |
tags: | added: libvirt live-migration |
Related fix proposed to branch: master
Review: https:/
Changed in nova: | |
assignee: | Vladyslav Drok (vdrok) → Lee Yarwood (lyarwood) |
Change abandoned by Vladyslav Drok (<email address hidden>) on branch: master
Review: https:/
Fix proposed to branch: stable/queens
Review: https:/
Fix proposed to branch: stable/pike
Review: https:/
Fix proposed to branch: stable/ocata
Review: https:/
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: master
commit 016986f4706e881
Author: Lee Yarwood <email address hidden>
Date: Fri May 11 15:04:48 2018 +0100
libvirt: Report the virtual size of RAW disks
If642e51a4e
the actual size of preallocated file based disks but missed that this
value was later used as the virtual disk size for RAW disks.
This is an issue as nova.virt.
disks as sparse files with a reported actual size much smaller than the
virtual size. During block based LM this then results in disks
on the destination being created with a much smaller virtual size than
the disk should have leading to errors during the transfer.
Closes-Bug: #1770640
Change-Id: I464bc2b88123a0
Changed in nova: | |
status: | In Progress → Fix Released |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/queens
commit 2dc9795a0df40c5
Author: Lee Yarwood <email address hidden>
Date: Fri May 11 15:04:48 2018 +0100
libvirt: Report the virtual size of RAW disks
If642e51a4e
the actual size of preallocated file based disks but missed that this
value was later used as the virtual disk size for RAW disks.
This is an issue as nova.virt.
disks as sparse files with a reported actual size much smaller than the
virtual size. During block based LM this then results in disks
on the destination being created with a much smaller virtual size than
the disk should have leading to errors during the transfer.
Closes-Bug: #1770640
Change-Id: I464bc2b88123a0
(cherry picked from commit 016986f4706e881
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/pike
commit 938c0a745325fa7
Author: Lee Yarwood <email address hidden>
Date: Fri May 11 15:04:48 2018 +0100
libvirt: Report the virtual size of RAW disks
If642e51a4e
the actual size of preallocated file based disks but missed that this
value was later used as the virtual disk size for RAW disks.
This is an issue as nova.virt.
disks as sparse files with a reported actual size much smaller than the
virtual size. During block based LM this then results in disks
on the destination being created with a much smaller virtual size than
the disk should have leading to errors during the transfer.
Conflicts:
NOTE(mriedem): The trivial test conflict is due to not having change
I11e329ac5f
Closes-Bug: #1770640
Change-Id: I464bc2b88123a0
(cherry picked from commit 016986f4706e881
(cherry picked from commit 2dc9795a0df40c5
This issue was fixed in the openstack/nova 17.0.5 release.
This issue was fixed in the openstack/nova 16.1.4 release.
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/ocata
commit 34f4167974868f8
Author: Lee Yarwood <email address hidden>
Date: Fri May 11 15:04:48 2018 +0100
libvirt: Report the virtual size of RAW disks
If642e51a4e
the actual size of preallocated file based disks but missed that this
value was later used as the virtual disk size for RAW disks.
This is an issue as nova.virt.
disks as sparse files with a reported actual size much smaller than the
virtual size. During block based LM this then results in disks
on the destination being created with a much smaller virtual size than
the disk should have leading to errors during the transfer.
Conflicts:
NOTE(mriedem): The conflict is due to not having change
I9616a602ee
Closes-Bug: #1770640
Change-Id: I464bc2b88123a0
(cherry picked from commit 016986f4706e881
(cherry picked from commit 2dc9795a0df40c5
(cherry picked from commit 938c0a745325fa7
This issue was fixed in the openstack/nova 18.0.0.0b2 development milestone.
This issue was fixed in the openstack/nova 15.1.3 release.
Dr. Jens Harbott (j-harbott) wrote : | #15 |
FTR, this does not only affect the config drive for me, but migration of the root disk with local storage, too. (Using images_type = raw in case that matters.) Found on 16.1.2, applying https:/
Changed in nova (Ubuntu Bionic): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in nova (Ubuntu Cosmic): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in nova (Ubuntu Cosmic): | |
status: | Triaged → Fix Released |
Corey Bryant (corey.bryant) wrote : | #16 |
In Ubuntu queens, this fix is included in the 17.0.5 stable point release via https:/
Corey Bryant (corey.bryant) wrote : | #17 |
In Ubuntu pike, this fix is included in the 16.1.4 stable point release via https:/
Changed in nova (Ubuntu Bionic): | |
status: | Triaged → Fix Committed |
Corey Bryant (corey.bryant) wrote : | #18 |
In Ubuntu ocata, this fix is included in the 15.1.3 stable point release via https:/
This issue was fixed in the openstack/nova 16.1.5 release.
Corey Bryant (corey.bryant) wrote : | #20 |
This has been fix released for ocata, pike, queens (and bionic) in the above mentioned stable point releases.
Changed in nova (Ubuntu Bionic): | |
status: | Fix Committed → Fix Released |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: master
commit 00d9a3a2e6b4fbb
Author: Matt Riedemann <email address hidden>
Date: Fri May 11 09:36:10 2018 -0400
Test live migration with config drive
Tempest live migration tests do not create the server
with config drive so we do not, by default, have test
coverage of live migrating an instance with a config
drive. This change forces nova to create a config
drive for all instances in the nova-live-migration
job which will also run evacuate tests with a config
drive attached.
Change-Id: I8157b9e6e9da49
Related-Bug: #1770640
Fix proposed to branch: master /review. openstack. org/567858
Review: https:/