When moving VHDs on the filesystem a coalesce may be in progress. The result of this is that the VHD file is not valid when it is copied as it is being actively changed - and the VHD cookie is invalid.
Seen in XenServer CI: http://dd6b71949550285df7dc-dda4e480e005aaa13ec303551d2d8155.r49.cf1.rackcdn.com/36/109836/4/23874/run_tests.log
2014-08-28 12:26:37.538 | Traceback (most recent call last):
2014-08-28 12:26:37.543 | File "tempest/api/compute/servers/test_server_actions.py", line 251, in test_resize_server_revert
2014-08-28 12:26:37.550 | self.client.wait_for_server_status(self.server_id, 'VERIFY_RESIZE')
2014-08-28 12:26:37.556 | File "tempest/services/compute/json/servers_client.py", line 179, in wait_for_server_status
2014-08-28 12:26:37.563 | raise_on_error=raise_on_error)
2014-08-28 12:26:37.570 | File "tempest/common/waiters.py", line 77, in wait_for_server_status
2014-08-28 12:26:37.577 | server_id=server_id)
2014-08-28 12:26:37.583 | BuildErrorException: Server e58677ac-dd72-4f10-9615-cb6763f34f50 failed to build and is in ERROR status
2014-08-28 12:26:37.589 | Details: {u'message': u'[\'XENAPI_PLUGIN_FAILURE\', \'move_vhds_into_sr\', \'Exception\', "VDI \'/var/run/sr-mount/16f5c980-eeb6-0fd3-e9b1-dec616309984/os-images/instancee58677ac-dd72-4f10-9615-cb6763f34f50/535cd7f2-80a5-463a-935c-9c4f52ba0ecf.vhd\' has an invalid footer: \' invalid cook', u'code': 500, u'created': u'2014-08-28T11:57:01Z'}
I think the easiest fix here is to repair the VHDs on import.
My current theory is that because 'wait_for_coalesce' assumes (and has always assumed) that a single coalesce is going to happen (which is not necessarily correct) we might be trying to copy the VHDs while a coalesce is in progress.
Coalesce of the chain a-b-c (where c is the leaf) happens by:
1) Copying the blocks changed in b into a (to give a'-b-c)
2) Re-parenting C to a' (a'-c b)
3) Deleting b.
During step 1, the size of the VHD is extended, the new blocks written, and an updated footer put at the end of the extended VHD. If a file-level copy of the VHD is made after it has been extended but before the new footer is written the footer will be invalid. For this reason Citrix XenServer is looking at moving towards ignoring the footer and only using a 'backup footer' which is actually at the head of the VHD. This change is likely to be too invasive to be considered for a hotfix.
It seems that this can be repaired with the (very cheap) vhd-util repair option.
This may have been exasperated by https:/
The behaviour of bug 1317792 was as follows:
1) The chain a-b-c was imported
2) c was snapshotted giving a-b-c-d; we waited for 'd' to coalesce back into 'b'
3) b was coalesced into a, giving a-c-d
4) c was coalesced into a, giving a-d
5) 'wait_for_coalesce' failed with a timeout
The fix for this issue was to wait for 'd' to coalesce back into anything other than 'c'; in this case 'a' or 'b'. As such, we might stop waiting at step 3 meaning the copy happens while 'c' was being coalesced.
Even without this fix, the above scenario could have occurred if the GC decided to coalesce 'c' first then the copy happened while 'b' was being coalesced.
Copying the VHDs in this state and fixing them up afterwards is, in my view, preferable to reverting to the previous behaviour.
In terms of moving forward without breaking bug 1317792 again, I think the following are options:
1) Use vhd-util repair to fix up the VHDs after the fact. As described above, the VHDs will still be valid as b is not removed from the chain until c is re-parented to a'. As such, any 'incorrect' data in a' will not be read because it is guaranteed that b contains the correct data.
2) Changing wait_for_coalesce to wait for _all_coalescing to be complete, based on XenServer's understanding of whether the GC is still running (would need a XAPI plugin to poll the GC to make sure it's not running at the point we copy)
3) Adding a XAPI plugin to manually lock the SR or VDI (using /opt/xensource/
4) (least preferred) add more logic to Nova to guess when it thinks GC will be able to coalesce or not. We currently have some logic that looks for siblings but if we were to follow options 2 or 3 then we can probably delete ...