[vCenter] Can't create snapshot (>2GB) with vSphere 6

[Ilya Bumarskov]

Fuel 9.0 mos-iso#495

Detailed bug description:
 Creation of snapshot stack in 'Image Uploading' state.

Steps to reproduce:
   - Create new env with vCenter 6.0
   - Add following nodes:
         * Controller
   - Deploy cluster
   - Create instance with 3 GB disk.
   - Try to create snapshot from instance.

Expected results:
  Snapshot was created successfully.

Actual result:
  Snapshot creation stacks in 'Image Uploading' state
  and after some time snapshot Killed. On vCenter side we see 'Export OVF template' task failed by timeout.

First look at this related issues:
https://bugs.launchpad.net/nova/+bug/1546454
https://bugs.launchpad.net/nova/+bug/1278690
https://bugs.launchpad.net/nova/+bug/1495429

Fix on review for bugs 1546454, 1278690, 1495429:
https://review.openstack.org/#/c/281134

Related issues:
https://bugs.launchpad.net/fuel/+bug/1529056
https://bugs.launchpad.net/glance/+bug/1498645

[Sheena Conant (sheena-conant)]

Given that this particular bug requires that the disk be completely full to manifest, Nathan and I agree that this can be a High priority bug with a comment in release notes.

[Gregory Elkinbard (gelkinbard)]

This was found at Reliance. And really is an issue for any non-trivial size snapshots.
This does need to be addressed, but the only work around available to Mirantis is to adjust the timeouts.

[Ilya Elterman (ielterman)]

Greg, please have your team figure out the right timeout setting and then have it release note with the recommendation to change the timeout.

[Alexander Arzhanov (aarzhanov)]

Update after investigate this issue:

1. Increasing the timeouts does not help fix the problem!

2. To reproduce this bug not needed generate a file to fill all disk space (steps to reproduce in bug description updated)

3. To resolve this problem need apply this patch https://review.openstack.org/#/c/281134 (Please note "First look at this related issues" in bug description). I tested the fix https://review.openstack.org/#/c/281134 on Fuel 9.0 mos-iso#495 and vCenter 6 and "Export OVF template task failed by timeout" not reproduced.

Timeouts, I've tried to change - https://kb.vmware.com/kb/2144686

[Alexander Arzhanov (aarzhanov)]

For release notes:

1. Not updating the NFC lease - this is addressed with https://review.openstack.org/#/c/281134
2. HTTP timeout when uploading the snapshotted image to Glance - this happens because the OVF exports is taking long time when the root disk is large and HTTP upload to the vSphere datastore times out. The workaround it to increase the HTTP read timeout in vCenter:

Modify the readTimeoutMs setting to 600000 in vpxd.cfg:

<config>
  <vmacore>
    <http>
      <readTimeoutMs>600000</readTimeoutMs>
    </http>
  </vmacore>
..
</config>
Stop and restart vCenter.

[Roman Podoliaka (rpodolyaka)]

https://review.openstack.org/#/c/281134 (complete refactoring of this part of vmware driver in Nova) is not going to be back ported to stable/mitaka, thus, we should restraint from cherry-picking it do 9.0/mitaka downstream branch, unless absolutely necessary.

IMO, the workaround described by Alexander in #6 and increasing of the NFC timeout value (https://kb.vmware.com/kb/2038497) should be sufficient for 9.x.

[Sergey Nikitin (snikitin)]

Marked as "In progress" because the patch is on review https://review.openstack.org/#/c/281134

[Sergey Nikitin (snikitin)]

fix is merged in newton

[Maria Zlatkova (mzlatkova)]

A release note has been added to 9.2 known issues: https://review.fuel-infra.org/#/c/29729.