NFS based Nova Live Migration eratically fails
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Fix Released
|
Medium
|
Tom Patzig | ||
Newton |
Fix Committed
|
Medium
|
Matt Riedemann |
Bug Description
Hello,
in our productive Openstack environment we encountered in the last weeks that Openstack Nova VM Live migrations fails.
Currently this is only visible in our automated test environment. Every 15 minutes an automated test is started and it fails 3-4 times a day.
On the Nova instance path we have mounted a central NetApp NFS share to support real Live migrations between different hypervisors.
When we analysed the issue we found the error message and trace:
BadRequest: <Compute-Node> is not on shared storage: Live migration can not be used without shared storage except a booted from volume VM which does not have a local disk. (HTTP 400) (Request-ID: req-8e709fd1-
Traceback (most recent call last):
File "/usr/lib/
getattr(
File "/usr/lib/
block_
File "/usr/lib/
f = func(self, *args, **kwargs)
File "/usr/lib/
disk_
File "/usr/lib/
disk_
File "/usr/lib/
return methods[
File "/usr/lib/
'disk_
File "/usr/lib/
info=info, **kwargs)
File "/usr/lib/
return self.api.
File "/usr/lib/
return self._cs_
File "/usr/lib/
resp, body = self._time_
File "/usr/lib/
resp, body = self.request(url, method, **kwargs)
File "/usr/lib/
raise exceptions.
BadRequest: <Compute-Node> is not on shared storage: Live migration can not be used without shared storage except a booted from volume VM which does not have a local disk. (HTTP 400) (Request-ID: req-8e709fd1-
We examined the respective hypervisors for some problems with the NFS share/mount, but everything looks really good. Also the message log file shows no issues during the test timeframe.
The next step was to examine the Nova code to get a hint why Nova is bringing up such an error.
In the Nova code we found the test procedure how Nova checks if there is a shared filesystem between source and destination hypervisor.
In "nova/nova/
In function „check_
# Create file on storage, to be checked on source host
filename = self._create_
After that – in the same class - in function „check_
dest_check_
self.
will be checked if the temporary file exists. And this will sometimes fail and migration returns with this error message because the file on the source hypervisor is not yet available:
elif not (dest_check_
reason = _("Live migration can not be used "
"a booted from volume VM which "
Changed in nova: | |
assignee: | nobody → Tom Patzig (tom-patzig) |
Changed in nova: | |
assignee: | Tom Patzig (tom-patzig) → Matt Riedemann (mriedem) |
Changed in nova: | |
assignee: | Matt Riedemann (mriedem) → Tom Patzig (tom-patzig) |
Just to add, the setup is Liberty based.
And we observed that for this testfile exists check, it could take up to 30s, until the file created on the target is visible on the source HV.