When restoring many backups, for example 41, the backup service uses a lot of memory and end up getting killed becase we run out of memory.
The amount of memory required when the volumes are RBD, and the backups are not, is even worse.
Here's the explanation of the peak memory we will need for 1 restore:
- First we use as much as the size of the chunk we have stored, which in our case is compressed, so if we assume a 50% compression ratio, it would be 50% of the original chunk which can be 1999994880 bytes.
So here we use 0.93GB when the ChunkedBackupDriver reads the object [1]
with self._get_object_reader(
container, object_name,
extra_metadata=extra_metadata) as reader:
body = reader.read()
- Then when we decompress the data [2] we will need an additional 1,86GB which is the full original chunk size:
decompressed = decompressor.decompress(body)
- Then when the ChunkedBackupDriver writes the data [3]:
volume_file.write(decompressed)
What happens behind that call, because this is an RBD volume, is that we use the osb-brick connector which uses librbd Image object to do the writing [4]:
def write(self, data):
self._rbd_volume.image.write(data, self._offset)
The write method in librbd calls the rbd_write2 method [5]
ret = rbd_write2(self.image, _offset, length, _data, _fadvise_flags)
And the method will call the create_write_raw method with nullptr as the aio_completion parameter [6]:
bl.push_back(create_write_raw(ictx, buf, len, nullptr));
And because of this the create_write_raw method will copy the data in a different buffer [7]:
if (ictx->disable_zero_copy || aio_completion == nullptr) {
// must copy the buffer if writeback/writearound cache is in-use (or using
// non-AIO)
return buffer::copy(buf, len);
And we end up using another 1.86GB of RAM
So we end up needing 0.93GB + 1.86GB + 1.86GB for a single restore operation. If we now do 41 simultaneous operations, we will end up using 190.65GB, hence the OOM kill.
I see 2 improvements that can be done in the Cinder code:
- Help Python free memory faster by setting to None the body variable as soon as we decompress it, and setting to None the decompressed variable as soon as we've written it.
- Introduce a max concurrent backup & restore operations feature that will queue operations that exceed them.
To mitigate the problem, deployments can do any of:
- Reduce the number of concurrent restore operations
- Disable compression
- Reduce the size of the chunks with the backup_file_size variable
[1]: https://opendev.org/openstack/cinder/src/commit/a154a1360be62eed0e2bf20937503b55659f4701/cinder/backup/chunkeddriver.py#L712
[2]: https://opendev.org/openstack/cinder/src/commit/a154a1360be62eed0e2bf20937503b55659f4701/cinder/backup/chunkeddriver.py#L719
[3]: https://opendev.org/openstack/cinder/src/commit/a154a1360be62eed0e2bf20937503b55659f4701/cinder/backup/chunkeddriver.py#L720
[4]: https://opendev.org/openstack/os-brick/src/commit/49d5616f86d637c846d54cd48c5ed4e17bd6695e/os_brick/initiator/linuxrbd.py#L195
[5]: https://github.com/ceph/ceph/blob/53febd478dfc7282f0948853c117061d96cda9b1/src/pybind/rbd/rbd.pyx#L4321
[6]: https://github.com/ceph/ceph/blob/b2e825debc4d47cede8df86b96af94893241ddf7/src/librbd/librbd.cc#L5826
[7]: https://github.com/ceph/ceph/blob/b2e825debc4d47cede8df86b96af94893241ddf7/src/librbd/librbd.cc#L91-L94
Related fix proposed to branch: master
Review: https:/