cinder gets out of memory at json.dumps when doing a backup of a large volume

Bug #1965799 reported by David Caro
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
In Progress
Low
Unassigned

Bug Description

We are running cinder 17.2.0-1 on debian, using posix backend, and when trying to do a backup of a huge volume we always hit the error:
```
                                                      2022-03-05 08:36:23.108 18548 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/backup/chunkeddriver.py", line 289, in _write_sha256file
                                                      2022-03-05 08:36:23.108 18548 ERROR oslo_messaging.rpc.server sha256file_json = json.dumps(sha256file, sort_keys=True, indent=2)
                                                      2022-03-05 08:36:23.108 18548 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.7/json/__init__.py", line 238, in dumps
                                                      2022-03-05 08:36:23.108 18548 ERROR oslo_messaging.rpc.server **kw).encode(obj)
                                                      2022-03-05 08:36:23.108 18548 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.7/json/encoder.py", line 202, in encode
                                                      2022-03-05 08:36:23.108 18548 ERROR oslo_messaging.rpc.server return ''.join(chunks)
                                                      2022-03-05 08:36:23.108 18548 ERROR oslo_messaging.rpc.server MemoryE
```

As there's a spike of memory usage right there (join of the chunks duplicates it).

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/834527

Changed in cinder:
status: New → In Progress
Changed in cinder:
importance: Undecided → Low
tags: added: backup-service memory
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.