OpenStack Compute (nova)

nova-lvm appears to be running out of swift storage space

Bug #1913451 reported by Lee Yarwood
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Lee Yarwood

Bug Description

Description
===========

nova-lvm is often failing due to failures creating snapshots of instances during tempest runs:

https://zuul.opendev.org/t/openstack/builds?job_name=nova-lvm&branch=master

I *think* swift that is being used by g-api as the backing store is running out of space due to the nova-lvm job creating two file backed 24G volume groups, one for n-cpu and another for c-vol:

https://zuul.opendev.org/t/openstack/build/4f259f6d6cd444c2acea08a6e409a7f9/log/controller/logs/screen-n-cpu.txt#37491

Jan 27 11:34:04.214271 ubuntu-focal-rax-ord-0022741499 nova-compute[90693]: ERROR oslo_messaging.rpc.server glanceclient.exc.HTTPBadGateway: HTTP 502 Bad Gateway: Bad Gateway: The proxy server received an invalid: response from an upstream server.: Apache/2.4.41 (Ubuntu) Server at 10.209.161.68 Port 80

https://zuul.opendev.org/t/openstack/build/4f259f6d6cd444c2acea08a6e409a7f9/log/controller/logs/screen-g-api.txt

Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 <email address hidden>[93210]: ERROR glance.common.wsgi [None req-a3f9e82f-5a4c-40fa-a102-7a1a3a7998a6 tempest-ServerActionsTestJSON-1412104246 tempest-ServerActionsTestJSON-1412104246-project] Caught error: Failed to add object to Swift.
Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 <email address hidden>[93210]: ERROR glance.common.wsgi Traceback (most recent call last):
Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 <email address hidden>[93210]: ERROR glance.common.wsgi File "/usr/local/lib/python3.8/dist-packages/glance_store/_drivers/swift/store.py", line 1014, in add
Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 <email address hidden>[93210]: ERROR glance.common.wsgi self._delete_stale_chunks(
Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 <email address hidden>[93210]: ERROR glance.common.wsgi File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 <email address hidden>[93210]: ERROR glance.common.wsgi self.force_reraise()
Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 <email address hidden>[93210]: ERROR glance.common.wsgi File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 <email address hidden>[93210]: ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb)
Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 <email address hidden>[93210]: ERROR glance.common.wsgi File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 <email address hidden>[93210]: ERROR glance.common.wsgi raise value
Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 <email address hidden>[93210]: ERROR glance.common.wsgi File "/usr/local/lib/python3.8/dist-packages/glance_store/_drivers/swift/store.py", line 1003, in add
Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 <email address hidden>[93210]: ERROR glance.common.wsgi manager.get_connection().put_object(
Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 <email address hidden>[93210]: ERROR glance.common.wsgi File "/usr/local/lib/python3.8/dist-packages/swiftclient/client.py", line 2015, in put_object
Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 <email address hidden>[93210]: ERROR glance.common.wsgi return self._retry(reset_func, put_object, container, obj, contents,
Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 <email address hidden>[93210]: ERROR glance.common.wsgi File "/usr/local/lib/python3.8/dist-packages/swiftclient/client.py", line 1898, in _retry
Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 <email address hidden>[93210]: ERROR glance.common.wsgi reset_func(func, *args, **kwargs)
Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 <email address hidden>[93210]: ERROR glance.common.wsgi File "/usr/local/lib/python3.8/dist-packages/swiftclient/client.py", line 1995, in _default_reset
Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 <email address hidden>[93210]: ERROR glance.common.wsgi raise ClientException('put_object(%r, %r, ...) failure and no '
Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 <email address hidden>[93210]: ERROR glance.common.wsgi swiftclient.exceptions.ClientException: put_object('glance', '0d60bd7d-7b97-4cb1-9105-19b917379ee4-00005', ...) failure and no ability to reset contents for reupload.

https://zuul.opendev.org/t/openstack/build/4f259f6d6cd444c2acea08a6e409a7f9/log/controller/logs/screen-s-proxy.txt?severity=0

Jan 27 11:34:01.340337 ubuntu-focal-rax-ord-0022741499 proxy-server[81160]: ERROR with Object server 127.0.0.1:6613/sdb1 re: Trying to write to /AUTH_49fdf92e98d9449b92eddfd7528e34b8/glance/0d60bd7d-7b97-4cb1-9105-19b917379ee4-00005: #012Traceback (most recent call last):#012 File "/opt/stack/swift/swift/proxy/controllers/obj.py", line 1678, in _send_chunk#012 self.conn.send(to_send)#012 File "/usr/local/lib/python3.8/dist-packages/eventlet/green/http/client.py", line 1049, in send#012 self.sock.sendall(data)#012 File "/usr/local/lib/python3.8/dist-packages/eventlet/greenio/base.py", line 405, in sendall#012 tail += self.send(data[tail:], flags)#012 File "/usr/local/lib/python3.8/dist-packages/eventlet/greenio/base.py", line 396, in send#012 return self._send_loop(self.fd.send, data, flags)#012 File "/usr/local/lib/python3.8/dist-packages/eventlet/greenio/base.py", line 383, in _send_loop#012 return send_method(data, *args)#012BrokenPipeError: [Errno 32] Broken pipe (txn: tx7f73b39856eb45c898ce6-0060114fa5) (client_ip: 10.209.161.68)
Jan 27 11:34:01.348569 ubuntu-focal-rax-ord-0022741499 swift-proxy-server[81160]: proxy-server: ERROR with Object server 127.0.0.1:6613/sdb1 re: Trying to write to /AUTH_49fdf92e98d9449b92eddfd7528e34b8/glance/0d60bd7d-7b97-4cb1-9105-19b917379ee4-00005: #012Traceback (most recent call last):#012 File "/opt/stack/swift/swift/proxy/controllers/obj.py", line 1678, in _send_chunk#012 self.conn.send(to_send)#012 File "/usr/local/lib/python3.8/dist-packages/eventlet/green/http/client.py", line 1049, in send#012 self.sock.sendall(data)#012 File "/usr/local/lib/python3.8/dist-packages/eventlet/greenio/base.py", line 405, in sendall#012 tail += self.send(data[tail:], flags)#012 File "/usr/local/lib/python3.8/dist-packages/eventlet/greenio/base.py", line 396, in send#012 return self._send_loop(self.fd.send, data, flags)#012 File "/usr/local/lib/python3.8/dist-packages/eventlet/greenio/base.py", line 383, in _send_loop#012 return send_method(data, *args)#012BrokenPipeError: [Errno 32] Broken pipe (txn: tx7f73b39856eb45c898ce6-0060114fa5) (client_ip: 10.209.161.68)
Jan 27 11:34:01.348569 ubuntu-focal-rax-ord-0022741499 swift-proxy-server[81160]: proxy-server: Object PUT exceptions during send, 0/1 required connections (txn: tx7f73b39856eb45c898ce6-0060114fa5) (client_ip: 10.209.161.68)

Lee Yarwood (lyarwood)
summary: - nova-lvm
+ nova-lvm appears to be running out of swift storage space
Revision history for this message
Lee Yarwood (lyarwood) wrote :

https://review.opendev.org/c/openstack/nova/+/772702 - quick attempt at avoiding this.

Revision history for this message
Balazs Gibizer (balazs-gibizer) wrote :

Fix merged and the nova-lvm job marked voting again. Since then the job seems to be running green so I'm closing this bugreport.

Changed in nova:
assignee: nobody → Lee Yarwood (lyarwood)
importance: Undecided → Medium
status: New → Fix Released
tags: added: gate-failure lvm
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 23.0.0.0rc1

This issue was fixed in the openstack/nova 23.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.