tripleo

[train] MemoryError: [tripleotraincentos8/centos-binary-nova-libvirt] Memory Error failing on few jobs

Bug #1895288 reported by Bhagyashri Shewale on 2020-09-11

This bug report is a duplicate of: Bug #1895290: [train] ERROR root [ ] Image prepare failed: A process in the process pool was terminated abruptly while the future was running or pending failing on few jobs. Edit Remove

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	tripleo	Triaged	High	Unassigned	tripleo victoria-3 "tripleo victoria"

Bug Description

Below jobs are failing because of MemoryError: [tripleotraincentos8/centos-binary-nova-libvirt] Memory Error

Job affected:

periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train
periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-train
periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-train
periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset001-train

Error log:

2020-09-11 06:36:55,252 218447 ERROR root [ ] Image prepare failed: [tripleotraincentos8/centos-binary-nova-libvirt] Memory Error:
concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_export.py", line 115, in export_stream
    for chunk in layer_stream:
  File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py", line 1774, in _layer_stream_registry
    request=blob_req)
  File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py", line 176, in check_status
    text = getattr(request, 'text', 'unknown')
  File "/usr/lib/python3.6/site-packages/requests/models.py", line 861, in text
    content = str(self.content, encoding, errors='replace')
MemoryError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.6/concurrent/futures/process.py", line 175, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/usr/lib64/python3.6/concurrent/futures/process.py", line 153, in _process_chunk
    return [fn(*args) for args in chunk]
  File "/usr/lib64/python3.6/concurrent/futures/process.py", line 153, in <listcomp>
    return [fn(*args) for args in chunk]
  File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py", line 2572, in upload_task
    return uploader.upload_image(task)
  File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py", line 1558, in upload_image
    multi_arch=t.multi_arch
  File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 292, in wrapped_f
    return self.call(f, *args, **kw)
  File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 358, in call
    do = self.iter(retry_state=retry_state)
  File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 319, in iter
    return fut.result()
  File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 361, in call
    result = fn(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py", line 1905, in _copy_registry_to_registry
    raise e
  File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 292, in wrapped_f
    return self.call(f, *args, **kw)
  File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 358, in call
    do = self.iter(retry_state=retry_state)
  File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 319, in iter
    return fut.result()
  File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 361, in call
    result = fn(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py", line 1835, in _copy_layer_registry_to_registry
    target_session)
  File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py", line 2228, in _copy_stream_to_registry
    target_url, layer, layer_stream, verify_digest=verify_digest)
  File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_export.py", line 60, in wrapper
    return f(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_export.py", line 130, in export_stream
    raise MemoryError(memory_error)
MemoryError: [tripleotraincentos8/centos-binary-nova-libvirt] Memory Error:
"""