Image prepare fails with throwing exception: Image prepare failed: an integer is required (got type NoneType)

[Sagi (Sergey) Shnaidman]

http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-scenario001-standalone-master/34a9d62/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

2019-11-18 01:33:45 23452 ERROR root [ ] Image prepare failed: an integer is required (got type NoneType)
concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/lib64/python3.6/concurrent/futures/process.py", line 175, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/usr/lib64/python3.6/concurrent/futures/process.py", line 153, in _process_chunk
    return [fn(*args) for args in chunk]
  File "/usr/lib64/python3.6/concurrent/futures/process.py", line 153, in <listcomp>
    return [fn(*args) for args in chunk]
  File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py", line 2155, in upload_task
    return uploader.upload_image(task)
  File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py", line 1281, in upload_image
    lock=lock
  File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 292, in wrapped_f
    return self.call(f, *args, **kw)
  File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 358, in call
    do = self.iter(retry_state=retry_state)
  File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 319, in iter
    return fut.result()
  File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 361, in call
    result = fn(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py", line 1583, in _copy_registry_to_registry
    raise e
  File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 292, in wrapped_f
    return self.call(f, *args, **kw)
  File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 358, in call
    do = self.iter(retry_state=retry_state)
  File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 319, in iter
    return fut.result()
  File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 361, in call
    result = fn(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py", line 1501, in _copy_layer_registry_to_registry
    cls._layer_fetch_lock(layer, lock)
  File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 292, in wrapped_f
    return self.call(f, *args, **kw)
  File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 358, in call
    do = self.iter(retry_state=retry_state)
  File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 319, in iter
    return fut.result()
  File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 361, in call
    result = fn(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py", line 1136, in _layer_fetch_lock
    if layer in lock.objects():
  File "<string>", line 2, in __contains__
  File "/usr/lib64/python3.6/multiprocessing/managers.py", line 757, in _callmethod
    kind, result = conn.recv()
  File "/usr/lib64/python3.6/multiprocessing/connection.py", line 254, in recv
    buf = self._recv_bytes()
  File "/usr/lib64/python3.6/multiprocessing/connection.py", line 415, in _recv_bytes
    return self._recv(size)
  File "/usr/lib64/python3.6/multiprocessing/connection.py", line 383, in _recv
    chunk = read(handle, remaining)
TypeError: an integer is required (got type NoneType)
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/bin/tripleo-container-image-prepare", line 138, in <module>
    lock=lock)
  File "/usr/lib/python3.6/site-packages/tripleo_common/image/kolla_builder.py", line 235, in container_images_prepare_multi
    uploader.upload()
  File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py", line 271, in upload
    uploader.run_tasks()
  File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py", line 2095, in run_tasks
    for result in p.map(upload_task, self.upload_tasks):
  File "/usr/lib64/python3.6/concurrent/futures/process.py", line 366, in _chain_from_iterable_of_lists
    for element in iterable:
  File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 586, in result_iterator
    yield fs.pop().result()
  File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
TypeError: an integer is required (got type NoneType)

[Sagi (Sergey) Shnaidman (sshnaidm)]

Setting for "High" priority, because it happens sometimes.

[Bogdan Dobrelya (bogdando)]

Looks like an internal issue with py3 and MP https://stackoverflow.com/questions/29277150/python-3-4-multiprocessing-bug-on-lock-acquire-typeerror-integer-required

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/694773

[OpenStack Infra (hudson-openstack)]

Change abandoned by Bogdan Dobrelya (bogdando) (<email address hidden>) on branch: master
Review: https://review.opendev.org/694773

[Bogdan Dobrelya (bogdando)]

this is probably not relevant any more

[Marios Andreou (marios-b)]

This is an automated action. Bug status has been set to 'Incomplete' and target milestone has been removed due to inactivity. If you disagree please re-set these values and reach out to us on freenode #tripleo