ERROR:root:Image prepare failed: Modifying image 192.168.24.1:8787/tripleo*

[wes hayutin]

Completed modify and upload for image\nERROR:root:Image prepare failed: Modifying image 192.168.24.1:8787/tripleomaster/centos-binary-ironic-api:e154150bc7c073535f9e373604a2e8a2476baeca_cd5cd47d failed\nTraceback (most recent call last):\n File \"/usr/bin/tripleo-container-image-prepare\", line 138, in <module>\n lock=lock)\n File \"/usr/lib/python2.7/site-packages/tripleo_common/image/kolla_builder.py\", line 233, in container_images_prepare_multi\n uploader.upload()\n File \"/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py\", line 272, in upload\n uploader.run_tasks()\n File \"/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py\", line 2282, in run_tasks\n for result in p.map(upload_task, self.upload_tasks):\n File \"/usr/lib/python2.7/site-packages/concurrent/futures/_base.py\", line 605, in result_iterator\n yield future.result()\n File \"/usr/lib/python2.7/site-packages/concurrent/futures/_base.py\", line 429, in result\n return self.__get_result()\n File \"/usr/lib/python2.7/site-packages/concurrent/futures/_base.py\", line 381, in __get_result\n raise exception_type, self._exception, self._traceback\nImageUploaderException: Modifying image 192.168.24.1:8787/tripleomaster/centos-binary-ironic-api:e154150bc7c073535f9e373604a2e8a2476baeca_cd5cd47d failed",

https://49745d51d52e030f3f3f-689ad0b9b6f06bc0c51bfb99bf86ea04.ssl.cf2.rackcdn.com/704046/1/gate/tripleo-ci-centos-7-containers-undercloud-minion/ee2ecb9/logs/undercloud/home/zuul/install-undercloud.log

Seeing multiple hits on this per day

[wes hayutin (weshayutin)]

http://logstash.openstack.org/#/dashboard/file/logstash.json?query=build_name:%20*tripleo-ci*%20AND%20build_status:%20FAILURE%20AND%20message:%20%5C%22*ValueError:%20No%20JSON%20object%20could%20be%20decoded*%5C%22

[Bogdan Dobrelya (bogdando)]

The issue here is using yum cache with multi-process workers, see https://pastebin.com/QNxVymwe snippet.

Apparently https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/undercloud-deploy/templates/containers-prepare-parameter.yaml.j2#L18
should not be used after we've switched to MP mode for container images prepare (https://review.opendev.org/#/c/687288/ and backports).

Before that change, we used to always have the first container image downloaded in a single worker mode, pre-populating the yum cache in a safe fashion. Then other workers started with using overlay mounts w/o data races. But currently we start all workers which makes multiple rw mounts to the cache and corrupts data.

[Bogdan Dobrelya (bogdando)]

https://review.opendev.org/#/c/704335/