(intermittent/race?) Image prepare failed: [Errno 17] File exists

[Rabi Mishra]

Seen at [1][2][3] the container image prepare is failing during upload with :

        2020-02-27 00:46:07,106 139393 INFO tripleo_common.image.image_uploader [ ] [docker.io/tripleomaster/centos-binary-nova-libvirt:da4c8b0acc02c322be90322cef0517f785b14b27_b5a8f808] Completed modify and upload for image
        2020-02-27 00:46:07,185 139362 ERROR root [ ] Image prepare failed: [Errno 17] File exists
        Traceback (most recent call last):
          File "/usr/bin/tripleo-container-image-prepare", line 138, in <module>
            lock=lock)
          File "/usr/lib/python2.7/site-packages/tripleo_common/image/kolla_builder.py", line 233, in container_images_prepare_multi
            uploader.upload()
          File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 276, in upload
            uploader.run_tasks()
          File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 2310, in run_tasks
            for result in p.map(upload_task, self.upload_tasks):
          File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 605, in result_iterator
            yield future.result()
          File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 422, in result
            return self.__get_result()
          File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 381, in __get_result
            raise exception_type, self._exception, self._traceback
        OSError: [Errno 17] File exists

Looks like some issue with image layers and it's itermittent. Not sure if the some of the recent changes have fixed it though.

[1] https://f0a6031bc8a0bf8f57a7-5ff3dddcf26e50a416e8b904201353d9.ssl.cf2.rackcdn.com/709094/2/gate/tripleo-ci-centos-7-containers-multinode/35a0497/logs/undercloud/home/zuul/overcloud_deploy.log
[2] http://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_dec/709546/5/gate/tripleo-ci-centos-7-undercloud-containers/dec4ef7/logs/undercloud/var/log/tripleo-container-image-prepare.log
[3] https://8595f72fb8fbc262390d-a9f7bac229b8a8eda2bb7cc55aa2a966.ssl.cf1.rackcdn.com/642032/15/check/tripleo-ci-centos-7-containers-multinode/75742a5/logs/undercloud/var/log/tripleo-container-image-prepare.log

[Alex Schultz (alex-schultz)]

I haven't seen this one yet. I'll poke through the ci log to see how much it's happening. There's likely a race condition somewhere but the traceback is of little help.

[wes hayutin (weshayutin)]

https://f0a6031bc8a0bf8f57a7-5ff3dddcf26e50a416e8b904201353d9.ssl.cf2.rackcdn.com/709094/2/gate/tripleo-ci-centos-7-containers-multinode/35a0497/

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_dec/709546/5/gate/tripleo-ci-centos-7-undercloud-containers/dec4ef7/

[Marios Andreou (marios-b)]

thanks very much ramishra, i was just looking/collecting logs and saw this bug while trying to file one for this issue. I updated the description with some more links thank you

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/710836

[OpenStack Infra (hudson-openstack)]

Change abandoned by Bogdan Dobrelya (bogdando) (<email address hidden>) on branch: master
Review: https://review.opendev.org/710836

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/710836
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=28b8bf0fe637683123500d15ca20fdd427c2fd7d
Submitter: Zuul
Branch: master

commit 28b8bf0fe637683123500d15ca20fdd427c2fd7d
Author: Bogdan Dobrelya <email address hidden>
Date: Mon Mar 2 18:10:32 2020 +0100

    Handle race for the already existing layer

    Hard copying a large layer takes time.
    Renaming of a blob may also be not instant for some cases.
    Handle race for the already existing layers by skipping competing
    layers symlinking or renaming blobs made by concurrent workers

    Change-Id: I5fb83a13e385ed1a35a6cb735e50df2b5b368164
    Closes-bug: #1864953
    Signed-off-by: Bogdan Dobrelya <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/714112

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/714112
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=419598117d1690e3228f34b4bdbd49333b535ef8
Submitter: Zuul
Branch: stable/train

commit 419598117d1690e3228f34b4bdbd49333b535ef8
Author: Bogdan Dobrelya <email address hidden>
Date: Mon Mar 2 18:10:32 2020 +0100

    Handle race for the already existing layer

    Hard copying a large layer takes time.
    Renaming of a blob may also be not instant for some cases.
    Handle race for the already existing layers by skipping competing
    layers symlinking or renaming blobs made by concurrent workers

    Change-Id: I5fb83a13e385ed1a35a6cb735e50df2b5b368164
    Closes-bug: #1864953
    Signed-off-by: Bogdan Dobrelya <email address hidden>
    (cherry picked from commit 28b8bf0fe637683123500d15ca20fdd427c2fd7d)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/714271

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common 12.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common stein-eol release.