(intermittent/race?) Image prepare failed: [Errno 17] File exists

Bug #1864953 reported by Rabi Mishra
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Medium
Bogdan Dobrelya

Bug Description

Seen at [1][2][3] the container image prepare is failing during upload with :

        2020-02-27 00:46:07,106 139393 INFO tripleo_common.image.image_uploader [ ] [docker.io/tripleomaster/centos-binary-nova-libvirt:da4c8b0acc02c322be90322cef0517f785b14b27_b5a8f808] Completed modify and upload for image
        2020-02-27 00:46:07,185 139362 ERROR root [ ] Image prepare failed: [Errno 17] File exists
        Traceback (most recent call last):
          File "/usr/bin/tripleo-container-image-prepare", line 138, in <module>
            lock=lock)
          File "/usr/lib/python2.7/site-packages/tripleo_common/image/kolla_builder.py", line 233, in container_images_prepare_multi
            uploader.upload()
          File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 276, in upload
            uploader.run_tasks()
          File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 2310, in run_tasks
            for result in p.map(upload_task, self.upload_tasks):
          File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 605, in result_iterator
            yield future.result()
          File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 422, in result
            return self.__get_result()
          File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 381, in __get_result
            raise exception_type, self._exception, self._traceback
        OSError: [Errno 17] File exists

Looks like some issue with image layers and it's itermittent. Not sure if the some of the recent changes have fixed it though.

[1] https://f0a6031bc8a0bf8f57a7-5ff3dddcf26e50a416e8b904201353d9.ssl.cf2.rackcdn.com/709094/2/gate/tripleo-ci-centos-7-containers-multinode/35a0497/logs/undercloud/home/zuul/overcloud_deploy.log
[2] http://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_dec/709546/5/gate/tripleo-ci-centos-7-undercloud-containers/dec4ef7/logs/undercloud/var/log/tripleo-container-image-prepare.log
[3] https://8595f72fb8fbc262390d-a9f7bac229b8a8eda2bb7cc55aa2a966.ssl.cf1.rackcdn.com/642032/15/check/tripleo-ci-centos-7-containers-multinode/75742a5/logs/undercloud/var/log/tripleo-container-image-prepare.log

Revision history for this message
Alex Schultz (alex-schultz) wrote :

I haven't seen this one yet. I'll poke through the ci log to see how much it's happening. There's likely a race condition somewhere but the traceback is of little help.

Changed in tripleo:
status: New → Triaged
importance: Undecided → Medium
milestone: none → ussuri-3
Revision history for this message
wes hayutin (weshayutin) wrote :
Revision history for this message
Marios Andreou (marios-b) wrote :

thanks very much ramishra, i was just looking/collecting logs and saw this bug while trying to file one for this issue. I updated the description with some more links thank you

description: updated
summary: - Image prepare failed: [Errno 17] File exists
+ (intermittent/race?) Image prepare failed: [Errno 17] File exists
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-common (master)

Fix proposed to branch: master
Review: https://review.opendev.org/710836

Changed in tripleo:
assignee: nobody → Bogdan Dobrelya (bogdando)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-common (master)

Change abandoned by Bogdan Dobrelya (bogdando) (<email address hidden>) on branch: master
Review: https://review.opendev.org/710836

Changed in tripleo:
assignee: Bogdan Dobrelya (bogdando) → nobody
status: In Progress → Triaged
Changed in tripleo:
assignee: nobody → Bogdan Dobrelya (bogdando)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-common (master)

Reviewed: https://review.opendev.org/710836
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=28b8bf0fe637683123500d15ca20fdd427c2fd7d
Submitter: Zuul
Branch: master

commit 28b8bf0fe637683123500d15ca20fdd427c2fd7d
Author: Bogdan Dobrelya <email address hidden>
Date: Mon Mar 2 18:10:32 2020 +0100

    Handle race for the already existing layer

    Hard copying a large layer takes time.
    Renaming of a blob may also be not instant for some cases.
    Handle race for the already existing layers by skipping competing
    layers symlinking or renaming blobs made by concurrent workers

    Change-Id: I5fb83a13e385ed1a35a6cb735e50df2b5b368164
    Closes-bug: #1864953
    Signed-off-by: Bogdan Dobrelya <email address hidden>

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-common (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/714112

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-common (stable/train)

Reviewed: https://review.opendev.org/714112
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=419598117d1690e3228f34b4bdbd49333b535ef8
Submitter: Zuul
Branch: stable/train

commit 419598117d1690e3228f34b4bdbd49333b535ef8
Author: Bogdan Dobrelya <email address hidden>
Date: Mon Mar 2 18:10:32 2020 +0100

    Handle race for the already existing layer

    Hard copying a large layer takes time.
    Renaming of a blob may also be not instant for some cases.
    Handle race for the already existing layers by skipping competing
    layers symlinking or renaming blobs made by concurrent workers

    Change-Id: I5fb83a13e385ed1a35a6cb735e50df2b5b368164
    Closes-bug: #1864953
    Signed-off-by: Bogdan Dobrelya <email address hidden>
    (cherry picked from commit 28b8bf0fe637683123500d15ca20fdd427c2fd7d)

tags: added: in-stable-train
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-common (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/714271

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-common 12.3.0

This issue was fixed in the openstack/tripleo-common 12.3.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-common stein-eol

This issue was fixed in the openstack/tripleo-common stein-eol release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.