Unable to push to swift storage backend
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Docker Registry Charm |
Triaged
|
Medium
|
Kevin W Monroe |
Bug Description
Somehow the swift backend can time out and stop an image from being uploaded. Debug level errors from the registry look like this:
{"err.code"
The " Timeout expired while waiting for segments" message, and 0 length data file make me think that the issue is this upstream bug: https:/
For example, in that failed upload, the files in question look like this in swift:
$ swift stat --lh docker-
Account: AUTH_18fdda09da
Container: docker-
Object: files/docker/
Content Type: application/
Content Length: 71
Last Modified: Tue, 11 Dec 2018 05:39:19 GMT
ETag: 239c76eecd4fd38
Accept-Ranges: bytes
X-Timestamp: 1544506758.53727
X-Trans-Id: txae919e1f3a5a4
$ swift stat --lh docker-
Account: AUTH_18fdda09da
Container: docker-
Object: files/docker/
Content Type: application/
Content Length: 0
Last Modified: Tue, 11 Dec 2018 05:39:15 GMT
ETag: "d41d8cd98f00b2
Manifest: docker-
Accept-Ranges: bytes
X-Timestamp: 1544506754.17360
X-Trans-Id: tx2f2d17d03c024
Anyone pushing an image that contains the layer in question will run into the error, since the files in swift are named based on the sha256 hash. The upstream issue doesn't really have much in the way of fixes.
The workaround for now is to delete the files, and then re-push the image in question.
summary: |
- Unable to push to registry with swift storage backend + Unable to push to swift storage backend |
description: | updated |
Changed in layer-docker-registry: | |
status: | New → Triaged |
Changed in layer-docker-registry: | |
assignee: | nobody → Kevin W Monroe (kwmonroe) |
assignee: | Kevin W Monroe (kwmonroe) → nobody |
status: | Triaged → In Progress |
assignee: | nobody → Kevin W Monroe (kwmonroe) |
Changed in layer-docker-registry: | |
status: | Fix Committed → Fix Released |
Changed in layer-docker-registry: | |
status: | Fix Released → New |
Changed in layer-docker-registry: | |
importance: | High → Medium |
I can reproduce the error manually by deleting a segment from a layer:
'swift delete docker-registry segments/ 2f6/46f636b6572 2f7.... ..'
Or all of them like this):
'swift list docker-registry |grep segments |xargs swift delete docker-registry'
The following seems to recover things consistently:
REGISTRY=$(docker ps |awk '{print $1}'|tail -n 1) docker- registry
echo "Removing $f"
swift delete $CONTAINER "files$f" > /dev/null 2>&1
CONTAINER=
FILES=$(docker logs $REGISTRY | jq -r -R 'fromjson? |.["err.detail"] | match("swift: Timeout expired while waiting for segments of (.+sha256.+) to show up"; "g") | .captures[0] .string' |sort |uniq)
for f in $FILES; do
swift stat --lh $CONTAINER "files$f" 2>&1 |grep -q "Content Length: 0"
if [ $? -eq 0 ]; then
fi
done
Then push again and things will work.