Can't delete big object using swift delete command

[Maksym Livshyn]

Steps to reproduce:

1. Create a big object (10 GB) via performing following command:
$ dd if=/dev/zero of=<OBJECT_NAME> bs={OBJECT_SIZE_IN_MEGABAYTES}M \
count={OBJECT_COUNT}

2.Install and configure swiftclient cli utility

3. Create container via swiftclient cli utility
$ swift post <CONTAINER_NAME>

4.Put prepared in step number 1 object to created in step number 3 container
$ swift upload <CONTAINER_NAME> -S 1073741824 <OBJECT_NAME>

Note: data has been uploaded and another <CONTAINER_NAME_segments> has been created

5. Try to delete <OBJECT_NAME>
swift delete <CONTAINER_NAME> <OBJECT_NAME>

Actual results:
Error Deleting: SwiftContainer/BIG: Object HEAD failed: http://10.200.153.31:8080/swift/v1/SwiftContainer/BIG 500 Internal Server Error

6. Delete <OBJECT_NAME> from <CONTAINER_NAME_segments>

7. Try to delete <OBJECT_NAME> from <CONTAINER_NAME>
swift delete <CONTAINER_NAME> <OBJECT_NAME>

Actual results:
I've got internal server error. Please look at the trace attached.

Expected result: object should be deleted

MOS 7.0
.
api: '1.0'
astute_sha: 6c5b73f93e24cc781c809db9159927655ced5012
auth_required: true
build_id: '301'
build_number: '301'
feature_groups:
- mirantis
fuel-agent_sha: 50e90af6e3d560e9085ff71d2950cfbcca91af67
fuel-library_sha: 5d50055aeca1dd0dc53b43825dc4c8f7780be9dd
fuel-nailgun-agent_sha: d7027952870a35db8dc52f185bb1158cdd3d1ebd
fuel-ostf_sha: 2cd967dccd66cfc3a0abd6af9f31e5b4d150a11c
fuelmain_sha: a65d453215edb0284a2e4761be7a156bb5627677
nailgun_sha: 4162b0c15adb425b37608c787944d1983f543aa8
openstack_version: 2015.1.0-7.0
production: docker
python-fuelclient_sha: 486bde57cda1badb68f915f66c61b544108606f3
release: '7.0'
release_versions:
  2015.1.0-7.0:
    VERSION:
      api: '1.0'
      astute_sha: 6c5b73f93e24cc781c809db9159927655ced5012
      build_id: '301'
      build_number: '301'
      feature_groups:
      - mirantis
      fuel-agent_sha: 50e90af6e3d560e9085ff71d2950cfbcca91af67
      fuel-library_sha: 5d50055aeca1dd0dc53b43825dc4c8f7780be9dd
      fuel-nailgun-agent_sha: d7027952870a35db8dc52f185bb1158cdd3d1ebd
      fuel-ostf_sha: 2cd967dccd66cfc3a0abd6af9f31e5b4d150a11c
      fuelmain_sha: a65d453215edb0284a2e4761be7a156bb5627677
      nailgun_sha: 4162b0c15adb425b37608c787944d1983f543aa8
      openstack_version: 2015.1.0-7.0
      production: docker
      python-fuelclient_sha: 486bde57cda1badb68f915f66c61b544108606f3
      release: '7.0'

[Alyona Kiseleva (akiselyova)]

Is it on cluster with Ceph storage or Swift directly? If second, give, please, swift-all.log file, if it's possible.

[Radoslaw Zarzynski (rzarzynski)]

It looks that Keystone provided endpoint URL typical
for RadosGW:

    Error Deleting: SwiftContainer/BIG: Object HEAD failed:
    http://10.200.153.31:8080/swift/v1/SwiftContainer/BIG
    500 Internal Server Error

If so, we will need the RadosGW's logs.

[Maksym Livshyn (mlivshyn)]

Sorry, forgot to add info about cloud: murano, sahara, ceilometer, ceph (volume, object, ephemeral) storage, vxlan, 4 compute, 3 contoller, 6 ceph, 3 mongo.

[Maksym Livshyn (mlivshyn)]

Attached RadosGW log

[Radoslaw Zarzynski (rzarzynski)]

Apparently we have here another manifestation of the issue [1]
with error handling during GET/HEAD operation on Swift's DLO.

In swift_trace.txt [2] it can be found that swiftclient, during
the delete operation, invokes HEAD on an object to remove:

DEBUG:urllib3.connectionpool:"HEAD /swift/v1/SwiftContainer/BIG HTTP/1.1" 500 0
INFO:swiftclient:REQ: curl -i http://10.200.153.31:8080/swift/v1/SwiftContainer/BIG -I -H "X-Auth-Token: 0654b266dc9a488d9a0aef04d9aa840e"
INFO:swiftclient:RESP STATUS: 500 Internal Server Error

In ceph-client.radosgw.gateway.log [3] we've got:
2016-06-01 09:35:51.988211 7ff5ee7c4700 1 ====== starting new request req=0x7ff61001f6b0 =====
2016-06-01 09:35:52.009023 7ff5ee7c4700 0 validated token: admin:admin expires: 1464777350
2016-06-01 09:35:52.014859 7ff5ee7c4700 0 could not get bucket info for bucket=SwiftContainer_segments
2016-06-01 09:35:52.014863 7ff5ee7c4700 0 ERROR: failed to handle user manifest ret=-2
2016-06-01 09:35:52.014873 7ff5ee7c4700 1 ====== req done req=0x7ff61001f6b0 http_status=200 ======

This means that RadosGW was not able to find the container
"SwiftContainer_segments" that is responsible for storing
segments of the "SwiftContainer/BIG" DLO. Most likely user
has deleted it previously. Unfortunately, RadosGW, in such
circumstances, generates wrong HTTP header. Apache, that is
used a front-end, signalizes that to client as 500 Internal
Server Error.

[1] https://bugs.launchpad.net/mos/+bug/1580937
[2] https://bugs.launchpad.net/mos/+bug/1587815/+attachment/4674281/+files/swift_trace.txt
[3] https://bugs.launchpad.net/mos/+bug/1587815/+attachment/4674342/+files/ceph-client.radosgw.gateway.log

[Radoslaw Zarzynski (rzarzynski)]

@Maksym Livshyn

I'm going to set this bug as a duplicate of #1580937 [1].
Do we have any reasons to think that the container mentioned
as <CONTAINER_NAME_segments> had at least one proper segment
of the <OBJECT_NAME> when the error occurred?

[1] https://bugs.launchpad.net/mos/+bug/1580937

[Maksym Livshyn (mlivshyn)]

I was able to delete <CONTAINER_NAME_segments> that had segments (11 segments) of the <OBJECT_NAME> container. But after that I couldn't delete <OBJECT_NAME> and got an error message which stack trace can be found in bug

[Radoslaw Zarzynski (rzarzynski)]

As we're sure now there was no any segment of the DLO during
the HEAD operation (issued by swiftclient at the removal),
I'm marking this bug as a duplication of #1580937 [1].

[1] https://bugs.launchpad.net/mos/+bug/1580937