spurious files next to object hashdir trigger bad quarantine

Bug #1646502 reported by clayg
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Object Storage (swift)
New
Undecided
Unassigned

Bug Description

When the object datadir tree has a bogus file where a hashdir should be it will causes the whole suffix to get quarnatined:

ubuntu@saio:~$ tree /srv/node4/sdb4/objects
/srv/node4/sdb4/objects
└── 645
    └── 35a
        ├── 7089ab48d955ab0851fc51cc17a34afd # bogus file
        └── a161102fba1710ef912af194b8d4635a # normal valid data
            └── 1480572570.98264.data

3 directories, 2 files
ubuntu@saio:~$ swift-init object-replicator once -nv -c 4
WARNING: Unable to modify max process limit. Running as non-root?
Running object-replicator once...(/etc/swift/object-server/4.conf.d)
object-6040: Running object replicator in script mode.
object-6040: STDERR: ERROR:root:Quarantined /srv/node4/sdb4/objects/645/35a/7089ab48d955ab0851fc51cc17a34afd to /srv/node4/sdb4/quarantined/objects/35a because it is not a directory#012Traceback (most recent call last):#012 File "/vagrant/swift/swift/obj/diskfile.py", line 951, in _hash_suffix_dir#012 ondisk_info = self.cleanup_ondisk_files(hsh_path, reclaim_age)#012 File "/vagrant/swift/swift/obj/diskfile.py", line 906, in cleanup_ondisk_files#012 files = listdir(hsh_path)#012 File "/vagrant/swift/swift/common/utils.py", line 3104, in listdir#012 return os.listdir(path)#012OSError: [Errno 20] Not a directory: '/srv/node4/sdb4/objects/645/35a/7089ab48d955ab0851fc51cc17a34afd'
object-6040: 1/1 (100.00%) partitions replicated in 0.03s (29.83/sec, 0s remaining)
object-6040: 2 successes, 0 failures
object-6040: Object replication complete (once). (0.00 minutes)
object-6040: Exited
ubuntu@saio:~$ tree /srv/node4/sdb4/objects
/srv/node4/sdb4/objects
└── 645
    └── hashes.pkl

1 directory, 1 file
ubuntu@saio:~$ ls /srv/node4/sdb4/quarantined/objects/35a
7089ab48d955ab0851fc51cc17a34afd a161102fba1710ef912af194b8d4635a

What seems to be going on in the diskfile here, is just that the quarantine interface is expecting a data file instead of a hashdir. i.e.

this one works:

https://github.com/openstack/swift/blob/43a175ebd284e825c0f6e5b79a23d8a32f62326e/swift/obj/diskfile.py#L2044

this one is quite wrong:

https://github.com/openstack/swift/blob/43a175ebd284e825c0f6e5b79a23d8a32f62326e/swift/obj/diskfile.py#L957

clayg (clay-gerrard)
description: updated
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.