Corruption of object suffix directories becoming files

[Gerry Drudy]

Am running swift 1.7.5.
I sometimes notice a swift object suffix that is now a file (and not always 0 length) rather than a directory. For example:

-rw------- 1 swift swift 47984 2013-01-11 13:18 /srv/node/disk8/objects/1002854/5d6
-rw------- 1 swift swift 45878 2013-01-11 19:11 /srv/node/disk8/objects/102304/996
-rw------- 1 swift swift 44981 2013-01-11 19:11 /srv/node/disk8/objects/107795/422
-rw------- 1 swift swift 42914 2013-01-11 18:30 /srv/node/disk8/objects/1093863/e0d
-rw------- 1 swift swift 0 2012-09-08 06:42 /srv/node/disk8/objects/1101327/a0b

Some of these files now look like hashes.pkl files:

$ sudo strings /srv/node/disk8/objects/1002854/5d6 | head
9fdU 627de79c8e45dea77a0ab3f8403917f7U
9feU 6d55735daf02c87c966da2762dbc5a53U
fa8U 33c5fb6c26f9ea6bf1a18f544e87b72cU
dc1U 88132ddb5363a2874fdea24d57150a98U
edfU 4b028c6b2a7ca2b616ac5fe26e8b1d82U
9faU 3982e12effb41eefa25ade8e493f3594U
eddU 15fcf305cfdb0722815449a57e849ee3U
fa5U b13012039dbe6983aebce6ec396fe688U
164U 064776b1f09d535b8265ff7827acb523U
3a8U 0deaa2e8da8e5d114547c59183378eb9U

I haven't spotted anything in the background.log relating to these suffix corruptions.

[Gerry Drudy (gerry-drudy)]

Though the problem can be spotted by entries like the following in rsyncd.log:

2013/01/31 06:25:05 [10386] rsync to object/disk8/objects/142776 from object2 (1.2.3.4)
2013/01/31 06:25:05 [10386] receiving file list
2013/01/31 06:25:06 [10386] default_perms_for_dir: sys_acl_get_file(5b3, ACL_TYPE_DEFAULT): Permission denied, falling back on umask
2013/01/31 06:25:06 [10386] rsync: recv_generator: mkdir "/disk8/objects/142776/5b3/116dc4cc1114526f7301fdfdfdcb45b3" (in object) failed: Not a directory (20)
2013/01/31 06:25:06 [10386] *** Skipping any contents from this failed directory ***

[Samuel Merritt (torgomatic)]

Sounds like filesystem corruption to me.

There's some code in the Swift object replicator to take those busted directories out of hashes.pkl when it finds them (see ObjectReplicator.get_hashes(); look for PathNotDir).

Other than skipping them, there's probably not much Swift can do.

[Gerry Drudy (gerry-drudy)]

Note, some of the suffix directories are zero byte files, similar to the way partition directories were reported as zero byte files in https://bugs.launchpad.net/swift/+bug/1045954.

[John Dickinson (notmyname)]

Gerry is this something you are still seeing?

[Launchpad Janitor (janitor)]

[Expired for OpenStack Object Storage (swift) because there has been no activity for 60 days.]