remove files and containers after replication (1 minutes after upload)

[NeOn4eg]

(sorry for my english and russian lang in logs)
On this how-to http://docs.openstack.org/developer/swift/howto_installmultinode.html on 3 servers ubuntu 12.04 install 2 storage and 1 proxy node.

Upload file:
swift -A https://172.24.82.2:8080/auth/v1.0 -U system:root -K testpass upload myfiles /var/log/syslog.1

Look stat:
Account: AUTH_system
Containers: 1
   Objects: 0
     Bytes: 0

After minute, stat:
Account: AUTH_system
Containers: 0
   Objects: 0
     Bytes: 0

removed.

My rings:
container.builder, build version 6
262144 partitions, 2 replicas, 2 zones, 6 devices, 0.00 balance
The minimum number of hours before a partition can be reassigned is 1
Devices: id zone ip address port name weight partitions balance meta
             0 0 172.24.82.102 6001 sdc 100.00 113976 0.00
             1 0 172.24.82.102 6001 sdb 30.00 34192 -0.00
             2 0 172.24.82.102 6001 sde 100.00 113976 0.00
             3 1 172.24.82.103 6001 sdb 30.00 34192 -0.00
             4 1 172.24.82.103 6001 sdc 100.00 113976 0.00
             5 1 172.24.82.103 6001 sdd 100.00 113976 0.00

Proxy logs:
proxy-server - 172.24.82.2 09/Jul/2014/07/04/41 GET /auth/v1.0/ HTTP/1.0 200 - - - - - - - - 0.0006
proxy-server 172.24.82.2 172.24.82.2 09/Jul/2014/07/04/41 HEAD /v1/AUTH_system HTTP/1.0 204 - - system%2CAUTH_tke187c0b5a39945dfa7e9a3d0979714dd - - - tx16587c16454c4aaeb85fa7c33417a600 - 0.0177 -
proxy-server - 172.24.82.2 09/Jul/2014/07/04/54 GET /auth/v1.0/ HTTP/1.0 200 - - - - - - - - 0.0006
proxy-server 172.24.82.2 172.24.82.2 09/Jul/2014/07/04/55 PUT /v1/AUTH_system/test HTTP/1.0 201 - - system%2CAUTH_tke187c0b5a39945dfa7e9a3d0979714dd - - - tx24fc2a93c18d41f78059a1bf0d22b1e6 - 0.7671 -
proxy-server - 172.24.82.2 09/Jul/2014/07/04/55 GET /auth/v1.0/ HTTP/1.0 200 - - - - - - - - 0.0007
proxy-server 172.24.82.2 172.24.82.2 09/Jul/2014/07/04/55 HEAD /v1/AUTH_system/test/var/log/syslog.1 HTTP/1.0 200 - - system%2CAUTH_tke187c0b5a39945dfa7e9a3d0979714dd - - - txc665431a36324bcf8dd1953492e98a18 - 0.0036 -
proxy-server 172.24.82.2 172.24.82.2 09/Jul/2014/07/04/55 PUT /v1/AUTH_system/test/var/log/syslog.1 HTTP/1.0 201 - - system%2CAUTH_tke187c0b5a39945dfa7e9a3d0979714dd 1365261 - - tx8140e43a043c4215b27d355edbb2b33a - 0.0805 -
account-replicator Beginning replication run
account-replicator Копирование ЗАВЕРШЕНО
account-replicator Attempted to replicate 0 dbs in 0.00286 seconds (0.00000/s)

Storage 1 logs:
account-server 172.24.82.2 - - [09/Jul/2014:07:04:54 +0000] "HEAD /sdb/233800/AUTH_system" 404 - "tx24fc2a93c18d41f78059a1bf0d22b1e6" "-" "-" 0.0003 ""
account-server 172.24.82.2 - - [09/Jul/2014:07:04:54 +0000] "PUT /sdb/233800/AUTH_system" 201 - "tx24fc2a93c18d41f78059a1bf0d22b1e6" "-" "-" 0.0054 ""
account-server 172.24.82.102 - - [09/Jul/2014:07:04:55 +0000] "PUT /sdb/233800/AUTH_system/test" 201 - "tx24fc2a93c18d41f78059a1bf0d22b1e6" "PUT http://172.24.82.102:6001/sde/162733/AUTH_system/test" "container-server 999" 0.0011 ""
container-server 172.24.82.2 - - [09/Jul/2014:07:04:55 +0000] "PUT /sde/162733/AUTH_system/test" 201 - "tx24fc2a93c18d41f78059a1bf0d22b1e6" "-" "-" 0.6693
object-server 172.24.82.2 - - [09/Jul/2014:07:04:55 +0000] "HEAD /sdc/48782/AUTH_system/test/var/log/syslog.1" 200 1365261 "-" "txc665431a36324bcf8dd1953492e98a18" "-" 0.0005
container-server 172.24.82.2 - - [09/Jul/2014:07:04:55 +0000] "HEAD /sde/162733/AUTH_system/test" 204 - "tx8140e43a043c4215b27d355edbb2b33a" "-" "-" 0.0016
container-server 172.24.82.103 - - [09/Jul/2014:07:04:55 +0000] "PUT /sde/162733/AUTH_system/test/var/log/syslog.1" 201 - "tx8140e43a043c4215b27d355edbb2b33a" "PUT http://172.24.82.2:8080/sdc/48782/AUTH_system/test/var/log/syslog.1" "obj-server 24891" 0.0005
object-server 172.24.82.2 - - [09/Jul/2014:07:04:55 +0000] "PUT /sdc/48782/AUTH_system/test/var/log/syslog.1" 201 - "-" "tx8140e43a043c4215b27d355edbb2b33a" "-" 0.0725
object-auditor Object audit (ALL) "forever" mode completed: 7.88s. Total quarantined: 0, Total errors: 0, Total files/sec: 12.81 , Total bytes/sec: 9997171.31, Auditing time: 7.87, Rate: 1.00
object-replicator Starting object replication pass.
object-replicator 101/101 (100.00%) partitions replicated in 0.95s (106.17/sec, 0s remaining)
object-replicator 101 suffixes checked - 0.99% hashed, 0.00% synced
object-replicator Partition times: max 0.6752s, min 0.0022s, med 0.0024s
object-replicator Object replication complete. (0.02 minutes)
object-auditor Begin object audit "forever" mode (ZBF)
account-replicator Beginning replication run
account-replicator Found /srv/node/sdb/accounts/233800/a35/b36718b6492ea6810268065dee709a35/b36718b6492ea6810268065dee709a35.db for /AUTH_system when it should be on partition 183708;will replicate out and remove.
account-replicator Replication run OVER
account-replicator Attempted to replicate 2 dbs in 0.71388 seconds (2.80160/s)
account-replicator Removed 1 dbs
account-replicator 3 успешно, 0 ошибочно
account-replicator no_change:1 ts_repl:0 diff:2 rsync:0 diff_capped:0 hashmatch:0 empty:0
object-auditor Object audit (ZBF) "forever" mode completed: 2.00s. Total quarantined: 0, Total errors: 0, Total files/sec: 50.48 , Total bytes/sec: 0.00, Auditing time: 1.99, Rate: 0.99
container-replicator Beginning replication run
container-replicator Found /srv/node/sde/containers/162733/81c/4b752fd55e4d9f362fabcef74f9aa81c/4b752fd55e4d9f362fabcef74f9aa81c.db for /AUTH_system/test when it should be on partition 77268; will replicate out and remove.
container-replicator Replication run OVER
container-replicator Attempted to replicate 11 dbs in 0.14508 seconds (75.81914/s)
container-replicator Removed 1 dbs
container-replicator 11 успешно, 0 ошибочно
container-replicator no_change:11 ts_repl:0 diff:0 rsync:0 diff_capped:0 hashmatch:0 empty:0
object-auditor Begin object audit "forever" mode (ALL)
object-auditor Object audit (ALL) "forever" mode completed: 7.88s. Total quarantined: 0, Total errors: 0, Total files/sec: 12.82 , Total bytes/sec: 9997602.58, Auditing time: 7.87, Rate: 1.00

Storage 2 logs:
account-server 172.24.82.103 - - [09/Jul/2014:07:04:55 +0000] "PUT /sdb/233800/AUTH_system/test" 201 - "tx24fc2a93c18d41f78059a1bf0d22b1e6" "PUT http://172.24.82.103:6001/sdc/162733/AUTH_system/test" "container-server 24887" 0.0012 ""
container-server 172.24.82.2 - - [09/Jul/2014:07:04:55 +0000] "PUT /sdc/162733/AUTH_system/test" 201 - "tx24fc2a93c18d41f78059a1bf0d22b1e6" "-" "-" 0.7502
object-server 172.24.82.2 - - [09/Jul/2014:07:04:55 +0000] "PUT /sdc/48782/AUTH_system/test/var/log/syslog.1" 201 - "-" "tx8140e43a043c4215b27d355edbb2b33a" "-" 0.0705
container-server 172.24.82.102 - - [09/Jul/2014:07:04:55 +0000] "PUT /sdc/162733/AUTH_system/test/var/log/syslog.1" 201 - "tx8140e43a043c4215b27d355edbb2b33a" "PUT http://172.24.82.2:8080/sdc/48782/AUTH_system/test/var/log/syslog.1" "obj-server 1007" 0.0005
object-auditor Object audit (ALL) "forever" mode completed: 7.89s. Total quarantined: 0, Total errors: 0, Total files/sec: 12.80 , Total bytes/sec: 9988169.79, Auditing time: 7.88, Rate: 1.00
object-replicator Starting object replication pass.
object-replicator 101/101 (100.00%) partitions replicated in 0.82s (123.22/sec, 0s remaining)
object-replicator 101 suffixes checked - 0.00% hashed, 0.00% synced
object-replicator Partition times: max 0.5463s, min 0.0023s, med 0.0024s
object-replicator Object replication complete. (0.01 minutes)
object-auditor Begin object audit "forever" mode (ZBF)
object-auditor Object audit (ZBF) "forever" mode completed: 2.00s. Total quarantined: 0, Total errors: 0, Total files/sec: 50.47 , Total bytes/sec: 0.00, Auditing time: 1.99, Rate: 0.99
container-replicator Beginning replication run
container-replicator Found /srv/node/sdc/containers/162733/81c/4b752fd55e4d9f362fabcef74f9aa81c/4b752fd55e4d9f362fabcef74f9aa81c.db for /AUTH_system/test when it should be on partition 77268; will replicate out and remove.
container-replicator Replication run OVER
container-replicator Attempted to replicate 11 dbs in 0.71983 seconds (15.28132/s)
container-replicator Removed 1 dbs
container-replicator 12 успешно, 0 ошибочно
container-replicator no_change:10 ts_repl:0 diff:2 rsync:0 diff_capped:0 hashmatch:0 empty:0
account-replicator Beginning replication run
account-replicator Found /srv/node/sdb/accounts/233800/a35/b36718b6492ea6810268065dee709a35/b36718b6492ea6810268065dee709a35.db for /AUTH_system when it should be on partition183708; will replicate out and remove.
account-replicator Replication run OVER
account-replicator Attempted to replicate 2 dbs in 0.02748 seconds (72.77771/s)
account-replicator Removed 1 dbs
account-replicator 2 успешно, 0 ошибочно
account-replicator no_change:1 ts_repl:0 diff:1 rsync:0 diff_capped:0 hashmatch:0 empty:0

What is it?

[clayg (clay-gerrard)]

I don't see anything from the container-updater in the logs. The container-updater is the only thing that syncs container stats to the account.

When account stat shows 0 containers can you still:

    swift stat myfiles

    swift stat myfiles /var/log/syslog.1

[r-leandro (r-leandro)]

I'm with the same problem.

After sync, I can't list my files or containers.

May 13 17:58:05 swift-object-01 account-replicator: Replication run OVER
May 13 17:58:05 swift-object-01 account-replicator: Attempted to replicate 9 dbs in 0.31212 seconds (28.83486/s)
May 13 17:58:05 swift-object-01 account-replicator: Removed 3 dbs
May 13 17:58:05 swift-object-01 account-replicator: 18 successes, 0 failures

[clayg (clay-gerrard)]

Run the container-updater:

swift-init container-updater start

[Matt Wear (wearmg)]

Do you have a mismatch between the swift_hash_path_suffix / prefix in /etc/swift/swift.conf between your proxy nodes and your storage nodes?

[Launchpad Janitor (janitor)]

[Expired for OpenStack Object Storage (swift) because there has been no activity for 60 days.]

[Donny Davis (donny-g)]

This is really old, but I just ran into it so I thought it would be useful to say what the error actually is.

As mentioned in comment #4, its is in fact from a mismatch in /etc/swift/swift.conf swift_hash_path_suffix / prefix

[John Dickinson (notmyname)]

Donny, thank you for adding that information here!

[David Hill (david-hill-ubisoft)]

Well, we do hit this issue too here and we have the same value everywhere:

swift_hash_path_suffix = secret

The proxy node and storage node are hosted on the same host.