| 2015-11-10 09:31:13 |
Christopher Bartz |
description |
If the delay_reaping value in the [account-reaper] section of account-server.conf is higher than the reclaim_age value of the [account-replicator] (or the difference is not high, especially less than the interval of [container-updater] in container-server.conf),
then there exist the possiblity that the container databases of a deleted account are not reclaimable.
This is how it happens:
1. Account-reaper reaps the account, and thus sets the status of all container databases of the account to DELETED. Furthermore, the deleted_timestamp is adjusted, but NOT the reported_deleted_timestamp.
2. Account-replicator runs and removes the account database.
3. Container-Updater runs, but cannot update the containers which are still on the disk, because the respective account database is no longer existent. The reported_deleted_timestamp is still 0 after the run.
4. Container-Replicator cannot remove the container databases, because they have no up-to-date report status.
Step 3 & 4 are repeating forever. The container databases rest on the disk, and the account-server logs an error on step 3, because the account-server cannot find the account database. This error looks like this:
ERROR __call__ error with PUT /disk/807/AUTH_test/container : #012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/swift/account/server.py", line 271, in __call__#012 res = method(req)#012 File "/usr/lib/python2.7/site-packages/swift/common/utils.py", line 2693, in wrapped#012 return func(*a, **kw)#012 File "/usr/lib/python2.7/site-packages/swift/common/utils.py", line 1230, in _timing_stats#012 resp = func(ctrl, *args, **kwargs)#012 File "/usr/lib/python2.7/site-packages/swift/account/server.py", line 131, in PUT#012 container_policy_index)#012 File "/usr/lib/python2.7/site-packages/swift/account/backend.py", line 268, in put_container#012 self.put_record(record)#012 File "/usr/lib/python2.7/site-packages/swift/common/db.py", line 563, in put_record#012 raise DatabaseConnectionError(self.db_file, "DB doesn't exist")#012DatabaseConnectionError: DB connection error (/srv/node/disk/accounts/807/a36/c9f75af529ec2adc4b78f3ac71b9fa36/c9f75af529ec2adc4b78f3ac71b9fa36.db, 0):#012DB doesn't exist
This bug is similar to #1332439. It was observed using openstack-swift 2.5.0 on centos7.
One userside workaround could be to adjust the interval of the container-updater (so that its clear it runs before the account-replicator which removes the account-database), or to set the delay_reaping value much lower than the reclaim_age of the account. At least there should be some documentation about this issue in the official docs. There is no
warning (for example in the configuration reference) of the interdependence of these two values. |
If the delay_reaping value in the [account-reaper] section of account-server.conf is higher than the reclaim_age value of the [account-replicator] (or the difference is not high, especially less than the interval of [container-updater] in container-server.conf),
then there exist the possiblity that the container databases of a deleted account are never deleted.
This is how it happens:
1. Account-reaper reaps the account, and thus sets the status of all container databases of the account to DELETED. Furthermore, the deleted_timestamp is adjusted, but NOT the reported_deleted_timestamp.
2. Account-replicator runs and removes the account database.
3. Container-Updater runs, but cannot update the containers which are still on the disk, because the respective account database is no longer existent. The reported_deleted_timestamp is still 0 after the run.
4. Container-Replicator cannot remove the container databases, because they have no up-to-date report status.
Step 3 & 4 are repeating forever. The container databases rest on the disk, and the account-server logs an error on step 3, because the account-server cannot find the account database. This error looks like this:
ERROR __call__ error with PUT /disk/807/AUTH_test/container : #012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/swift/account/server.py", line 271, in __call__#012 res = method(req)#012 File "/usr/lib/python2.7/site-packages/swift/common/utils.py", line 2693, in wrapped#012 return func(*a, **kw)#012 File "/usr/lib/python2.7/site-packages/swift/common/utils.py", line 1230, in _timing_stats#012 resp = func(ctrl, *args, **kwargs)#012 File "/usr/lib/python2.7/site-packages/swift/account/server.py", line 131, in PUT#012 container_policy_index)#012 File "/usr/lib/python2.7/site-packages/swift/account/backend.py", line 268, in put_container#012 self.put_record(record)#012 File "/usr/lib/python2.7/site-packages/swift/common/db.py", line 563, in put_record#012 raise DatabaseConnectionError(self.db_file, "DB doesn't exist")#012DatabaseConnectionError: DB connection error (/srv/node/disk/accounts/807/a36/c9f75af529ec2adc4b78f3ac71b9fa36/c9f75af529ec2adc4b78f3ac71b9fa36.db, 0):#012DB doesn't exist
This bug is similar to #1332439. It was observed using openstack-swift 2.5.0 on centos7.
One userside workaround could be to adjust the interval of the container-updater (so that its clear it runs before the account-replicator which removes the account-database), or to set the delay_reaping value much lower than the reclaim_age of the account. At least there should be some documentation about this issue in the official docs. There is no
warning (for example in the configuration reference) of the interdependence of these two values. |
|
