Repeatedly "storage error [503 Inconsistent file state], last errno: File exists
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
apt-cacher-ng (Ubuntu) |
Confirmed
|
Undecided
|
Unassigned |
Bug Description
Since upgrading to Bionic (18.04) our apt-cacher-ng often reports this error in its log:
"storage error [503 Inconsistent file state], last errno: File exists"
This results in the client not being served the file in question, and 'apt update' failing.
The problem happens especially often with some 'local' repositories, where the 'Release' file sometimes gets cached properly, and moments later, is not available anymore on the next 'apt update' run. And a few minutes later, the file may be available again.
Running acng with 'Debug: 7' results in this:
Returning to last state, 6
Decoded request URI: http://
Processing new job, GET http://
Download started, storeHeader for install/
install/
Response header to be sent in the next cycle:
HTTP/1.1 503 Inconsistent file state
Content-Length: 145
Content-Type: text/html
Date: Thu Apr 25 13:02:08 2019
Server: Debian Apt-Cacher NG/3.1
X-Original-Source: http://
Connection: Keep-Alive
A few minutes later (from the same client, and without restarting acng, it works again:
Returning to last state, 6
Decoded request URI: http://
Processing new job, GET http://
Download started, storeHeader for install/
Response header to be sent in the next cycle:
HTTP/1.1 200 OK
Content-Length: 1658
Content-Type: application/
Date: Thu Apr 25 13:04:30 2019
Server: Debian Apt-Cacher NG/3.1
X-Original-Source: http://
Connection: Keep-Alive
apt policy apt-cacher-ng:
apt-cacher-ng:
Installed: 3.1-1build1
Candidate: 3.1-1build1
lsb_release -rd:
Description: Ubuntu 18.04.2 LTS
Release: 18.04
The apt-cacher-ng is running on a virtual machine, with the above configuration. There is plenty of diskspace available. The problem even occurs after removing acng, clearing /var/cache/
Further analysis shows that this problem is due to a fun interaction between lighttpd an ACNG. We use lighttpd to serve a local Debian/Ubuntu package repository. In about half the cases where we run 'apt update', it will fail with error 503 on the 'Release' file of such a repository.
What happens is:
1) Client asks ACNG for a new file. ACNG downloads it, serves it to the client, adds it to /var/cache/ apt-cacher- ng/<somewhere>
2) Subsequent requests for this file will work as expected
3) If the file hasn't been requested for a while, ACNG will run 'fileitem: :DoDelayedUnreg AndCheck'
4) The file is still in /var/cache/ apt-cacher- ng/<somewhere> , but ACNG has forgotten about it. A new request will fail with error 503 because ACNG detects the file being in the cache ('Inconsistent File State'). ACNG deletes the file in question, and the next request will work.
This only happens if there is no 'Last Modified' HTTP header sent by lighttpd. I've modified the lighttpd.conf to add this header, and ACNG no longer gives the '503 Inconsistent File State' error.
Lighttpd.conf addition (from https:/ /www.anexia- it.com/ blog/en/ the-tale- of-lighttpd- not-sending- the-last- modified- header/ ):
mimetype.assign += ("" => "application/ octet-stream" )
This still seems like a bug in ACNG, but this work-around might be useful for some.