Repeatedly "storage error [503 Inconsistent file state], last errno: File exists

Bug #1826405 reported by Paul Boven on 2019-04-25
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
apt-cacher-ng (Ubuntu)
Undecided
Unassigned

Bug Description

Since upgrading to Bionic (18.04) our apt-cacher-ng often reports this error in its log:
"storage error [503 Inconsistent file state], last errno: File exists"

This results in the client not being served the file in question, and 'apt update' failing.

The problem happens especially often with some 'local' repositories, where the 'Release' file sometimes gets cached properly, and moments later, is not available anymore on the next 'apt update' run. And a few minutes later, the file may be available again.

Running acng with 'Debug: 7' results in this:

Returning to last state, 6
Decoded request URI: http://install/Xenial-16.04-amd64/binary/Release
Processing new job, GET http://install/Xenial-16.04-amd64/binary/Release HTTP/1.1
Download started, storeHeader for install/Xenial-16.04-amd64/binary/Release, current status: 1
install/Xenial-16.04-amd64/binary/Release storage error [503 Inconsistent file state], last errno: File exists
Response header to be sent in the next cycle:
HTTP/1.1 503 Inconsistent file state
Content-Length: 145
Content-Type: text/html
Date: Thu Apr 25 13:02:08 2019
Server: Debian Apt-Cacher NG/3.1
X-Original-Source: http://install/Xenial-16.04-amd64/binary/Release
Connection: Keep-Alive

A few minutes later (from the same client, and without restarting acng, it works again:

Returning to last state, 6
Decoded request URI: http://install/Xenial-16.04-amd64/binary/Release
Processing new job, GET http://install/Xenial-16.04-amd64/binary/Release HTTP/1.1
Download started, storeHeader for install/Xenial-16.04-amd64/binary/Release, current status: 1
Response header to be sent in the next cycle:
HTTP/1.1 200 OK
Content-Length: 1658
Content-Type: application/octet-stream
Date: Thu Apr 25 13:04:30 2019
Server: Debian Apt-Cacher NG/3.1
X-Original-Source: http://install/Xenial-16.04-amd64/binary/Release
Connection: Keep-Alive

apt policy apt-cacher-ng:
apt-cacher-ng:
  Installed: 3.1-1build1
  Candidate: 3.1-1build1

lsb_release -rd:
Description: Ubuntu 18.04.2 LTS
Release: 18.04

The apt-cacher-ng is running on a virtual machine, with the above configuration. There is plenty of diskspace available. The problem even occurs after removing acng, clearing /var/cache/apt-cacher-ng/, and re-installing the package.

Paul Boven (p-boven) wrote :

Further analysis shows that this problem is due to a fun interaction between lighttpd an ACNG. We use lighttpd to serve a local Debian/Ubuntu package repository. In about half the cases where we run 'apt update', it will fail with error 503 on the 'Release' file of such a repository.

What happens is:

1) Client asks ACNG for a new file. ACNG downloads it, serves it to the client, adds it to /var/cache/apt-cacher-ng/<somewhere>

2) Subsequent requests for this file will work as expected

3) If the file hasn't been requested for a while, ACNG will run 'fileitem::DoDelayedUnregAndCheck'

4) The file is still in /var/cache/apt-cacher-ng/<somewhere>, but ACNG has forgotten about it. A new request will fail with error 503 because ACNG detects the file being in the cache ('Inconsistent File State'). ACNG deletes the file in question, and the next request will work.

This only happens if there is no 'Last Modified' HTTP header sent by lighttpd. I've modified the lighttpd.conf to add this header, and ACNG no longer gives the '503 Inconsistent File State' error.

Lighttpd.conf addition (from https://www.anexia-it.com/blog/en/the-tale-of-lighttpd-not-sending-the-last-modified-header/ ):

mimetype.assign += ("" => "application/octet-stream" )

This still seems like a bug in ACNG, but this work-around might be useful for some.

Same bug different cause:
VfileUseRangeOps: 0
is no longer usable.
Ubuntu dis-upgrader-all tar.gz files (for do-release-upgrade) always trigger the same bug with the "storage error [503 Inconsistent file state], last errno: File exists".

As our proxys seems to be fixed regarding if-range header, using the default "VfileUseRangeOps: 1" fixed our problem.

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in apt-cacher-ng (Ubuntu):
status: New → Confirmed

Hoo my....
Our proxys are NOT fixed.
I go back to VfileUseRangeOps: 0.
I need to choose between a functional repository for daily updates or a functional repository for distribution release upgrade.
Will try if with VfileUseRangeOps: -1 I could get both.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers