Duplicate sources.list entries can result in corrupted downloads

[Shawn Walker]

Hit http://security.ubuntu.com warty-security/main Packages
Hit http://security.ubuntu.com warty-security/main Release
Hit http://archive.ubuntu.com warty/main Packages
Hit http://archive.ubuntu.com warty/main Release
Hit http://security.ubuntu.com warty-security/restricted Packages
Hit http://security.ubuntu.com warty-security/restricted Release
Hit http://security.ubuntu.com warty-security/main Sources
Hit http://security.ubuntu.com warty-security/main Release
Hit http://security.ubuntu.com warty-security/restricted Sources
Hit http://security.ubuntu.com warty-security/restricted Release
Hit http://archive.ubuntu.com warty/restricted Packages
Hit http://archive.ubuntu.com warty/restricted Release
Get:1 http://archive.ubuntu.com warty/main Sources [171kB]
Hit http://archive.ubuntu.com warty/main Release
Get:2 http://archive.ubuntu.com warty/restricted Sources [753B]
Hit http://archive.ubuntu.com warty/restricted Release
Hit http://archive.ubuntu.com warty/main Packages
Hit http://archive.ubuntu.com warty/main Release
Hit http://archive.ubuntu.com warty/restricted Packages
Hit http://archive.ubuntu.com warty/restricted Release
Hit http://archive.ubuntu.com warty/universe Packages
Hit http://archive.ubuntu.com warty/universe Release
Get:3 http://archive.ubuntu.com warty/main Sources [171kB]
Hit http://archive.ubuntu.com warty/main Release
Get:4 http://archive.ubuntu.com warty/restricted Sources [753B]
Hit http://archive.ubuntu.com warty/restricted Release
Hit http://archive.ubuntu.com warty/universe Sources
Hit http://archive.ubuntu.com warty/universe Release
99% [Working]
gzip: stdin: not in gzip format
Err http://archive.ubuntu.com warty/main Sources
  Sub-process gzip returned an error code (1)
99% [4 Sources gzip 0]
gzip: stdin: not in gzip format
Err http://archive.ubuntu.com warty/restricted Sources
  Sub-process gzip returned an error code (1)
Fetched 172kB in 1s (104kB/s)
Failed to fetch
http://archive.ubuntu.com/ubuntu/dists/warty/main/source/Sources.gz Sub-process
gzip returned an error code (1)
Failed to fetch
http://archive.ubuntu.com/ubuntu/dists/warty/restricted/source/Sources.gz
Sub-process gzip returned an error code (1)
Reading Package Lists... Done
E: Some index files failed to download, they have been ignored, or old ones used
instead.

Will attach strace output.

The most interesting thing is that when stracing using this line:
sudo strace -f -o /tmp/update.trace -s 2048 apt-get update

If I strace enough times, and then remove from partial whatever list fails and
then update again (but only when using strace) it eventually downloads the lists
correctly and no longer errors out.

[Shawn Walker (adonijah)]

Created an attachment (id=107)
strace output

[Shawn Walker (adonijah)]

I've verified that my ISP is not using a transparent proxy.

I can consistently download Sources.gz from the failed URLs listed in the
apt-get update output using wget or a browser, and then use gzip -t successfully.

The errors only occur when using apt-get update.

[Matt Zimmerman (mdz)]

Too many people have been reporting this for the usual explanations (broken
proxies, etc.) to entirely explain the symptoms, though I've never seen this
failure myself. Some of the others I'd seen were on mirrors, but this one is
archive.ubuntu.com.

Is it possible that a 5xx error is being returned or something? James, please
check the logs

[Matt Zimmerman (mdz)]

Looking at the strace log, the file received by the gzip method seems to contain
only NULs, at least the first few K which are displayed by strace

[Colin Watson (cjwatson)]

It was alleged on IRC that this might be amd64-specific, although I haven't yet
been able to try to gather backup evidence for this myself.

[Matt Zimmerman (mdz)]

I've never seen it on amd64 either

[Glen Stampoultzis (gstamp)]

I'm getting this very frequently on a p4. Also apt-get install will frequently
stall during installation. Makes the system pretty much unusable.

[James Troup (elmo)]

(In reply to comment #7)
> I'm getting this very frequently on a p4. Also apt-get install will frequently
> stall during installation. Makes the system pretty much unusable.

I can't reproduce this at all and there's nothing in the apache logs to indicate
any problems at all. The Sources.gz files are rsynced from the master archive
which means they're atomically renamed from .*mumble* to Sources.gz so I don't
see how it could be related to archive pulses either. Glenn, can you provide
the IP address you'll be coming from please?

[Shawn Walker (adonijah)]

I just did an apt-get update doing this:

sudo -s
apt-get update

and received the error, at 11:25 am CST, Sep 23rd, 2004

I'm currently using IP address 65.26.82.239 althoug that changes from time to time

[Glen Stampoultzis (gstamp)]

My IP is 165.228.129.11. It's not fixed however. When my router is reconnected
it could change.

[Matt Zimmerman (mdz)]

The fact that it works when using wget generally indicates that the problem is
due to apt's use of persistent connections and pipelining, which are sometimes
not implemented correctly by HTTP-speaking devices. I'm fairly certain there
isn't a bug in apt here, and there certainly isn't an archive problem.

A good way to test is to use:

sudo apt-get -o Acquire::http::Pipeline-Depth=1 update

if that works, and a plain apt-get update fails, then something between you and
the server is mangling the conversation.

[Shawn Walker (adonijah)]

The bad news is that doesn't fix it :) And really, it isn't sporadic, it's all
the time that it doesn't work when you run it normally...

[Glen Stampoultzis (gstamp)]

Strangely this is all working perfectly for me today (Sep 28) whether I use your
Pipeline trick or not. Will post back if I have further problems down the track.

[John Hornbeck (hornbeck)]

I was getting this same error and stopped it by changing all of the deb-src
directories from http to ftp. I can change back to give error report if need be.

[Matt Zimmerman (mdz)]

Are any of you able to reproduce the failure with:

wget -O- http://archive.ubuntu.com/ubuntu/dists/warty/main/source/Sources.gz |
gunzip >/dev/null

by trying it repeatedly?

I've never heard reports of this in Debian, but google finds a number of
messages about it. James, have you ever heard of this happening to people using
a Debian mirror?

It seems almost certainly ISP-related, given that it happens to some folks and
not to others...unfortunately it's nearly impossible to tell what sort of
equipment might be mangling the connection

[John Hornbeck (hornbeck)]

It works fine when I try using that method.

[Shawn Walker (adonijah)]

It works using a browser, wget, etc. Only fails when using apt-get.

[Shawn Walker (adonijah)]

I sincerely doubt that it's ISP related, given that I've never had this problem
with Debian, nor the apt installation I have on my RedHat install.

[Matt Zimmerman (mdz)]

(In reply to comment #18)
> I sincerely doubt that it's ISP related, given that I've never had this problem
> with Debian, nor the apt installation I have on my RedHat install.

Debian and Ubuntu's apt packages are identical apart from translation updates,
so if your experience implies that this problem doesn't exist under Debian, it's
not a bug in apt.
What else is different between your Debian and Ubuntu systems?

[Shawn Walker (adonijah)]

It was a much older version of Debian, which I no longer have installed. I'm not
sure it's a bug in apt either, but I know it's a bug. The key is that if I run
apt-get using strace, and then remove the file it places into partial/ and then
run update again, it succesfully updates some of the things it couldn't update
before. This to me implies a race condition or threading problem.

[Shawn Walker (adonijah)]

BTW, I am a programmer. So, if you need me to recompile with debuginfo, do a
stop at a certain point, do a backtrace, or dump through certain variables
somewhere I can do that. I haven't had the motivation to dig that deep
personally yet. Speaking of which, how does one go about building a debian apt
package with debug enabled ... ?

[Matt Zimmerman (mdz)]

(In reply to comment #20)
> It was a much older version of Debian, which I no longer have installed. I'm not
> sure it's a bug in apt either, but I know it's a bug. The key is that if I run
> apt-get using strace, and then remove the file it places into partial/ and then
> run update again, it succesfully updates some of the things it couldn't update
> before. This to me implies a race condition or threading problem.

The fact that it only happens some of the time certainly makes it sound like a
race condition. The trouble with this proposition is that the overall download
process is essentially synchronous: apt spawns a child process which performs
the download, and when it has completed, it spawns gzip to decompress the file.
 No threads, no locks, just a bit of polling to wait for completion.

What is especially strange is that this seems to happen only with certain files.
 No one has seen it happen when downloading actual .debs, or even Packages.gz
files as far as I know: only Sources.gz files. apt downloads these files using
the same code.

I would be interested to see a copy of the file from partial/ after one of these
failed downloads. Is gzip able to decompress it? Does it contain any data
besides nulls?

What type of filesystem are you using for /var?

[Matt Zimmerman (mdz)]

I noticed this thread on debian-user:

http://lists.debian.org/debian-user/2003/07/msg00751.html

(which, as expected, targets a broken proxy as the cause)

I would be interested to know the results if you applied the patch from that thread.

[Shawn Walker (adonijah)]

To answer your question, the gzip is just nulls as far as I can tell, and gzip
cannot decompress it.

reiserfs is the filesystem for / (where var is).

And to answer this bug possibly, I just solved my problem by:

1) deleting /var/cache/apt/*bin (i've done this before, don't get excited)

2) look at my /etc/apt/sources.list file, see if you spot the problem i noticed :)
deb cdrom:[Ubuntu 4.10 _Warty Warthog_ - Unofficial amd64 Binary-1 (20040915)]/
unstable main restricted

deb http://archive.ubuntu.com/ubuntu warty main restricted
deb-src http://archive.ubuntu.com/ubuntu warty main restricted

## Uncomment the following two lines to fetch updated software from the network
## and be able to use more than 12000 unsupported packages from the universe
archive.
deb http://archive.ubuntu.com/ubuntu warty main restricted universe
deb-src http://archive.ubuntu.com/ubuntu warty main restricted universe

deb http://security.ubuntu.com/ubuntu warty-security main restricted
deb-src http://security.ubuntu.com/ubuntu warty-security main restricted

3) I commented the earlier entries for main/restricted and saved sources.list

4) did an apt-get update, worked first try

I feel stupid, but I guess I just read the file comments about uncommenting the
below lines and forgot to comment the above. My guess is this is what created
the race condition, if apt uses pipelining as you suggest, at some point it was
probably trying to download the file and update it from the same source twice
causing one to corrupt the other...

So, I guess change the default sources.list to make sure the user knows to
comment or remove the other two lines and make apt smarter about this situation
in case the user does something stupid? :D

[Matt Zimmerman (mdz)]

I was afraid that something like this would happen, when we provided the
examples in that form. apt relies on the uniqueness of the entries in order to
form unique filenames for the download.

Thanks for tracking it down; I'll leave this bug open until there is some way to
warn the user when they have made this mistake
(http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=249509).

[Shawn Walker (adonijah)]

I should note that apt *already* warns users when they have duplicate
sources.list entries, but I didn't see that warning when it had 'subprocess
errors from gzip' trying to download the various sources.gz files, whether
through an error on my part, or because fatal errors cause normal warnings not
to show I don't know.

I just think it should fatal if they have duplicate entries instead of warning
to prevent the corruption that it causes apparently.

[Colin Watson (cjwatson)]

The duplicate entries have been removed from the default sources.list in new
installs.

[mike (mrevelle)]

(In reply to comment #14)
> I was getting this same error and stopped it by changing all of the deb-src
> directories from http to ftp. I can change back to give error report if need be.

I replaced http:// with ftp:// for all archive.* sites and it worked. You have
just saved me the problem of reinstalling ubuntu after a full week of
troubleshooting

[Matt Zimmerman (mdz)]

(In reply to comment #28)
> (In reply to comment #14)
> > I was getting this same error and stopped it by changing all of the deb-src
> > directories from http to ftp. I can change back to give error report if
need be.
>
> I replaced http:// with ftp:// for all archive.* sites and it worked. You have
> just saved me the problem of reinstalling ubuntu after a full week of
> troubleshooting

Reinstall Ubuntu? The fix for this problem is to remove the duplicate entries;
it's coincidental that ftp doesn't trigger the problem.

[mike (mrevelle)]

(In reply to comment #29)
> (In reply to comment #28)
> > (In reply to comment #14)
> > > I was getting this same error and stopped it by changing all of the deb-src
> > > directories from http to ftp. I can change back to give error report if
> need be.
> >
> > I replaced http:// with ftp:// for all archive.* sites and it worked. You have
> > just saved me the problem of reinstalling ubuntu after a full week of
> > troubleshooting
>
> Reinstall Ubuntu? The fix for this problem is to remove the duplicate entries;
> it's coincidental that ftp doesn't trigger the problem.
>

Nope, I've used many different sources.list files among other troubleshooting
methods with the help of people on FreeNode and
the only resolve for it was changing all http:// addresses regarding archive.*
to ftp:// - my current (also the sources.list
i was using when this problem started) is from http://paste.ubuntulinux.nl/969

[Matt Zimmerman (mdz)]

(In reply to comment #30)
> (In reply to comment #29)
> > (In reply to comment #28)
> > > (In reply to comment #14)
> > > > I was getting this same error and stopped it by changing all of the deb-src
> > > > directories from http to ftp. I can change back to give error report if
> > need be.
> > >
> > > I replaced http:// with ftp:// for all archive.* sites and it worked. You have
> > > just saved me the problem of reinstalling ubuntu after a full week of
> > > troubleshooting
> >
> > Reinstall Ubuntu? The fix for this problem is to remove the duplicate entries;
> > it's coincidental that ftp doesn't trigger the problem.
> >
>
> Nope, I've used many different sources.list files among other troubleshooting
> methods with the help of people on FreeNode and
> the only resolve for it was changing all http:// addresses regarding archive.*
> to ftp:// - my current (also the sources.list
> i was using when this problem started) is from http://paste.ubuntulinux.nl/969
>

If you were truly experiencing these symptoms with that sources.list, then you
were experiencing a different bug, and I'd appreciate further analysis.

[Carthik Sharma (carthik)]

Can anyone reproduce this issue with the latest Dapper? I cannot seem to, inspite of trying multiple times.

If this is fixed can it please be marked as such?

Thanks for reporting this bug, and following up on the issue.

[Shawn Walker (adonijah)]

Unfortunately, I haven't run Ubuntu for quite some time, and I'm currently not in a position to install Ubuntu and try to reproduce this again at the moment. I may be able to try again sometime in the next month. But, I can't at the moment. Thanks for following up though.

[SVAKSHA (svaksha)]

Hello Shawn,
Further to your comment on 2006-04-10, I am closing this bug since Warty will not be supported after April2006. Please reopen the bug report if the problem persists in Dapper. Thanks for reporting this bug.