apt's lists/partial fills disk

[Shahbaz Youssefi]

Test case:
- there is a artificial test in test/integration now
- as this is server side dependent a regression test should be enough

Since I upgraded to vivid, apt fills the disk space at /var/lib/apt/lists/partial. This was triggered by the automatic update every day, and as a solution I killed the processes named `http` spawned by apt, and removed the offending file from lists/partial manually. Until now, `apt-get update` and `apt-get upgrade` had been working. However, today, when I tried `apt-get update`, it got stuck in some file, continuously increasing its size when at 100%.

I tried inspecting the file to see if there is a repetition of data, and there seems to be, as I have suspected.

First of all, the file is it.archive.ubuntu.com_ubuntu_dists_vivid_main_binary-i386_Packages.bz2 this time, but I'm not sure if the previous time it was also this file.

I used `tail` and `hexdump` to figure out whether a certain pattern shows up in the file. Here are some examples:

$ tail -c 200000 it.archive.ubuntu.com_ubuntu_dists_vivid_main_binary-i386_Packages.bz2 | hexdump | grep '\<bbdf\>'
0000c80 8d9e 63ad bbdf 40a7 3f3e 2b4a 9b95 aa5a
0010c80 8d9e 63ad bbdf 40a7 3f3e 2b4a 9b95 aa5a
0020c80 8d9e 63ad bbdf 40a7 3f3e 2b4a 9b95 aa5a
0030c80 8d9e 63ad bbdf 40a7 3f3e 2b4a 9b95 aa5a

$ tail -c 200000 it.archive.ubuntu.com_ubuntu_dists_vivid_main_binary-i386_Packages.bz2 | hexdump | grep '\<9b95\>'
0000c80 8d9e 63ad bbdf 40a7 3f3e 2b4a 9b95 aa5a
0010c80 8d9e 63ad bbdf 40a7 3f3e 2b4a 9b95 aa5a
0020c80 8d9e 63ad bbdf 40a7 3f3e 2b4a 9b95 aa5a
0030c80 8d9e 63ad bbdf 40a7 3f3e 2b4a 9b95 aa5a

$ tail -c 200000 it.archive.ubuntu.com_ubuntu_dists_vivid_main_binary-i386_Packages.bz2 | hexdump | grep '\<1234\>'
0002630 fd50 b08d 7082 1234 85b3 a61e 2921 e0cd
0012630 fd50 b08d 7082 1234 85b3 a61e 2921 e0cd
0022630 fd50 b08d 7082 1234 85b3 a61e 2921 e0cd

$ tail -c 200000 it.archive.ubuntu.com_ubuntu_dists_vivid_main_binary-i386_Packages.bz2 | hexdump | grep '\<76a8\>'
000fa90 747d 76a8 078a 67b6 60d9 83f5 5ff3 787c
001fa90 747d 76a8 078a 67b6 60d9 83f5 5ff3 787c
002fa90 747d 76a8 078a 67b6 60d9 83f5 5ff3 787c

As you can see, it seems that after the error, a certain pattern seems to repeat every exactly 0x10000 bytes. I attached a copy of those 0x10000 bytes that keep repeating (from an arbitrary offset).

---

$ lsb_release -rd
Description: Ubuntu Vivid Vervet (development branch)
Release: 15.04

$ apt-cache policy apt
apt:
  Installed: 1.0.9.7ubuntu4
  Candidate: 1.0.9.7ubuntu4
  Version table:
 *** 1.0.9.7ubuntu4 0
        500 http://it.archive.ubuntu.com/ubuntu/ vivid/main amd64 Packages
        100 /var/lib/dpkg/status

---

Do let me know if I can test further to see what could be wrong. I'll keep an eye next times the behavior repeat to make sure the file is the same / the pattern is the same and I'll report back. If you think the problem could be the Italy server, let me know and I'll try with a different one.

ProblemType: Bug
DistroRelease: Ubuntu 15.04
Package: apt 1.0.9.7ubuntu4
ProcVersionSignature: Ubuntu 3.19.0-13.13-generic 3.19.3
Uname: Linux 3.19.0-13-generic x86_64
NonfreeKernelModules: nvidia
ApportVersion: 2.17-0ubuntu2
Architecture: amd64
CurrentDesktop: Unity
Date: Fri Apr 17 00:04:05 2015
SourcePackage: apt
UpgradeStatus: Upgraded to vivid on 2015-03-28 (19 days ago)

[Shahbaz Youssefi (shabbyx)]

This is the next day, and this time the offending file is: it.archive.ubuntu.com_ubuntu_dists_vivid_restricted_binary-amd64_Packages.bz2

The repeating data is still 0x10000 bytes, but the contents are different (another copy attached).

[Shahbaz Youssefi (shabbyx)]

I changed the server to US and the problem doesn't seem to occur any more. Perhaps this is an issue with the Italy server, but nevertheless it is strange that apt would continuously write to the disk due to an error on the server.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Cimdrap (giosaci)]

i was on italian server too with same issue. Just migrated to principal server.....let's see what happens

[Alberto Salvia Novella (es20490446e)]

Please:
  1. Report to <https://www.debian.org/Bugs/>.
  2. Paste the new report URL here.
  3. Set this bug status back to "confirmed".

Thank you.

[Brian Murray (brian-murray)]

I've been unable to recreate this on a Vivid amd64 chroot after adding i386 as an architecture.

[Brian Murray (brian-murray)]

What IP address are you connecting to? There may be multiple servers for it.archive.ubuntu.com and only one may be a problem.

[Matteo Croce (teknoraver)]

same here:

$ ll /var/lib/apt/lists/partial/it.archive.ubuntu.com_ubuntu_dists_vivid-updates_universe_binary-amd64_Packages.bz2
-rw-r--r-- 1 root root 95G apr 25 14:37 /var/lib/apt/lists/partial/it.archive.ubuntu.com_ubuntu_dists_vivid-updates_universe_binary-amd64_Packages.bz2

[Andreas Brudin (andreas-brudin)]

I have the same problems, using the Swedish update servers. I booted my computer and leaved it for lunch, and when I came back the CPU was running at 100% (http by apt). I kept it running for a while, and then I starting investigating this issue. I checked my /var/lib/apt/lists/partial/ and noticed that I had a file there over 100G big and growing all the time, so I rebooted, deleted the file and changed to the principal servers.

[Brian Murray (brian-murray)]

For people experiencing this bug it'd be useful to know specifically what mirror (domain name) they are connecting to and which IP address for that domain name.

[Anton Blanchard (anton-samba)]

I've seen this a few times. It appears to be an issue with recovering partially downloaded files, and my workaround was to rm /var/lib/apt/lists/partial/*

Looking through some notes from a few weeks ago, the backtrace was:

(gdb) backtrace
#0 0x00003fffb5b5d6ec in SHA1Transform (state=state@entry=0x3fffecb9b5e0,
    buffer=buffer@entry=0x3fffecb97043 "f\350'\315\060\372C343\345\030\301b\212'\371\376\271\202\237uiF\247\325\361\245\017o\277\344\031\245\275\370\361\256=C\033\277Y\b\242\021\257I\233\245\321FU\220\216{(\327J\nlB\021\364\360\373\311\066$\304\303\001\031\\\352Þ‚\024\271\\\321\f\023\255\342tK?b\f\361\306j\037\325\323\nH\036\061\026J\313A0d\306Bo\006\031\246q8\361W\247b@\243\373\322\316a\240y\226tC\213\271#'\"\301N\030pDb\203D\234gEz\237\031\341\062\020ÄŠÙªK\276\232\201!\332T\227\\\026\242\034\270\031ÛŒ\273\272\036\033\006\257q\255u\350\243*\332\373\256\256\205m1 \213\266*\\eN\033\272a"...)
    at /build/buildd/apt-1.0.9.7ubuntu4/apt-pkg/contrib/sha1.cc:108
#1 0x00003fffb5b5ec9c in SHA1Summation::Add (this=0x3fffecb9b598,
    data=0x3fffecb87c10 "n\202\341\067\231\267g\251N\316\324[0\361\070\230`sxtй\300b\200Y\347\252V\323 \310ƃ\273I'\236", len=65536)
    at /build/buildd/apt-1.0.9.7ubuntu4/apt-pkg/contrib/sha1.cc:262
#2 0x00003fffb5c7a6e0 in Add (Size=65536,
    Data=0x3fffecb87c10 "n\202\341\067\231\267g\251N\316\324[0\361\070\230`sxtй\300b\200Y\347\252V\323 \310ƃ\273I'\236", this=0x3fffecb9b530)
    at ../build/include/apt-pkg/hashes.h:167
#3 CircleBuf::Write (this=0x3fffecb83b90, Fd=4)
    at /build/buildd/apt-1.0.9.7ubuntu4/methods/http.cc:222
#4 0x00003fffb5c7b518 in HttpServerState::Go (this=0x3fffecb83960, File=
    0x3fffecb81980, ToFile=true)
    at /build/buildd/apt-1.0.9.7ubuntu4/methods/http.cc:652

We were stuck in the circular buffer code, the input and output pointers got out of whack such that we would write the same buffer continually.

[diverbelow (groovygravy10-gmail)]

I also experiencing this issue 15.04 64 bit even with picking one mirror. Until this bug has been fixed, I had to setup an hourly cron job, to remove everything in partial directory.

[bart (bart-wp)]

I'm randomly experiencing this problem using the German servers for 15.04. It does not affect a specific package, each time this happens some package keeps on growing until the disk is full.

[Alex Kretzschmar (alexktz)]

I can make this issue come and go by disabling the gnome3.16 PPAs.

[Alex Kretzschmar (alexktz)]

Specifically this occurs when a 404 for a particularly repo (i.e. vivid not available in a PPA). Then a second apt-get update causes the /partial dir to fill up until my root is full.

[Anton Blanchard (anton-samba)]

I took another look at this. The problem begins in ServerState::HeaderLine() Where we parse a Content-Range header:

Content-Range: bytes 587-600/601

So we set StartPos to 587. Then in HttpServerState::RunData() we do

In.Limit(Size - StartPos);

where Size is 14. We pass a negative number into Limit() which means MaxGet is set to something less than OutP, and we loop forever.

[Anton Blanchard (anton-samba)]

I think I see the issue. We parse both Content-Length: and Content-Range: headers.

When parsing the Content-Range: header, we set Size to the total size of the file.

If the Content-Length header comes last, we reset Size to the size of the data we are receiving, not the total size of the file.

The attached patch fixes it for me, although we do get the infamous Hash sum mismatch which clears after rerunning apt-get:

Failed to fetch http://ports.ubuntu.com/ubuntu-ports/dists/vivid-security/universe/i18n/Translation-en Hash Sum mismatch

Another issue, but I'd love to fix that one day.

[Adam Conrad (adconrad)]

Michael, can you have a look at this one?

[RASHID (chrash786)]

i already get rid of this issue. i tried many things including auto remove
apt.....now this /var/lib/apt/lists/partial file receives some file while i
make some update, but after update the new files are removed by itself. I
dont know which app is doing this activity.. Please make a query to an
expert.

On Wed, May 13, 2015 at 3:21 AM, Anton Blanchard <email address hidden> wrote:

> I took another look at this. The problem begins in
> ServerState::HeaderLine() Where we parse a Content-Range header:
>
> Content-Range: bytes 587-600/601
>
> So we set StartPos to 587. Then in HttpServerState::RunData() we do
>
> In.Limit(Size - StartPos);
>
> where Size is 14. We pass a negative number into Limit() which means
> MaxGet is set to something less than OutP, and we loop forever.
>
> --
> You received this bug notification because you are subscribed to a
> duplicate bug report (1448684).
> https://bugs.launchpad.net/bugs/1445239
>
> Title:
> apt's lists/partial fills disk
>
> Status in APT:
> New
> Status in apt package in Ubuntu:
> Confirmed
> Status in apt source package in Vivid:
> Confirmed
>
> Bug description:
> Since I upgraded to vivid, apt fills the disk space at
> /var/lib/apt/lists/partial. This was triggered by the automatic update
> every day, and as a solution I killed the processes named `http`
> spawned by apt, and removed the offending file from lists/partial
> manually. Until now, `apt-get update` and `apt-get upgrade` had been
> working. However, today, when I tried `apt-get update`, it got stuck
> in some file, continuously increasing its size when at 100%.
>
> I tried inspecting the file to see if there is a repetition of data,
> and there seems to be, as I have suspected.
>
> First of all, the file is it.archive.ubuntu
> .com_ubuntu_dists_vivid_main_binary-i386_Packages.bz2 this time, but
> I'm not sure if the previous time it was also this file.
>
> I used `tail` and `hexdump` to figure out whether a certain pattern
> shows up in the file. Here are some examples:
>
> $ tail -c 200000
> it.archive.ubuntu.com_ubuntu_dists_vivid_main_binary-i386_Packages.bz2 |
> hexdump | grep '\<bbdf\>'
> 0000c80 8d9e 63ad bbdf 40a7 3f3e 2b4a 9b95 aa5a
> 0010c80 8d9e 63ad bbdf 40a7 3f3e 2b4a 9b95 aa5a
> 0020c80 8d9e 63ad bbdf 40a7 3f3e 2b4a 9b95 aa5a
> 0030c80 8d9e 63ad bbdf 40a7 3f3e 2b4a 9b95 aa5a
>
> $ tail -c 200000
> it.archive.ubuntu.com_ubuntu_dists_vivid_main_binary-i386_Packages.bz2 |
> hexdump | grep '\<9b95\>'
> 0000c80 8d9e 63ad bbdf 40a7 3f3e 2b4a 9b95 aa5a
> 0010c80 8d9e 63ad bbdf 40a7 3f3e 2b4a 9b95 aa5a
> 0020c80 8d9e 63ad bbdf 40a7 3f3e 2b4a 9b95 aa5a
> 0030c80 8d9e 63ad bbdf 40a7 3f3e 2b4a 9b95 aa5a
>
> $ tail -c 200000
> it.archive.ubuntu.com_ubuntu_dists_vivid_main_binary-i386_Packages.bz2 |
> hexdump | grep '\<1234\>'
> 0002630 fd50 b08d 7082 1234 85b3 a61e 2921 e0cd
> 0012630 fd50 b08d 7082 1234 85b3 a61e 2921 e0cd
> 0022630 fd50 b08d 7082 1234 85b3 a61e 2921 e0cd
>
> $ tail -c 200000
> it.archive.ubuntu.com_ubuntu_dists_vivid_main_binary-i386_Packages.bz2 |
> hexdump | grep '\<76a8\>'
> 000fa90 747d 76a8 078a 67b6 60d9 83f5 5ff3 787c
> 001fa90 747d 76a8 078a 6...

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "Don't parse Content-Length if Content-Range has been parsed" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Alex Kretzschmar (alexktz)]

Issue occurred again for me today when I interrupted apt-get update and then reran a few minutes later.

[Michael Vogt (mvo)]

This is fixed in git now and I will prepare a wily and vivid-proposed upload.

[Michael Vogt (mvo)]

Thanks a lot to Anton for the patch and the excellent analysis of the issue!

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 1.0.9.10ubuntu1

---------------
apt (1.0.9.10ubuntu1) wily; urgency=low

  * merged from debian/sid

apt (1.0.9.10) unstable; urgency=medium

  [ Michael Vogt ]
  * Fix crash in pkgDPkgPM::WriteApportReport(() (LP: #1436626)
  * Move sysconf(_SC_OPEN_MAX); out of the for() loop to avoid unneeded
    syscalls
  * Fix endless loop in apt-get update that can cause disk fillup
    (LP: #1445239)

  [ Helmut Grohne ]
  * parse arch-qualified Provides correctly (Closes: 777071)

 -- Michael Vogt <email address hidden> Fri, 22 May 2015 18:08:10 +0200

[Kyle Beauchamp (kyleabeauchamp)]

I have a fully updated 15.04 machine and I'm still having issues with apt cache eating up my disk. I get into a situation where `http` is my most active process and my apt partial cache is growing quickly. See the logs below, which show the top processes, the large file sizes in the partial cache, and the file size growth in the partial cache. When the disk fills completely, the system is then no longer able to boot into unity and I am required to manually delete files with a command line login. ..

top:
 2581 root 20 0 40260 5012 4660 R 97.0 0.0 11:04.46 http
 2582 root 20 0 40260 4992 4636 R 97.0 0.0 10:40.69 http

:~$ ls /var/lib/apt/lists/partial/ -lS|head -n 5
total 93832500
-rw-r--r-- 1 root root 49198669906 May 29 10:41 security.ubuntu.com_ubuntu_dists_vivid-security_universe_binary-i386_Packages.bz2
-rw-r--r-- 1 root root 46855568434 May 29 10:41 us.archive.ubuntu.com_ubuntu_dists_vivid-updates_main_binary-i386_Packages.bz2
-rw-r--r-- 1 root root 7017495 Apr 24 14:45 us.archive.ubuntu.com_ubuntu_dists_vivid_universe_source_Sources.bz2
-rw-r--r-- 1 root root 6486447 Apr 24 14:40 us.archive.ubuntu.com_ubuntu_dists_vivid_universe_binary-i386_Packages.bz2
:~$ ls /var/lib/apt/lists/partial/ -lS|head -n 5
total 94526900
-rw-r--r-- 1 root root 49566396416 May 29 10:41 security.ubuntu.com_ubuntu_dists_vivid-security_universe_binary-i386_Packages.bz2
-rw-r--r-- 1 root root 47198911538 May 29 10:41 us.archive.ubuntu.com_ubuntu_dists_vivid-updates_main_binary-i386_Packages.bz2
-rw-r--r-- 1 root root 7017495 Apr 24 14:45 us.archive.ubuntu.com_ubuntu_dists_vivid_universe_source_Sources.bz2
-rw-r--r-- 1 root root 6486447 Apr 24 14:40 us.archive.ubuntu.com_ubuntu_dists_vivid_universe_binary-i386_Packages.bz2

[Brian Murray (brian-murray)]

This has only been fixed in the development release of Ubuntu, Wily Werewolf, and we are awaiting a fix from the developer for Vivid (Ubuntu 15.04).

[Michael Vogt (mvo)]

This is uploaded as a SRU and in https://launchpad.net/ubuntu/vivid/+queue?queue_state=1&queue_text=

[Brian Murray (brian-murray)]

Hello Shahbaz, or anyone else affected,

Accepted apt into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt/1.0.9.7ubuntu4.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Shahbaz Youssefi (shabbyx)]

Hi Brian,

Sorry for my lack of responsiveness, but I'd essentially been moving countries and I'm no longer in Italy to begin with. On another computer where I'm typing this response, this issue doesn't happen to begin with (Canada servers). Hopefully others would be able to test this out.

[Alexander List (alexlist)]

Brian,

I've run across the same problem and have installed your package from -proposed.

I will let you know after 2-3d if the problem resurfaces. If you don't hear from me by the end of the week, consider the problem fixed for me.

[Alexander List (alexlist)]

Brian,

I was using sg.archive.u.c

[Xavi Ivars (xavi-ivars)]

It's still affecting me. I'm using the CICA server (in Spain)

[Brian Murray (brian-murray)]

@Xavi could you test the proposed package as suggested in comment #29?

[Josh Holtrop (jholtrop)]

I had been observing a disk full a few times due to this issue, including this morning, after which I found this report. I just installed apt 1.0.9.7ubuntu4.1 from vivid-proposed using "apt-get install apt/vivid-proposed".

I now have:
$ apt-cache policy apt
apt:
  Installed: 1.0.9.7ubuntu4.1
  Candidate: 1.0.9.7ubuntu4.1
  Version table:
 *** 1.0.9.7ubuntu4.1 0
        400 http://us.archive.ubuntu.com/ubuntu/ vivid-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     1.0.9.7ubuntu4 0
        500 http://us.archive.ubuntu.com/ubuntu/ vivid/main amd64 Packages

I ran "apt-get update" manually several times and did not see the disk-filling-up problem again. I will continue to keep my eye on it.

[Adam Conrad (adconrad)]

Marking verification-done based on comments 31 and 35.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 1.0.9.7ubuntu4.1

---------------
apt (1.0.9.7ubuntu4.1) vivid-proposed; urgency=low

  * test/integration/test-apt-download-progress:
    - disable as its too unreliable
  * Fix endless loop in apt-get update that can cause disk fillup
    LP: #1445239

 -- Michael Vogt <email address hidden> Wed, 10 Jun 2015 16:09:44 +0200

[Adam Conrad (adconrad)]

The verification of the Stable Release Update for apt has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.