"readline() on closed filehandle FILES" warnings caused by multiarch

[Fabien Tassin]

Binary package hint: popularity-contest

/etc/cron.weekly/popularity-contest in natty now shows:
readline() on closed filehandle FILES at /usr/sbin/popularity-contest line 104.
readline() on closed filehandle FILES at /usr/sbin/popularity-contest line 104.
readline() on closed filehandle FILES at /usr/sbin/popularity-contest line 104.
...

looking at the sources:

open PACKAGES, "dpkg-query --show --showformat='\${status} \${package}\\n'|";
while (<PACKAGES>)
{
  /^.*installed *(.+)$/ or next;
  my $pkg=$1;
  $popcon{$pkg}=[0,0,$pkg,"<NOFILES>"];
  open FILES, "$dpkg_db/$pkg.list";
  my $bestatime = undef;
  while (<FILES>) { <==================== 104

a simple printf shows why:

$ ls -l /var/lib/dpkg/info/zlib1g.list
ls: cannot access /var/lib/dpkg/info/zlib1g.list: No such file or directory
$ ls -l /var/lib/dpkg/info/zlib1g*.list
-rw-r--r-- 1 root root 227 2011-03-22 08:10 /var/lib/dpkg/info/zlib1g:amd64.list
-rw-r--r-- 1 root root 1267 2011-03-22 08:10 /var/lib/dpkg/info/zlib1g-dev.list

so it seems it's caused by the new multi-arch thingy.

ProblemType: Bug
DistroRelease: Ubuntu 11.04
Package: popularity-contest 1.51ubuntu1
ProcVersionSignature: Ubuntu 2.6.38-7.37-generic 2.6.38
Uname: Linux 2.6.38-7-generic x86_64
NonfreeKernelModules: nvidia
Architecture: amd64
Date: Thu Mar 24 20:57:57 2011
ProcEnviron:
 LANG=en_US.UTF-8
 LANGUAGE=en_US:en
 PATH=(custom, user)
 SHELL=/usr/bin/tcsh
SourcePackage: popularity-contest
UpgradeStatus: Upgraded to natty on 2009-07-28 (604 days ago)

[torgeh (l-launchpad-20-torgehusfeldt-spamgourmet-com)]

Hi,

had the same problem here, fixed it with the attached popularity-contest that tries again with the architecture-specific filename.

HTH

Torge H

[torgeh (l-launchpad-20-torgehusfeldt-spamgourmet-com)]

Hi,

sorry for the unrelated changes in my last post.

Now a real patch against "/usr/sbin/popularity-contest" as of "popularity-contest 1.51ubuntu1" with minimal changes.

The bug encountered raises a question though: why wasn't a "lint" rule along the lines of "never use open without interpreting the returned value" applied?

HTH

Torge H

[Andrey Bondarenko (abone)]

I've commited changes suggested by torgeh as a branch:
lp:~abone/ubuntu/natty/popularity-contest/fix-742017

By the way debian upstream also has a fix for the this in version 1.52, but they don't try to access arch file, just report if pkg.list is not accessible.

[torgeh (l-launchpad-20-torgehusfeldt-spamgourmet-com)]

Shoot,

I just realized two shortcomings of my first patch (from comment #3)
- the --showformat command-line argument was missing the crucial extra format-string (--showformat='\${status} \${package} \${package}:\${architecture}\\n')
- there is a much cleaner way to access the required information

See patch in comment #5

Sorry for the inconvenience

[Andrey Bondarenko (abone)]

Using dpkg-query is of course better in terms of compatibility and I would, usually, prefer it, but this change makes popularity-contest ugly slow. On my P4 3GHz box with about 2500 packages installed it takes about 5 minutes to complette, while previous version with direct file access complettes in 20 seconds.

Many xubuntu/lubuntu boxes have much less CPU power. Despite the fact popcon runs only once a day, I'm afraid many users would prefer not to enable it at all, if it will be so resource hungry. I'd better stick with direct file reading.

[torgeh (l-launchpad-20-torgehusfeldt-spamgourmet-com)]

Either that or "nice"-ing it -- as this is a background job it really doesn't matter when the results are available.
As long as it completes before the next run is started, of course.

Do you happen to know which resource is most strained by this patched installation?
If it is i/o we would need to ionice it, obviously.
> /usr/bin/ionice util-linux 2.17.2-9.1ubuntu4

Nonetheless, I will add a corrected version of the patch from comment #3

HTH,
Torge

[Andrey Bondarenko (abone)]

If you don't mind, I will subscribe ubuntu sponsors to this bug. We have a patch in "works for me" state and I've prepared it for merging into natty/oneiric branch. I suppose, applying this patch is better than current situation, where popcon loses all information about multiarch packages.

Forwarding patch to Debian is likely inappropriate. As I understand Debian, does not currently use dpkg multiarch.

[Stefan Pappalardo (sjuk)]

will there be a fix for natty?

[Andrey Bondarenko (abone)]

Fixes for natty and oneiric are proposed for merging. Both request are in pending state. Until then you may build package from a bzr branch. If you wish I may try to publish fixed package in my ppa, but I never did it before. :(

[Jamie Strandboge (jdstrand)]

Unsubscribing Ubuntu Sponsors team since Steve Langasek has assigned this to himself.

[Andrey Bondarenko (abone)]

The bug is fixed in Oneiric popularity-contest version 1.53ubuntu1

[Andrey Bondarenko (abone)]

I've updated related branch lp:~abone/ubuntu/natty/popularity-contest/fix-742017 with popularity-contest from version 1.53, should be enough to backport fix to Natty.

Binary package with the fixed vesion should be available soon in my ppa:
https://launchpad.net/~abone/+archive/ubuntu-fixes

[Steve Langasek (vorlon)]

If 1.53ubuntu1 fixed this bug, is the bug linkage to Debian wrong? The linked Debian bug #555790 is still open, whereas the changelog refers to Debian bug #622322.

[Andrey Bondarenko (abone)]

Bug linkage is ok. Debbug 555790 and debbug 622322 are both related to multiarch. As far as I know, until version 1.53 popularity-contest accessed to dkpg data directly to get list of files belonging to a package. After introduction of multiarch in dpkg several bugs arrised. Now popularity-contest always uses dpkg to get package info, so all the bugs related to direct dpkg database access becomes irrelevant and effectively fixed.

I'll try contact debian maintainers to clarify situation.