genhdlist2 doesn't reuse most of the existing hdlist

[Jeff Johnson]

rpmio needs partial reads? Likely not ...

[Paiiou (paiiou)]

Incorrect display in genhdlist2

I created apersonal mirror of Mageia-1

1- I do the updates with rsync.

2- I start the creation of files with gendistrib - blind

Here is the output in genhdlist2 with mageia (version 1 and version cauldron) :

genhdlist2 running - allow-empty-media - xml-info - synthesis-filter '. CZ: gzip -9' - xml-info-filter '. lzma: lzma -5' - file-deps / var / ftp/pub/personal/1/i586/media/media_info/file-deps / var/ftp/pub/personal/1/i586/media/core_release
filtering / var/ftp/pub/personal/1/i586/media/core_release/media_info/hdlist.cz Into hdlist.cz.tmp
Adding 14,102 new rpms not available in Existing hdlist

==> It displays the total number of files and not files added (0 file for release)

3- When I do the same with Mandriva 2010.2, the result is correct.

[Pascal Terjan (pterjan42)]

I had noticed that if fails to load part the existing hdlist and will re-add everything after the failing header, but did not have time to investigate yet :(

[Thierry-vignaud (thierry-vignaud)]

rpm -q genhdlist2 rpmtools?
Have you tried with 6.1-1.mga2?

[Thierry-vignaud (thierry-vignaud)]

Indeed URPM::parse_hdlist__XS() has rpm::headerRead() failing to read headers after a while.
The problem is either in generating hdlists or in reading them back.
As the position it stops reading hdlist is random, it would be the later.
It also happens with rpm-4.9.

one needs to look at what makes headerRead() pukes

[Thierry-vignaud (thierry-vignaud)]

@dexter: I think this is a regression of rpm due to the lost of ufd in rpmio.
It now expect only regular files, all the code managing regular files is now dead...
However, URPM::parse_hdlist__XS() uses a pipe as a fd...

[Paiiou (paiiou)]

(In reply to comment #2)
After some absence, I see you working on problem.
Just in case, I answer all the same questions

> rpm -q genhdlist2 rpmtools?
version 6.0-1.mga1

> Have you tried with 6.1-1.mga2?
I just test it. The problem remains

Regards

[Pascal Terjan (pterjan42)]

I tried to debug it, it seems headerRead expects timedRead(fd, (char *)&ei[2], len) to always read the full requested size but it does not (it often only gets 4992 bytes).

Just to test I changed the code to be

    if ((a=timedRead(fd, (char *)&ei[2], len)) != len) {
        int p = 0;
        while ((a > 0) && (a < len)) {
                len -= a;
                p += a;
                a = timedRead(fd, (char *)&ei[2]+p, len);
        }
        if (a < len) {
                goto exit;
        }
    }

And it can now load all the headers.

[Thierry-vignaud (thierry-vignaud)]

Created attachment 1181
test case from Panu Matilainen

[Thierry-vignaud (thierry-vignaud)]

Created attachment 1182
patch from Panu Matilainen that makes headerRead works properly again for the previous test case

[Thierry-vignaud (thierry-vignaud)]

Created attachment 1183
old patch of mine trying to use librpm, now working fine with previous rpm patch

[Thierry-vignaud (thierry-vignaud)]

Created attachment 1184
cleaning API after previous URPM patch

[Thierry-vignaud (thierry-vignaud)]

@Dexter & Pascal: Ping

[Jeff Johnson (n3npq)]

tracked at https://bugs.launchpad.net/rpm/+bug/913203

[Thierry-vignaud (thierry-vignaud)]

rpm-4.9.1.2-15.mga2 & perl-URPM-3.38.5-1.4 just pushed into core/updates_testing
Please test.

[Mageia (mageia)]

The patch you added in perl-URPM-3.38.5-1.4 is broken, it does not pass test anymore.

Also, please do not add it as a patch in the perl-URPM package, but commit directly in the /soft SVN

[Thierry-vignaud (thierry-vignaud)]

It is...

And it won't be commited in soft until we agree to go with this rpm patch

[Mageia (mageia)]

Well, perl-URPM does not build right now because of this patch.
Should I remove it?

[Thierry-vignaud (thierry-vignaud)]

perl-URPM-3.36 has been uploaded...

[Mageia (mageia)]

Thanks for the fix.
Though, I would like to have in my in core/release, not core/updates_testing.
We should probably remove your patch in the current/ branch, and you could work from another branch.

[Thierry-vignaud (thierry-vignaud)]

Upload in progress.

As for removing the patch, nope.
The patched method doesn't work anymore at least since we switched from rpm-4.6.x to rpm-4.8.x anyway...

[Paiiou (paiiou)]

Hey,

I just made an update rpm packages (testing) rpm, lib64rpm and perl-URPMI.

I made several tests to build my local mirror, after having made partial updates.
At home, the result is conclusive. Thank you all.

Packages can they be carried in core/release?
These corrections can they be made to Mageia 1?

[Thierry-vignaud (thierry-vignaud)]

rpm & perl-URPM are just being pushed into core/release for mga2