urpmi attempting to use repo-md

[Jeff Johnson]

URPMI in Mageia is attempting to use repo-md as well
as synthesis/hflists tracker.

[Cjw-v (cjw-v)]

Mageia 2 spec #034
also see https://mageia.org/wiki/doku.php?id=iso2:technical_specification

Mageia repositories currently have a media_info dir which contains a
line-oriented 'synthesis' file, plus some XML files with additional
information like package descriptions, and a hdlist which AFAIK is not
used by urpmi.

Fedora and opensuse repositories do not use this format for metadata but
instead have a repodata dir containing a 'primary' and other XML files,
which are all listed in a 'rpomd.xml' file.

The two schemes are similar which is not very surprising because the
requirements are the same. Most package managers with rpm support can use
the repomd style but urpmi can't. Only a few package managers support
synthesis style metadata. Switching to this 'standard' repository metadata
would give people more choice: use other package managers (e.g. yum or zypper)
in mageia and use urpmi on other distros. In the long term this should help
make urpmi easier to maintain: the "standard" metadata is a bit easier to extend,
urpmi will be more a standard tool: behavior can easily be compared to other
package managers, test suites may be shared.

Some not very useful index size numbers for mga cauldron x86_64 core/release :
1.7M synthesis.hdlist.cz
1M info.xml.lzma
5,6M changelog.xml.lzma
8,7M files.xml.lzma

4,4M primary.xml.gz
3,4M other.xml.gz
11M filelists.xml.gz (8,8M when compressed with xz)

Goals:
- no negative impact for people who use the default package tools
- basic support for other package managers (yum, zypper, apt) in mageia 2
  (better support where packagekit uses the package manager the user has
  chosen is not part of this spec but could be a follow-up feature)

the plan is:
- modify urpmi to only support repomd metadata
- add repomd metadata to the cauldron repository while keeping synthesis
- upload the new urpmi
- after either the mga2 or mga3 release:
  drop synthesis/hdlist metadata from cauldron

so there will be 1 or 2 stable releases that carry both types of
repository metadata. The build infrastructure needs to support this of
course. This adds some complexity and uses extra space on the mirrors.

Things that need to be changed:
- urpmi perl code
- maybe a fast xml reader in C, like yum has ?
- rpmdrake/installer ?
- build system

Open questions:
- Is there anything I missed, unique urpmi features that will be broken by
  such a change, other expected problems?
- A volunteer is needed for writing the needed perl code for perl-URPM etc.,
  otherwise the change won't happen.

[Marja11 (marja11)]

a comment here https://mageia.org/wiki/doku.php?do=show&id=iso2%3Atechnical_specification :

sounds interesting but needs to have 2 kinds of metadata in parallel while it's not all integrated - discussions needed

[Michael Scherer (misc-zarb)]

I think this would be helpful, yes.

I only have 2 thing to add :
1) is the format of repomd specified ? I remember seeing developpers of packagekit and zypper being unhappy with a change of format.

2) can we use createrepo ? is this as fast our current tools ? The slow pace of update and indexes generation on fedora is IMHO problematic ( slow pace being that we have to wait 2 days to get updates-testing on mirrors ), but I do not know if the problem is their tools or something else.

[Marja11 (marja11)]

@ cjw

can you answer misc's questions?

[Cjw-v (cjw-v)]

AFAICT the format is standard but it can be extended.

I don't know enough about all the features that urpmi and GUI package tools provide in mageia. Createrepo provides at least the basic info required for package installation and upgrades. WRT speed we should probably be more concerned about the time it takes to read repomd data (in urpmi/perl-URPM).

[Jeff Johnson (n3npq)]

tracked at https://bugs.launchpad.net/rpm/+bug/913192

[J-manuel (j-manuel)]

No news

[Thierry-vignaud (thierry-vignaud)]

I'm against such a move.
yum metadata are bigger and slower to parse.

For updating packages, compare:
1.7M synthesis.hdlist.cz
4,4M primary.xml.gz

For retrieving metadata from rpmdrake, compare:
8,7M files.xml.lzma
11M filelists.xml.gz

And yum really is quite much slower than urpmi and I suspect the used metada format doesn't help...

As for the comment about recompressing with XZ, that wouldn't change the ration since synthesis would gain from changing compressor (we already support gz, bzip2, xz, ...)

Last but not least, it would have big impacts on perl-URPM, urpmi, rpmdrake and drakx installer.

[Jeff Johnson (n3npq)]

You can be for or against as you wish.

Meanwhile you are comparing apples to potatoes if only
citing file sizes with different compressions.

Neither repository markup is incremental either: which
forces repeated downloads of same old, same old.

The critical flaw in synthesis files is that there is no means to
represent file dependencies _EXCEPT_ by mapping into
Provides: which isn't even close to being the correct engineering.

Have fun!

[Pascal Terjan (pterjan42)]

Also did someone check generation time?

Opensuse does not uses it on factory but only on the different smaller repositories and stable releases because "createrepo on factory takes hours and we prefer to push out ftp trees quicker".

[Thierry-vignaud (thierry-vignaud)]

I think we should close this one as WONTFIX

[Marja11 (marja11)]

(In reply to comment #9)
> I think we should close this one as WONTFIX

@ misc
@ cjw

If there are no objections, I intend to close this report as WONTFIX two weeks from now.

Whoever wants to object: please remove the NEEDINFO keyword and give your motivation

[J-manuel (j-manuel)]

keep for future spec

[Marja11 (marja11)]

Hi,

This bug was filed against cauldron, but we do not have cauldron at the moment.

Please report whether this bug is still valid for Mageia 2.

Thanks :)

Cheers,
marja

[Marja11 (marja11)]

(In reply to comment #11)
> keep for future spec

OK :)

[Boklm (boklm)]

As there is no plan to change this soon, I think it should be closed. It can still be reopened later if things change.

[Marja11 (marja11)]

(In reply to comment #14)
> As there is no plan to change this soon, I think it should be closed. It can
> still be reopened later if things change.

@ Leuhmanu

Sorry, tv and boklm for closing: you're the minority

Closing as WONTFIX for now

[fraterlinux (frateraec)]

I would like to request the reopening of this bug!

Switch to "standard" rpm metadata for package repositories.

This will allow us to use YUM and APT-RPM that are in repositories Mageia. In addition to improving support for other managers.

Standardization is important and is a current trend.

[Pascal Terjan (pterjan42)]

Unless someone works on generating those metadata at a decent speed, this is not going to happen.

Currently uploading a package takes a few minutes, it would take a few hours.

[fraterlinux (frateraec)]

Please advise as the other distros rpm do and how to do Mageia justify the impossibility.

[Pascal Terjan (pterjan42)]

As I quoted, opensuse does not do it on their main development repository because it takes several hours.

We may be able to do like them and only to it for stable releases but then it would not be consistent with the development version and not be properly tested.

I don't know how Fedora does it

[fraterlinux (frateraec)]

It could be for Mageia 3 or maybe Mageia 4 stable release?

[Pascal Terjan (pterjan42)]

Reading from an old RH advisory from 2008 there are some options that may help:

=====
As well, this updated package adds a new option flag, "--skip-stat". When
used in conjunction with "--update", createrepo skips calling stat(1) on
files to see if they have changed, and assumes that if the file name
matches, then the repodata can be re-used.

This option has shown a significant increase in performance in the Fedora
build system, cutting down the time it takes createrepo to run, from 40
minutes, to approximately 4.
=====

4 minutes is still quite high but more reasonable.

When uploading a package we have to run it 4 times (i586, x86_64, debug i586, debug x86_64) and currently the total of 4 runs of genhdlist2 takes about 3 minutes (and I had started working on making it faster).

Let's test the performance with that options.

On my home machine on core/release i586 (20115 packages):

First generation with genhdlist2 --clean

real 1m52.755s
user 1m36.790s
sys 0m1.950s

Update with genhdlist2 (and no new package)

real 1m0.889s
user 1m36.080s
sys 0m1.540s

First generation with createrepo

real 10m49.949s
user 6m15.240s
sys 0m12.380s

Update with createrepo --update --skip-stat (and no new package)

real 0m56.652s
user 0m49.190s
sys 0m3.470s

So creating from scratch (which does not happen so often, mostly when we have bugs causing signature problems) takes 5 times longer, for the 4 repositories it would be 44 minutes instead of 7 minutes 30 but updating incrementally (which happens at each package upload) is slightly faster.

[Pascal Terjan (pterjan42)]

Actually createrepo --update without --skip-stat is fast too :)

real 0m55.352s
user 0m49.310s
sys 0m3.330s

[4-alien (4-alien)]

isn't this fixed already?

[Pascal Terjan (pterjan42)]

As far as I know, nothing has been done.

[4-alien (4-alien)]

i thought we only used the xml and synthesis and not the hdlists themselves?

[4-alien (4-alien)]

should we move this feature to mga4 then?

[fraterlinux (frateraec)]

The biggest problem is not the technique. The greatest difficulty is to awaken the enterece in who can do it...

[Boklm (boklm)]

(In reply to AL13N from comment #25)
> i thought we only used the xml and synthesis and not the hdlists themselves?

Which is not the same as repomd.

(In reply to AL13N from comment #26)
> should we move this feature to mga4 then?

It is not clear whether we want this, and whether there is someone who wants to implement this.

[Thierry-vignaud (thierry-vignaud)]

There's no such thing as standard metadata for rpm.
IMHO "standard rpm metadata" is a lie as it's not for rpm at all for the higher level package manager (yum, urpmi, ...), each one having its format.
If I were to change format, I would consider switching to libsolv format instead (and to use libsolv as resolver), not yum format.

[4-alien (4-alien)]

ok, so better set to WONTFIX then until someone comes along and actually wants to do the work of changing format IF it's actually needed...

[Thierry-vignaud (thierry-vignaud)]

Closing. Again.

[4-alien (4-alien)]

rereading this, i think the stats speak for themselves. time to let the beast rest... forgood.

[fraterlinux (frateraec)]

Could you explain exactly what you mean?

[Matthew Dawkins (mattydaw)]

Thierry, would making genlisthd2 and urpmi a separate community project
from MGA or OpenMandriva be a good way to further the standardization?
Maybe even discuss using something like libsolv?

Regards,
Matthew Dawkins

On Wed, Apr 10, 2013 at 2:39 PM, fraterlinux <email address hidden> wrote:

> Could you explain exactly what you mean?
>
> --
> You received this bug notification because you are a member of rpm5.org,
> which is subscribed to RPM.
> https://bugs.launchpad.net/bugs/913192
>
> Title:
> urpmi attempting to use repo-md
>
> Status in rpm package manager:
> New
> Status in Mandriva Linux:
> Won't Fix
>
> Bug description:
> URPMI in Mageia is attempting to use repo-md as well
> as synthesis/hflists tracker.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/rpm/+bug/913192/+subscriptions
>

[4-alien (4-alien)]

it means that repomd is slower(x5) and bigger than what we have now...

it also means that format-chaning has effects on alot of parts of the distro...

it also means that the primary developer would rather want libsolv rather than repomd, because of his preferences.

iow: not gonna happen.

[fraterlinux (frateraec)]

The main problem is really urpmi and this rpm standardization would be useful to use other package managers. URPMI trying to be friendly often only creates problems. Eg: only show the latest version of the package in repository. Does not downgrade transparently. If you have installed community repositories URPMI hides official packages, etc ... Fortunately now we can use SMART on Mageia 3. I follow the distro since Mandrake 7 and urpmi has always been limited in some points.

[4-alien (4-alien)]

perhaps it can be improved instead? (in another bug report)

iinm --oldpackage was ported to urpmi, i think i saw a change about this...

also the latest version, that's just rpmdrake filtering, one could make a patch to show all if one wanted it bad enough...

listen, we can't use repomd, because the whole buildsystem would slow down 5x more..., we could maybe only do repomd for releases, but then it would have bad testing and such...

[fraterlinux (frateraec)]

I understand. So I said the question of urpmi. It is more easier to improve urpmi than to modify the standard. I had already reported the support -- downgrade for urpmi and it has already been partially implemented. Now then is to report more bugs enhancement of urpmi.

https://bugs.mageia.org/show_bug.cgi?id=6655