Dynamic mirrorlists support for smart (e.g. fedora)

[Rehan Khan]

Attached is a new mirror channel type which supports getting a list of mirrors (only) from a server.

This should be very useful for distributions that distribute their mirrors as a list of mirrors (e.g. fedora). It seems to work well with dynamically generated mirrors (e.g. from Fedora/Red Hat's geoip based Mirrormanager) and should work with other dynamic mirrorlist generators as long as the url does not contain any illegal filesystem characters (as in the url should be able to be a filename).

The channel type is called 'mirrorlist'. I have attached two files:

usr.lib.python_ver.site-packages.smart.tar.gz : this is the new channel definition. The two files in the archive (mirrorlist.py and mirrorlist_info.py) should be copied to /usr/lib/python2.5/site-packages/smart/channels (on fedora 8, modify for your particular distro).

etc.smart.channels.tar.gz : two example channel configuration files for fedora 8, one for core and one for updates. The two files should go in /etc/smart/channels. Note: run update once to get the latest mirrors.

One very cool thing is that the gui automagically allows you to create a channel of this type. Use : edit -> channels -> new -> 'provide channel information' -> ''mirrorlist Information' and supply a base url and the mirrorlist url.

Note: As it is a mirror channel the mirrors don't show up in the gui or smart mirrors --show but they do work very well (at least on my test machine). this was tested on bzr trunk but it should work on 0.52.

Please try out the channel and provide feedback!

[Peter (peter-redhat-bugs)]

Description of problem:
The config package fails to configure any mirrors

Version-Release number of selected component (if applicable):
All

How reproducible: Every time

Steps to Reproduce:
1.Install package
2.Look for mirror definition

Actual results:
No mirrors listed

Expected results:
All Fedora mirrors should be listed (and used)

Additional info:
It would be "nice to have" separate configurations for users, developers, and
testers so persons not interested in, e.g. testing, would not need to disable
unwanted repository references.

[Ville (ville-redhat-bugs)]

Unlike channel configuration, I'm not sure if it's possible to ship smart mirror
lists as files in a package.

Another thing is that it doesn't really make sense to include them in a package
because of their dynamic nature, mirrors get added and removed and we don't want
to be updating the configuration package because of this.

I suppose it'd be best if mirrormanager was taught to output smart mirror lists
in addition to yum ones, then people could run "smart mirror --sync
http://some.where/mirrormanager-for-smart" and we could include that info in
this package somehow.

http://fedoraproject.org/wiki/Infrastructure/MirrorManager
https://hosted.fedoraproject.org/projects/mirrormanager/

[Rehan (rehan-redhat-bugs)]

This issue has been around since at least fedora core 6 and it bugs me a lot.

I have tried a number of things to get smart working on mirrors.

1) smart channel --add type=mirrors
2) smart channel --add type=up2date-mirrors
3) bash script which parses the dynamic/localised mirror list returned by the
normal yum mirror list url.

The bash script works well but it needs to be run occasionally to update the
list. Smart will favour mirrors which are available more often than not so it
doesn't need to be run that often. It could be a good prototype for a
fedora-mirror.py which can be run from /etc/smart/distro.d.

On the other hand the mirrormanager output only needs to do one of two things:

1) put the baseurl as the first line and then a tab before all the other urls
(the actual mirrors). this has the disadvantage that yum will use the redhat
repos as a near mirror (which is bad of course)

2) do the above but only for a url passed with an option of &smart=yes
e.g. http://mirrors.fedoraproject.org/mirrorlist?repo=fedora-8&arch=i386&smart=yes

this might be the simplest option as it would only need a change in the script
which generates the mirrorlist sent to the client.
with this the mirrors could be added using a command something like:
smart channel --add "Fedora 8 - i386" type=mirrors alias=fedora enabled=yes
url=http://mirrors.fedoraproject.org/mirrorlist?repo=fedora-8&arch=i386&smart=yes

3) Not sure if the up2date mirror lists are still produced for fedora 8
(probably not, the wiki has a request to remove this information). However smart
 can still use this information.
smart channel --add "Fedora 8 - i386" type=up2date-mirrors alias=fedora
enabled=yes url=??????
(not sure how the url would look as I can't find much info for up2date. This
used to be done so if the code is still there it might be easy to put it back.

My preference would be to do 2 but 3 might be ok as well. I hope something gets
done because I use smart all the time and it *is* only using the fedora servers.
Somehow this makes me feel bad :(

BTW I think Selinux is denying smart access to some operations which I think is
quietly failing those operations. To see this do a smart upgrade or smart
install and then a tail messages and you will see entries relating to selinux
denying access to the temp files in /tmp/TMPXXXXX which smart generates. As you
guys might be more familiar with smart than I am, do you think this is worthy of
opening a bug?

I would love to see smart be able to automatically use mirrors. It is a
beautiful thing to see smart doing multiple downloads from different mirrors and
taking half the time of yum to do the downloads.

[Rehan (rehan-redhat-bugs)]

I should have said 'one of three things in my first post :)

Some additional info on using the output returned by the current mirrorlist
script. I believe that the comment line that is output with information about
the mirrors being supplied actually makes a mirror import fail in smart(using
the channel --add type=up2date-mirrors option). It's either that or because
something else in the format is not up2date mirror format. Probably the latter
but documentation on what an up2date mirrorlist should look like is pretty thin
(non-existant?)

[Ville (ville-redhat-bugs)]

Mirrormanager improvement ideas and requests would be much more likely to be
addressed when communicated through their ticket system instead of here in
Bugzilla, see https://hosted.fedoraproject.org/projects/mirrormanager/

[Rehan (rehan-redhat-bugs)]

Thats true. However I wanted to discuss it with people who know and use smart
on fedora. What is a good proposition to take to the mirrormanager guys that
would meet the needs of a Smart Package Manager user? As they say, there's more
than one way to skin a cat.

[Peter (peter-redhat-bugs)]

Re comment 2 by Rehan Kahn: If the bash script was published, it might alleviate
the current situation were, if the definitions provided by the RPM are used, all
smart usage on Fedora defaults to accessing the Red Hat repositories. This is
probably not a "good thing" from the perspective of efficient network usage.

[Rehan (rehan-redhat-bugs)]

Yes it's a very bad thing. That was why I wrote the script. I planned to
publish it somehow. However it's a sprawling mess and mutated into a kind of
repository manager for yum and smart.

If I remember correctly the only thing it actually did with reference to smart
and mirrors was wget the normal mirror list (e.g. the mirrorlist url from the
yum conf file) into a file, export only the lines beginning with ftp and http
(the comment lines used to confuse smart) and then parse the generated file and
import the mirrors. This worked quite well.

Let me see if I can dig it up.
R

[Axel (axel-redhat-bugs)]

Well, I also have a two liner for producing mirror lists for smart out of yum's
mirrors URL. But that isn't really scaling well: It would need to run before
every smart update or smart something --update call, e.g. it would have to
become a plugin to make sense. Furthermore the reimporting of mirrors kills the
scores, so it's competing to smart's own mirroring system.

So ATM we don't have a clear path of what is right for the end user. I think
this issue is better presented upstream.

Just for reference the two-liner I use is:

releasever=8; basearch=x86_64
grep -B1 mirrorlist /etc/yum.repos.d/fedora*repo | sed -e's,.*#,,'
-e's@.*mirrorlist=\(.*\)$@wget -nv -O - '\''\1'\'' | grep -v \\# | sed
-e"s,^,smart mirror --add $baseurl ,"@' -e"s,\$releasever,$releasever,g"
-e"s,\$basearch,$basearch,g" | grep -v -- ^-- | sh | sh

[Ville (ville-redhat-bugs)]

I also have a script for converting yum configs (including mirror lists)
available at http://scop.fedorapeople.org/scripts/yum2smart.py . Not sure if
it's any more useful in this particular case than Axel's "two-liner", but I've
found it useful in converting repo configs in general. I've intended to submit
is as a contrib script to smart upstream, but haven't got around to doing it yet.

[Axel (axel-redhat-bugs)]

Ville, I like your script as with some plugin magic it could be made the plugin
mentioned above.

Maybe Gustavo and friends will poke on it to massage it into a plugin if you
post it on list or labbix. Thanks!

[Rehan (rehan-redhat-bugs)]

I think one can safely assume that mirrors don't get a lot of churn. On that
basis perhaps it is better that a cron job runs once a week or month updating
the mirrors in smart. Smart can then sort out the 'bad' mirrors from the good
as it does this anyway. The fallback is always the baseurl during periods when
every mirror goes down :) This way there is no additional overhead for updating
the mirrors each time smart runs.

[Rehan (rehan-redhat-bugs)]

A plugin has been posted on the Smart bugtracker which might solve this issue.

http://tracker.labix.org/issue350

[Rehan Khan (rasker)]

The format for the channel definition is:

[<alias>]
name=<friendly name>
base=<the root url>
url=<the url pointing to the mirrorlist>
type=mirrorlist

[Rehan (rehan-redhat-bugs)]

A new mirror channel type (mirrorlist) might be able to solve this problem. As
it is a native Smart channel it integrates well with how Smart likes to do
things.

See https://bugs.launchpad.net/smart/+bug/247351

I have tested this on Fedora 8 and it works well. If more feedback is received
this might get into upstream pretty soon as it is sorely needed.

[Rehan Khan (rasker)]

Improving mirror support (making it easy to add mirrors) should be a critical activity as it prevents distro take up of Smart (it is surely a blocker if you are the one paying for the bandwidth!).

[netmask (netmask)]

Rehan, IMNSHO, this is not a proper implementation for mirrorlist. This should be implemented as an option to rpm-md channel, as it is already being worked on branch ~netmask/smart/mirrorlist

[netmask (netmask)]

As discussed on IRC.

< netmask> rasker: if I'm not mistaken this has been implemented ages ago when apt channels did not have a "mirror"
                 clausule, and the mirrors were provided on txt files on the main ftp server. considering rpm-md has a tag for
                 that, and yast/zypper will implement a tag either, also considering most distros provide an automatic
                 redirector, a new channel type is unecessary, and the old mirror channel should be maintained just as
                 backwards compatibility.

The solution will be implemented as adding the 'mirrorlist' tag to rpm-md, and any other tags needed by any future implementations.

[Rehan Khan (rasker)]

Putting a fix in place for a single problem only does not really resolve the issue. A generic solution is much better than a point solution. A mirrorlist channel is generic, is easier to maintain and improve in the codebase as a separate entity and follows the general design of Smart. As I said on IRC IF you finish the code for rpm-md then we can re-visit this and indeed IF you finish the code then it only solves one problem so if anything both should be considered for inclusion in Smart.

I think it would be better if you were to quote MY agreement rather than yourself in future.

[netmask (netmask)]

Please, reconsider ~netmask/smart/mirrorlist branch. The code is complete, tested, and requested for merging.

[Rehan Khan (rasker)]

Ok that rocks for rpm-md. I've tested it extensively (with fedora 8) and it works really well. It populates Smart's native mirror definition and so there is also the ability to view the mirrors in edit -> mirrors. That's pretty cool. Mind if I borrow that code, netmask, for the other channel types? ;)

In the release documentation we have to make it clear that distros need to update their channel definitions in /etc/smart otherwise the user will experience errors.

From a distro's perspective the only remaining issue is that on the initial install there are no mirrors and their servers will be hit hard (particularly on release days when the servers are already being hit hard). Some way to 'pre-install' mirrors would be advantageous. Perhaps a 'save-default-config' which saves the config file with a .default extension would help. This default config could then be included in the rpm/deb/whatever and smart would use that if it can't find a config on startup.

So with this and the mirrorlist channel type I think Smart will be able to cope with most things thrown at it in the way of mirror implementations not just what is out there now but whatever the future may bring. Fantastic progress, there are fewer and fewer technical reasons for distros to discount Smart.

[Axel (axel-redhat-bugs)]

I saw that there are conflicting suggestions on which one of the plugins/code patches to use for mirrorlist support. First merge reviews have been made, and it looks like an alternative approach with different end-user visible syntax will be used, but even this didn't make it into smart 1.0 (which has been packaged up and awaits the infrastructure systems to get back).

We shouldn't make any hasty assumptions on what upstream will choose to merge into the mainline code. It's good to see that this is being worked on and we'll get tis into Fedora as soon as there is some official blessing.

[Rehan (rehan-redhat-bugs)]

It's a shame neither implementation made it into the latest release. Just to clarify the two implementations. The new mirrorlist channel definition is currently the 'approved' method of storing dynamic mirrors (dynamic mirrors should not be editable by end users). The second method adds an import of the mirrors to the user editable internal smart channel format. Both work quite well with the only difference being that the mirrors are editable within smart in the second method.

As no agreement has been reached in approving the second method yet you should be ok using the mirrorlist channel (first method). It just adds two files to Smart and makes no other changes to the code. Smart automatically integrates the channel. There is an example of Fedora 8 mirrorlist .channel files in the upstream bugtracker.

[Axel (axel-redhat-bugs)]

(In reply to comment #15)
> The new mirrorlist channel definition is currently the 'approved' method of storing dynamic mirrors

Can you point me to the approval? If it is approved, why isn't it in the sources?

> As no agreement has been reached in approving the second method yet you should
> be ok using the mirrorlist channel (first method).

Actually I just found out that you are the author of the first method, and upstream seems to favour the other one (there were some comments that seem to have been resolved and are waiting for a new merge review by upstream).

I'm sorry, Fedora should stay close to upstream and if upstream looks like going one direction Fedora shouldn't be going the other way. Please note that I'm not commenting on whether method A or B is better, and your work on improving smart for Fedora is acknowledged!

[Rehan (rehan-redhat-bugs)]

(In reply to comment #16)
> (In reply to comment #15)
> > The new mirrorlist channel definition is currently the 'approved' method of storing dynamic mirrors
> Can you point me to the approval? If it is approved, why isn't it in the
> sources?

Good question! Don't know is the answer. Actually, although not well documented Smart is plugin based. The idea being that for certain additional functionality the end user can just add a plugin for that functionality. this applies to backends, channels, mirrors etc. There are some guidelines which are only discovered by asking on irc atm :) So adding a mirrorlist channel is implicitally approved as it is just another plugin. This is the way method one works. Method two on the other hand actually changes the way Smart works with rpm-md (rpm metadata) channels (from a higher level design point). Method 2 is a better way from the end user perspective as you can see and edit the mirrors. I use method 2 preferentially over method 1.

> > As no agreement has been reached in approving the second method yet you should
> > be ok using the mirrorlist channel (first method).
> Actually I just found out that you are the author of the first method, and
> upstream seems to favour the other one (there were some comments that seem to
> have been resolved and are waiting for a new merge review by upstream).
> I'm sorry, Fedora should stay close to upstream and if upstream looks like
> going one direction Fedora shouldn't be going the other way. Please note that
> I'm not commenting on whether method A or B is better, and your work on
> improving smart for Fedora is acknowledged!

Thanks, I'm not pushing either method though, you know from a self agrandisment perspective :) I am suggesting that Fedora use method one as a plugin suitable for Fedora (mirrorlist channel), right now, as it provides sorely needed functionality and does not alter the way Smart inherently works so it won't break future compatibility (so is upstream approved) . Should method two become incorporated in the base code then move to that in future as it is a nicer way of doing the same thing (mirror information is kept with repo information, mirrors are user editable, etc). Using either method in fedora will then increase it's likelyhood of being included in future Smart releases.

To summarise (the above is a bit abstract), Using a mirrorlist channel is implicitly approved (by upstream) as it is a plugin (method 1) and adding mirror functionality to rpm-md channels (method 2) needs explicit approval as it modifies the base code and might break backwards compatibility.

If you want I can provide the bits neccessary for both if you want.

[Gustavo Niemeyer (niemeyer)]

The implementation looks *really* nice, specially the idea of
moving getMirrors() to the base channel.

I'm adding some comments below, but it would be fantastic to
get this in place on time for 1.2:

[1]

The existence of MirrorsChannel doesn't seem to be necessary
anymore. Can we remove it?

[2]

This change will blow up any existing caches right after Smart
is upgraded. Talk to me online for hints about how to prevent
it.

[3]

The number of fetch steps will be wrong when there's no mirror file.

[4]

There are no tests for the changes in control.py

[5]

It would be great if the test for the channel was in a Python file
(e.g. metadata.py), specifically for testing the metadata channel,
and if this branch didn't change metadata.txt. The doctest file
was mostly a quick workaroud to get some tests in place, but it's
a poor way to verify incremental changes like this. For instance:
by introducing the test for mirrorlist, you're removing the test
that checks that it works fine *without* this option.

To get some idea for how to build this test file, check
tests/debloader.py, or just talk to me online. I'll be happy
to discuss this with you, or to help you.

[Bug (bug-redhat-bugs)]

This message is a reminder that Fedora 8 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 8. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora
'version' of '8'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version prior to Fedora 8's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that
we may not be able to fix it before Fedora 8 is end of life. If you
would still like to see this bug fixed and are able to reproduce it
against a later version of Fedora please change the 'version' of this
bug to the applicable version. If you are unable to change the version,
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

The process we are following is described here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

[netmask (netmask)]

[1]
I see MirrorsChannel class is in use by "mirrors" (which sounds like "apt" to me) and "up2date" (deprecated by Red Hat in favour of Yum) channel types. Maybe those channels will need a re-work, or to be deprecated, if you want to remove this class. This is not the scope of this bug report. ;)

[2]
I don't think it's too bad. Since it's a one-off operation, and the metadata will get fetched from the mirror instead of the main server, I don't think it's a big loss.

[3]
Ok, that can be fixed. What do you prefer? An if on the getFetchSteps, or on the fetch process itself?

[4]
Right, I need to work on that.

[5]
Hm, I need to think about this carefully.

[Gustavo Niemeyer (niemeyer)]

[1]

Yes, this is the scope of this bug report, because if the class was rendered unnecessary, your changes were the ones doing so, and so they must also fix that bug.

Is the class actually needed on the places you mention, or is everything needed already present in the base class?

[2]

I'm surprised to read that comment. Every single instance of Smart out there will blow up after it's upgraded with your changes. This won't get integrated like this.

[3]

As long as the code looks sensible and is correct, it's ok. I'll trust your judgment on that.

[4]

Ok.

[Bug (bug-redhat-bugs)]

Fedora 8 changed to end-of-life (EOL) status on 2009-01-07. Fedora 8 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.

[netmask (netmask)]

Gustavo, please tell me how you think I can prevent [2] from happening.

[Glenn D. (doiggl-hotmail)]

For OpenSuse could the Build Service or Redirector be utilized.

See:
http://en.opensuse.org/Build_Service/Redirector#openSUSE_download_redirector

[Anders F Björklund (afb)]

The Redirector seems to work with 302, which should be handled already ?

Metalink support was added in #314070, as an add-on to #247351 (mirrorlist)

[Anders F Björklund (afb)]

No new channel types, no changes to control, no explosions in cache. :-)
Just merged. http://bazaar.launchpad.net/~smartpm/smart/trunk/revision/881