RFE: [Performance] yum update slower as more packages go into each minor release, esp. bad on IA64 hw
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
RPM |
New
|
Undecided
|
Unassigned | ||
Fedora |
Fix Released
|
Low
|
In Red Hat Bugzilla #435475, Alexander (alexander-redhat-bugs) wrote : | #1 |
In Red Hat Bugzilla #435475, Don (don-redhat-bugs) wrote : | #2 |
just to verify: no workarounds for this?
In Red Hat Bugzilla #435475, Alexander (alexander-redhat-bugs) wrote : | #3 |
Don,
a proposed trick is
1) yum update yum
2) rpm --rebuilddb
3) yum update
but I haven't seen much difference. I'm performing tests once again right now.
Will post the results and logs later today.
In Red Hat Bugzilla #435475, James (james-redhat-bugs) wrote : | #4 |
So the first point is that this only happens when you have _large_ amounts of
packages, so if you don't have @everything ... things are much better.
If the timings suggest that it is doing more work per. pkg as the pkg count
gets higher, then you could try (I would run a yum update -y too, after this
finishes):
http://
...or something like it (it basically tries to install the updates one at a
time), and a supported core feature to do something like that might be making
it's way upstream for later release.
But my quick numbers suggested that wasn't the case.
In Red Hat Bugzilla #435475, Don (don-redhat-bugs) wrote : | #5 |
thanks James, Alex. adding to "Known Issues" (although i may move this to
"Installation-
<quote>
(ia64) When upgrading a large number of packages with yum update, the update
process may take a very long time to finish.
</quote>
please advise if there's any definitive way to ensure that the problem is
alleviated significantly, at the very least; i will document upon your verification.
In Red Hat Bugzilla #435475, Alexander (alexander-redhat-bugs) wrote : | #6 |
Don,
as of now I'm not aware of any way to alleviate this problem signifficantly. See
James' quote in comment #0.
I propose the following minor edit because the speed depends on the actual
hardware used, some machines are faster than others.
<quote>
(ia64) When upgrading a large number of packages with yum update, the update
process may take a very long time to finish on some hardware.
</quote>
In Red Hat Bugzilla #435475, Alexander (alexander-redhat-bugs) wrote : | #7 |
Bug #71184 has some nice ideas on how to minimize install time. Some of them may
apply here as well.
In Red Hat Bugzilla #435475, Denise (denise-redhat-bugs) wrote : | #8 |
Here is the preferred release note text. We're not planning a yum code change
for 5.2, so I'd also like to move this to 5.3.
(ia64) When upgrading, due to the larger number of packages in the
update, the update process will take longer to finish than previous
versions.
In Red Hat Bugzilla #435475, Don (don-redhat-bugs) wrote : | #9 |
thanks Denise, edited release note as requested.
In Red Hat Bugzilla #435475, Don (don-redhat-bugs) wrote : | #10 |
Hi,
the RHEL5.2 release notes will be dropped to translation on April 15, 2008, at
which point no further additions or revisions will be entertained.
a mockup of the RHEL5.2 release notes can be viewed at the following link:
http://
please use the aforementioned link to verify if your bugzilla is already in the
release notes (if it needs to be). each item in the release notes contains a
link to its original bug; as such, you can search through the release notes by
bug number.
Cheers,
Don
In Red Hat Bugzilla #435475, Denise (denise-redhat-bugs) wrote : | #11 |
Changing summary to better reflect the topic
In Red Hat Bugzilla #435475, James (james-redhat-bugs) wrote : | #12 |
AIUI it's just package releated ... so changing summary to that.
Also is it possible to get some feedback on the Fed-9/upstream releases we are
doing? As if we still need to do a significant amount of work I'd like to start
that very soon ... I'm hoping that we'll be close enough in 3.2.15, but
confirmation of that would be good.
In Red Hat Bugzilla #435475, RHEL (rhel-redhat-bugs) wrote : | #13 |
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release. Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products. This request is not yet committed for inclusion in an Update
release.
In Red Hat Bugzilla #435475, James (james-redhat-bugs) wrote : | #14 |
The needinfo was for someone testing a pre. Fed-9 version of YUM.
I can say that all the tests I've done the yum that'll be in 5.3 will be faster
at updates and use less memory. Obviously it still goes up relative to the
number of packages installed/available ... and noone in Fedora uses ia64, AFAIK :)
So, anyway, I'm going to devel ack this ... as I think we're pretty good now.
As always, if you could do any testing while we can still fix things that'd be
great (3.2.17++ should be going into the 5.3 branch soon, so it should be easily
available this week or so).
In Red Hat Bugzilla #435475, Ryan (ryan-redhat-bugs) wrote : | #15 |
Tracking this bug for the Red Hat Enterprise Linux 5.3 Release Notes.
In Red Hat Bugzilla #435475, Ryan (ryan-redhat-bugs) wrote : | #16 |
Release note added. If any revisions are required, please set the
"requires_
All revisions will be proofread by the Engineering Content Services team.
In Red Hat Bugzilla #435475, James (james-redhat-bugs) wrote : | #17 |
To anyone testing this ... can you try the packages in the repo at:
http://
...there isn't an ia64 arch, but every package is noarch ... so you should be able to install it over 3.2.8 just fine (ie. from GA do update yum, then install packages from the above repo.)
Some rough numbers on where we are relative to the older yum's would be great.
In Red Hat Bugzilla #435475, Alexander (alexander-redhat-bugs) wrote : | #18 |
(In reply to comment #21)
> Some rough numbers on where we are relative to the older yum's would be great.
I've kicked some tests and hopefully we'll have results by tomorrow. The tests are:
5.1 -> 5.2, 5.2 -> 5.3 nightly, 5.1 -> 5.2 (with yum updated) and 5.2 -> 5.3 nightly (with yum updated)
In Red Hat Bugzilla #435475, Alexander (alexander-redhat-bugs) wrote : | #19 |
(In reply to comment #21)
> To anyone testing this ... can you try the packages in the repo at:
>
> http://
>
James,
can you generate a repodata/ directory under noarch please?
Some results from previous releases on the same hardware:
RHEL 5.1 -> 5.2, @everything, 06:25:44 hrs
RHEL 5.2 -> 5.3 nightly - too few updated packages so far, will use 5.1 -> 5.2 for testing
In Red Hat Bugzilla #435475, James (james-redhat-bugs) wrote : | #20 |
Ok, I generated the repodata under noarch and made ppc and ia64 basearch directories.
In Red Hat Bugzilla #435475, James (james-redhat-bugs) wrote : | #21 |
Just rpm -e yum-skip-broken will fix it, I'll put a conflict into the specfile.
In Red Hat Bugzilla #435475, Alexander (alexander-redhat-bugs) wrote : | #22 |
(In reply to comment #23)
> RHEL 5.1 -> 5.2, @everything, 06:25:44 hrs
RHEL 5.1 -> 5.2, @everything/yum update yum/, yum update -> 06:16:07 hrs
In Red Hat Bugzilla #435475, James (james-redhat-bugs) wrote : | #23 |
Ok, so almost 10 minutes better ... which isn't much, however I'm now wondering how much time yum is taking and how much rpm.
I assume you are basically doing "time yum update -y" ... is it possible for you to do "time (echo n | yum update)". Which will test the time for everything upto the download and rpm-transaction.
Also, more generally, is it possible to get access to the machine you are running this on before the "yum update" is done?
I can get ia64 boxes from RHTS, but they don't have everything ... and a 5.1 => 5.2 update is on the order of 8-12 minutes (8 for 3.2.17, 12 for 3.0.1).
In Red Hat Bugzilla #435475, Alexander (alexander-redhat-bugs) wrote : | #24 |
(In reply to comment #28)
> Ok, so almost 10 minutes better ... which isn't much, however I'm now
> wondering how much time yum is taking and how much rpm.
>
> I assume you are basically doing "time yum update -y" ... is it possible for
> you to do "time (echo n | yum update)". Which will test the time for everything
> upto the download and rpm-transaction.
>
I need to clarify my results. The total time of about 6hrs reported is from the initial install to the end of the upgrade. Roughly half of that time is the install only. I'll perform more precise timing as described above.
> Also, more generally, is it possible to get access to the machine you are
> running this on before the "yum update" is done?
> I can get ia64 boxes from RHTS, but they don't have everything ... and a 5.1
> => 5.2 update is on the order of 8-12 minutes (8 for 3.2.17, 12 for 3.0.1).
I can provide you with access to the machine in question.
In Red Hat Bugzilla #435475, Alexander (alexander-redhat-bugs) wrote : | #25 |
New timings for only 'yum update', without actually installing anything:
5.1 -> 5.2: ()
real 5m45.746s
user 1m12.660s
sys 0m7.612s
NOTE:
yum tracebacks with (I guess it can't wait so long to read the 'n' character)
Install 33 Package(s)
Update 742 Package(s)
Remove 4 Package(s)
Total download size: 1.4 G
Is this ok [y/N]: Traceback (most recent call last):
File "/usr/bin/yum", line 29, in ?
yummain.
File "/usr/share/
base.
File "/usr/share/
if not self.userconfirm():
File "/usr/share/
choice = raw_input('Is this ok [y/N]: ')
EOFError: EOF when reading a line
5.1 -> 5.2 (yum updated):
real 2m10.889s
user 0m22.272s
sys 0m1.772s
We're 2/3 times faster with the new yum version but what is really slow is the package download/
Bug 71184 has some nice ideas how to speed up things. It's been open for years and the reporter has been active providing information but the bug was CLOSED/WONTFIX.
One thing to start with is parallel downloads of packages to utilize bandwidth. One thread will download the largest packages while another will start downloading the smallest ones (making more connections) and they meet in the middle.
James,
please let me know how you feel about the ideas in bug 71184 and I can start filing them as separate items which can go either to RHEL or Fedora.
Thanks,
Alexander
In Red Hat Bugzilla #435475, James (james-redhat-bugs) wrote : | #26 |
> We're 2/3 times faster with the new yum version but what is really slow is the
> package download/
Ok, yeh. I hadn't realized the yum working parts was so fast already ... was the first test with 3.2.8 or 3.0.l?
Post depsolving there's work we can do in yum, but I'd guess that most of the work actually needs to be done in rpm. Yum stages are roughly:
1. init - read configs. etc.
2. Run command
. Possibly download any metadata
3. If command is doing install/
. Possibly download any metadata
4. Do package downloads
. We basically do "for pkg in pkgs: download(pkg)"
. This is also not non-piplined, so each pkg add the latency of the request response.
5. Call "run rpm transaction", this is all inside rpm.
...AIUI from your last post you are getting to #4 in ~2m, and then taking over 3hrs for #4 and #5.
From yum itself we have no control over #5, although Panu ffesti etc. are working on speeding it up ... but I'd imagine that's going to be a RHEL6 thing.
We have #4 on the TODO list, to make it do parallel/piplined downloading etc. ... but I don't think we can guarantee that for RHEL5 (part of the process is changing to use curl/pycurl as the backend instead of urllib/httplib).
We also have on the TODO list to split a large transaction into N smaller ones, and then run each (serially at first, but parallelizing some of it might be possible). This _might_ help you, and we can likely get this into 5.5.
Looking at BZ 71184 the only thing that looks relevant to yum is the parallel/pipelined download.
In Red Hat Bugzilla #435475, James (james-redhat-bugs) wrote : | #27 |
As a quick test can you do:
rm -rf /var/cache/yum (yum clean has problems with RHN)
time yum update --downloadonly
...that shouldn't have changed much between 3.2.8 and 3.2.17+, so feel free to only test the later one. This'll give us the data on how much is the downloading and how much is rpm.
Can you also paste the total line, at the end of the download (this is probably fine on it's own if you have it from a previous run).
In Red Hat Bugzilla #435475, Alexander (alexander-redhat-bugs) wrote : | #28 |
(In reply to comment #31)
> Ok, yeh. I hadn't realized the yum working parts was so fast already ... was
> the first test with 3.2.8 or 3.0.l?
>
The slower one was 3.0.1 the faster one 3.2.17
> Post depsolving there's work we can do in yum, but I'd guess that most of the
> work actually needs to be done in rpm. Yum stages are roughly:
>
> 1. init - read configs. etc.
>
> 2. Run command
> . Possibly download any metadata
>
> 3. If command is doing install/
> . Possibly download any metadata
>
> 4. Do package downloads
> . We basically do "for pkg in pkgs: download(pkg)"
> . This is also not non-piplined, so each pkg add the latency of the request
> response.
>
> 5. Call "run rpm transaction", this is all inside rpm.
>
>
> ...AIUI from your last post you are getting to #4 in ~2m, and then taking over
> 3hrs for #4 and #5.
Indeed #4 takes not so long as I use a local http server but pipelined downloads will speed things up.
In Red Hat Bugzilla #435475, Alexander (alexander-redhat-bugs) wrote : | #29 |
(In reply to comment #32)
> As a quick test can you do:
>
> rm -rf /var/cache/yum (yum clean has problems with RHN)
> time yum update --downloadonly
>
> ...that shouldn't have changed much between 3.2.8 and 3.2.17+, so feel free to
> only test the later one. This'll give us the data on how much is the
> downloading and how much is rpm.
> Can you also paste the total line, at the end of the download (this is
> probably fine on it's own if you have it from a previous run).
I'd propose something else. If you can provide me with a patched yum package that will do some more verbose logging and print timestamps at different stages I'll be glad to run the test again.
I have no data how much the download takes from a previous run and this data is not likely to be very correct. The environment used is a shared one and the server/network may be idle or overloaded depending on other machines in the environment. I can run multiple tests (e.g. 10) and get an average of the results if you want.
In Red Hat Bugzilla #435475, Denise (denise-redhat-bugs) wrote : | #30 |
We could really close this since yum improved significantly, but this has now become the package installation performance tracker. I'm moving to 5.4/rpm so we dont lose this, but the release note can disappear.
Denise
In Red Hat Bugzilla #435475, Denise (denise-redhat-bugs) wrote : | #31 |
Release note updated. If any revisions are required, please set the
"requires_
All revisions will be proofread by the Engineering Content Services team.
Diffed Contents:
@@ -1,2 +1,4 @@
+This had a 5.2 release note that needs to be removed for 5.3 - the text was:
+
(ia64)
When upgrading a large number of packages with yum update, the update process will take longer to finish than previous versions.
In Red Hat Bugzilla #435475, Ryan (ryan-redhat-bugs) wrote : | #32 |
Thanks Denise!
I have removed this note from the 5.3 release notes.
In Red Hat Bugzilla #435475, errata-xmlrpc (errata-xmlrpc-redhat-bugs) wrote : | #33 |
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.
tags: | added: ia64 rhel yum |
Changed in fedora: | |
importance: | Unknown → Low |
status: | Unknown → Fix Released |
Description of problem:
This is a performance issue and most probably not a real bug.
Doing updates from 5.1->5.2 takes very long time compared to previous releases.
This is only on ia64
Version-Release number of selected component (if applicable):
yum-3.2.8-7.el5
How reproducible:
100% on selected hardware
Steps to Reproduce:
1. Install RHEL5.1 Server, all repos enabled, @everything
2. configure yum repos for RHEL 5.2
3. yum update
Actual results:
update takes about 7hrs.
Expected results:
update is as fast as possible
Additional info:
5.0 -> 5.1 updates take about 4hrs on the same hardware but for 5.1->5.2 case we
have more packages in the transaction.
Same results with `yum update` vs. `yum update yum` on that hardware.
James Antill provided me with a debugging yum package and says:
james: as from the numbers I saw it's not a real bug ... we are mostly linear in
time, as the package set is expanding