RFE: [Performance] yum update slower as more packages go into each minor release, esp. bad on IA64 hw

[Jeff Johnson]

tracker

[Alexander (alexander-redhat-bugs)]

Description of problem:
This is a performance issue and most probably not a real bug.
Doing updates from 5.1->5.2 takes very long time compared to previous releases.
This is only on ia64

Version-Release number of selected component (if applicable):
yum-3.2.8-7.el5

How reproducible:
100% on selected hardware

Steps to Reproduce:
1. Install RHEL5.1 Server, all repos enabled, @everything
2. configure yum repos for RHEL 5.2
3. yum update

Actual results:
update takes about 7hrs.

Expected results:
update is as fast as possible

Additional info:
5.0 -> 5.1 updates take about 4hrs on the same hardware but for 5.1->5.2 case we
have more packages in the transaction.
Same results with `yum update` vs. `yum update yum` on that hardware.

James Antill provided me with a debugging yum package and says:
james: as from the numbers I saw it's not a real bug ... we are mostly linear in
time, as the package set is expanding

[Don (don-redhat-bugs)]

just to verify: no workarounds for this?

[Alexander (alexander-redhat-bugs)]

Don,
a proposed trick is

1) yum update yum
2) rpm --rebuilddb
3) yum update

but I haven't seen much difference. I'm performing tests once again right now.
Will post the results and logs later today.

[James (james-redhat-bugs)]

 So the first point is that this only happens when you have _large_ amounts of
packages, so if you don't have @everything ... things are much better.

 If the timings suggest that it is doing more work per. pkg as the pkg count
gets higher, then you could try (I would run a yum update -y too, after this
finishes):

http://people.redhat.com/jantill/yum/commands/yum-iter-update.py

...or something like it (it basically tries to install the updates one at a
time), and a supported core feature to do something like that might be making
it's way upstream for later release.

 But my quick numbers suggested that wasn't the case.

[Don (don-redhat-bugs)]

thanks James, Alex. adding to "Known Issues" (although i may move this to
"Installation-Related Notes" soon):

<quote>
(ia64) When upgrading a large number of packages with yum update, the update
process may take a very long time to finish.
</quote>

please advise if there's any definitive way to ensure that the problem is
alleviated significantly, at the very least; i will document upon your verification.

[Alexander (alexander-redhat-bugs)]

Don,
as of now I'm not aware of any way to alleviate this problem signifficantly. See
James' quote in comment #0.

I propose the following minor edit because the speed depends on the actual
hardware used, some machines are faster than others.

<quote>
(ia64) When upgrading a large number of packages with yum update, the update
process may take a very long time to finish on some hardware.
</quote>

[Alexander (alexander-redhat-bugs)]

Bug #71184 has some nice ideas on how to minimize install time. Some of them may
apply here as well.

[Denise (denise-redhat-bugs)]

Here is the preferred release note text. We're not planning a yum code change
for 5.2, so I'd also like to move this to 5.3.

(ia64) When upgrading, due to the larger number of packages in the
update, the update process will take longer to finish than previous
versions.

[Don (don-redhat-bugs)]

thanks Denise, edited release note as requested.

[Don (don-redhat-bugs)]

Hi,
the RHEL5.2 release notes will be dropped to translation on April 15, 2008, at
which point no further additions or revisions will be entertained.

a mockup of the RHEL5.2 release notes can be viewed at the following link:
http://intranet.corp.redhat.com/ic/intranet/RHEL5u2relnotesmockup.html

please use the aforementioned link to verify if your bugzilla is already in the
release notes (if it needs to be). each item in the release notes contains a
link to its original bug; as such, you can search through the release notes by
bug number.

Cheers,
Don

[Denise (denise-redhat-bugs)]

Changing summary to better reflect the topic

[James (james-redhat-bugs)]

 AIUI it's just package releated ... so changing summary to that.

 Also is it possible to get some feedback on the Fed-9/upstream releases we are
doing? As if we still need to do a significant amount of work I'd like to start
that very soon ... I'm hoping that we'll be close enough in 3.2.15, but
confirmation of that would be good.

[RHEL (rhel-redhat-bugs)]

This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release. Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products. This request is not yet committed for inclusion in an Update
release.

[James (james-redhat-bugs)]

 The needinfo was for someone testing a pre. Fed-9 version of YUM.
 I can say that all the tests I've done the yum that'll be in 5.3 will be faster
at updates and use less memory. Obviously it still goes up relative to the
number of packages installed/available ... and noone in Fedora uses ia64, AFAIK :)

 So, anyway, I'm going to devel ack this ... as I think we're pretty good now.
As always, if you could do any testing while we can still fix things that'd be
great (3.2.17++ should be going into the 5.3 branch soon, so it should be easily
available this week or so).

[Ryan (ryan-redhat-bugs)]

Tracking this bug for the Red Hat Enterprise Linux 5.3 Release Notes.

[Ryan (ryan-redhat-bugs)]

Release note added. If any revisions are required, please set the
"requires_release_notes" flag to "?" and edit the "Release Notes" field accordingly.
All revisions will be proofread by the Engineering Content Services team.

[James (james-redhat-bugs)]

 To anyone testing this ... can you try the packages in the repo at:

http://people.redhat.com/jantill/53/10/

...there isn't an ia64 arch, but every package is noarch ... so you should be able to install it over 3.2.8 just fine (ie. from GA do update yum, then install packages from the above repo.)
 Some rough numbers on where we are relative to the older yum's would be great.

[Alexander (alexander-redhat-bugs)]

(In reply to comment #21)
> Some rough numbers on where we are relative to the older yum's would be great.

I've kicked some tests and hopefully we'll have results by tomorrow. The tests are:
5.1 -> 5.2, 5.2 -> 5.3 nightly, 5.1 -> 5.2 (with yum updated) and 5.2 -> 5.3 nightly (with yum updated)

[Alexander (alexander-redhat-bugs)]

(In reply to comment #21)
> To anyone testing this ... can you try the packages in the repo at:
>
> http://people.redhat.com/jantill/53/10/
>

James,
can you generate a repodata/ directory under noarch please?

Some results from previous releases on the same hardware:
RHEL 5.1 -> 5.2, @everything, 06:25:44 hrs
RHEL 5.2 -> 5.3 nightly - too few updated packages so far, will use 5.1 -> 5.2 for testing

[James (james-redhat-bugs)]

 Ok, I generated the repodata under noarch and made ppc and ia64 basearch directories.

[James (james-redhat-bugs)]

 Just rpm -e yum-skip-broken will fix it, I'll put a conflict into the specfile.

[Alexander (alexander-redhat-bugs)]

(In reply to comment #23)
> RHEL 5.1 -> 5.2, @everything, 06:25:44 hrs

RHEL 5.1 -> 5.2, @everything/yum update yum/, yum update -> 06:16:07 hrs

[James (james-redhat-bugs)]

 Ok, so almost 10 minutes better ... which isn't much, however I'm now wondering how much time yum is taking and how much rpm.

 I assume you are basically doing "time yum update -y" ... is it possible for you to do "time (echo n | yum update)". Which will test the time for everything upto the download and rpm-transaction.

 Also, more generally, is it possible to get access to the machine you are running this on before the "yum update" is done?
 I can get ia64 boxes from RHTS, but they don't have everything ... and a 5.1 => 5.2 update is on the order of 8-12 minutes (8 for 3.2.17, 12 for 3.0.1).

[Alexander (alexander-redhat-bugs)]

(In reply to comment #28)
> Ok, so almost 10 minutes better ... which isn't much, however I'm now
> wondering how much time yum is taking and how much rpm.
>
> I assume you are basically doing "time yum update -y" ... is it possible for
> you to do "time (echo n | yum update)". Which will test the time for everything
> upto the download and rpm-transaction.
>

I need to clarify my results. The total time of about 6hrs reported is from the initial install to the end of the upgrade. Roughly half of that time is the install only. I'll perform more precise timing as described above.

> Also, more generally, is it possible to get access to the machine you are
> running this on before the "yum update" is done?
> I can get ia64 boxes from RHTS, but they don't have everything ... and a 5.1
> => 5.2 update is on the order of 8-12 minutes (8 for 3.2.17, 12 for 3.0.1).

I can provide you with access to the machine in question.

[Alexander (alexander-redhat-bugs)]

New timings for only 'yum update', without actually installing anything:

5.1 -> 5.2: ()
real 5m45.746s
user 1m12.660s
sys 0m7.612s

NOTE:
yum tracebacks with (I guess it can't wait so long to read the 'n' character)

Install 33 Package(s)
Update 742 Package(s)
Remove 4 Package(s)

Total download size: 1.4 G
Is this ok [y/N]: Traceback (most recent call last):
  File "/usr/bin/yum", line 29, in ?
    yummain.main(sys.argv[1:])
  File "/usr/share/yum-cli/yummain.py", line 180, in main
    base.doTransaction()
  File "/usr/share/yum-cli/cli.py", line 390, in doTransaction
    if not self.userconfirm():
  File "/usr/share/yum-cli/output.py", line 123, in userconfirm
    choice = raw_input('Is this ok [y/N]: ')
EOFError: EOF when reading a line

5.1 -> 5.2 (yum updated):
real 2m10.889s
user 0m22.272s
sys 0m1.772s

We're 2/3 times faster with the new yum version but what is really slow is the package download/install/remove process. It's sequential if I'm not wrong.

Bug 71184 has some nice ideas how to speed up things. It's been open for years and the reporter has been active providing information but the bug was CLOSED/WONTFIX.

One thing to start with is parallel downloads of packages to utilize bandwidth. One thread will download the largest packages while another will start downloading the smallest ones (making more connections) and they meet in the middle.

James,
please let me know how you feel about the ideas in bug 71184 and I can start filing them as separate items which can go either to RHEL or Fedora.

Thanks,
Alexander

[James (james-redhat-bugs)]

> We're 2/3 times faster with the new yum version but what is really slow is the
> package download/install/remove process. It's sequential if I'm not wrong.

 Ok, yeh. I hadn't realized the yum working parts was so fast already ... was the first test with 3.2.8 or 3.0.l?

 Post depsolving there's work we can do in yum, but I'd guess that most of the work actually needs to be done in rpm. Yum stages are roughly:

1. init - read configs. etc.

2. Run command
  . Possibly download any metadata

3. If command is doing install/remove/update/etc. do a depsolving run
  . Possibly download any metadata

4. Do package downloads
  . We basically do "for pkg in pkgs: download(pkg)"
  . This is also not non-piplined, so each pkg add the latency of the request response.

5. Call "run rpm transaction", this is all inside rpm.

...AIUI from your last post you are getting to #4 in ~2m, and then taking over 3hrs for #4 and #5.
 From yum itself we have no control over #5, although Panu ffesti etc. are working on speeding it up ... but I'd imagine that's going to be a RHEL6 thing.
 We have #4 on the TODO list, to make it do parallel/piplined downloading etc. ... but I don't think we can guarantee that for RHEL5 (part of the process is changing to use curl/pycurl as the backend instead of urllib/httplib).

 We also have on the TODO list to split a large transaction into N smaller ones, and then run each (serially at first, but parallelizing some of it might be possible). This _might_ help you, and we can likely get this into 5.5.

 Looking at BZ 71184 the only thing that looks relevant to yum is the parallel/pipelined download.

[James (james-redhat-bugs)]

 As a quick test can you do:

rm -rf /var/cache/yum (yum clean has problems with RHN)
time yum update --downloadonly

...that shouldn't have changed much between 3.2.8 and 3.2.17+, so feel free to only test the later one. This'll give us the data on how much is the downloading and how much is rpm.
 Can you also paste the total line, at the end of the download (this is probably fine on it's own if you have it from a previous run).

[Alexander (alexander-redhat-bugs)]

(In reply to comment #31)
> Ok, yeh. I hadn't realized the yum working parts was so fast already ... was
> the first test with 3.2.8 or 3.0.l?
>

The slower one was 3.0.1 the faster one 3.2.17

> Post depsolving there's work we can do in yum, but I'd guess that most of the
> work actually needs to be done in rpm. Yum stages are roughly:
>
> 1. init - read configs. etc.
>
> 2. Run command
> . Possibly download any metadata
>
> 3. If command is doing install/remove/update/etc. do a depsolving run
> . Possibly download any metadata
>
> 4. Do package downloads
> . We basically do "for pkg in pkgs: download(pkg)"
> . This is also not non-piplined, so each pkg add the latency of the request
> response.
>
> 5. Call "run rpm transaction", this is all inside rpm.
>
>
> ...AIUI from your last post you are getting to #4 in ~2m, and then taking over
> 3hrs for #4 and #5.

Indeed #4 takes not so long as I use a local http server but pipelined downloads will speed things up.

[Alexander (alexander-redhat-bugs)]

(In reply to comment #32)
> As a quick test can you do:
>
> rm -rf /var/cache/yum (yum clean has problems with RHN)
> time yum update --downloadonly
>
> ...that shouldn't have changed much between 3.2.8 and 3.2.17+, so feel free to
> only test the later one. This'll give us the data on how much is the
> downloading and how much is rpm.
> Can you also paste the total line, at the end of the download (this is
> probably fine on it's own if you have it from a previous run).

I'd propose something else. If you can provide me with a patched yum package that will do some more verbose logging and print timestamps at different stages I'll be glad to run the test again.

I have no data how much the download takes from a previous run and this data is not likely to be very correct. The environment used is a shared one and the server/network may be idle or overloaded depending on other machines in the environment. I can run multiple tests (e.g. 10) and get an average of the results if you want.

[Denise (denise-redhat-bugs)]

We could really close this since yum improved significantly, but this has now become the package installation performance tracker. I'm moving to 5.4/rpm so we dont lose this, but the release note can disappear.
Denise

[Denise (denise-redhat-bugs)]

Release note updated. If any revisions are required, please set the
"requires_release_notes" flag to "?" and edit the "Release Notes" field accordingly.
All revisions will be proofread by the Engineering Content Services team.

Diffed Contents:
@@ -1,2 +1,4 @@
+This had a 5.2 release note that needs to be removed for 5.3 - the text was:
+
 (ia64)
 When upgrading a large number of packages with yum update, the update process will take longer to finish than previous versions.

[Ryan (ryan-redhat-bugs)]

Thanks Denise!

I have removed this note from the 5.3 release notes.

[errata-xmlrpc (errata-xmlrpc-redhat-bugs)]

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2009-1371.html