Make rpm -V be useful again.

[Jeff Johnson]

tracker

[Wade (wade-redhat-bugs)]

Description of problem:

This may seem a little ranty sorry in advance.

Sysadmins rely on rpm -V for confidence in their system, not only to figure out
if they have been hacked, but also to understand what has changed for backup and
change control understanding.

In the RHEL 5 lifecycle, multi-arch, sloppy programming and prelinking has made
the rpm verify command useless to most sysadmins, it can no longer be relied on
as a measure of change in a system.

The breakage comes in both initial package deployment and package updating,
attacking each package would be a mammoth job for any single person. So I ask,
because I honestly have no idea, how do we make this useful again for end users ?

If we don't plan to make it useful, we might as well remove it from the RPM
package and accept that we don't plan to track packages individual file changes.

So, where to from here ?

[LC (lc-redhat-bugs)]

I concur. My project is (was) planning to use the verify option as a check on
installed products.

[Bill (bill-redhat-bugs)]

Have you tested the yum-verify plugin in later Fedora? Is this closer to what you need?

[LC (lc-redhat-bugs)]

(In reply to comment #5)
> Have you tested the yum-verify plugin in later Fedora? Is this closer to what
> you need?

I don't think so at first blush:

from man yum-verify:
"verify-rpm
    Is meant to be 100% compatible with rpm -V output, and any differences should be considered as bugs."

on my test machine - "rpm -V" results:
[root@hugo pluginconf.d]# rpm -V auditproxy
CAP_AUDIT_WRITE not set
error: %verify(auditproxy-1.0.0-5.fc9.x86_64) scriptlet failed, exit status 1
CAP_AUDIT_WRITE not set
error: %verify(auditproxy-1.0.0-5.fc9.i386) scriptlet failed, exit status 1

and "yum verify-rpm results":
[root@hugo pluginconf.d]# yum verify-rpm auditproxy
Loaded plugins: refresh-packagekit, verify
verify-rpm done

After reading the config files which discuss highlighting, etc. maybe there is an error in my config (since I see no highlighting). But I doubt it would solve the real problem, in the original description, about prelink. After we install an rpm, the prelink daemon will find the executables installed and prelink them for us. If we've assigned a CAP to that executable, which we check in the rpm "%verify" scriptlet, after the prelink we no longer have the CAP set on the file. That's one of the real issues for me. The other is file signatures/checksums which also change after prelink. I think there is a "undo" prelink option we could run prior to testing checksums, however that doesn't solve the CAP loss.

[Bill (bill-redhat-bugs)]

prelink --undo is supposed to be done automatically for rpm verify. I suspect capability support would have to be added separately.

[Wade (wade-redhat-bugs)]

Bill,

Not really, as the problem really is that the packages are packaged incorrectly, things are modified after the install.

On this RHEL 5.1 (updated only from RHN) install with a few updates (all from RHN), have over 8000 modified files (see attachment). its not that the reporting is incorrect, its that the packages are being modified in the post install , which makes the report useless.

There are other things like PNG files and man page that have timestamps changed, and files that seem to be owned by 32 and 64 bit packages. ( /usr/libexec/camel-lock-helper-1.2 for example).

See the attached file for an example, this system has 836 package installed with over 8749 modified files.

Output of rpm -Va to be attached.

[Wade (wade-redhat-bugs)]

Created attachment 318160
List of modified files

[Panu (panu-redhat-bugs)]

The vast majority of timestamp differences comes directly from the way in which rpm operates on files shared between multiple packages, and yes it's especially bad on multilib systems like you've noticed: ELF files get their timestamps correctly as rpm completely skips the overlapping secondary arch ELF files, but for all others, the last file to come in during installation "wins". Primary arch packages are installed first to get the executables in place, which means that for everything else, the timestamps are those of the files in the secondary arch package. I had a brief look at it recently and I think it can be dealt with reasonably, and that alone solves the worst part of verify being next to useless on multilib.

As for the other verification related items:
- yes, capability support needs to be added separately
- rpm needs a sane API package+file verification, and python bindings so that yum can use that instead of rolling it's own (which at least currently misses %verifyscript functinality completely)

All of the above are on rpm upstream todo, which parts are ready in time for RHEL 6 is a different question however.

[Panu (panu-redhat-bugs)]

A small update here:
a) Upstream rpm has capability support now (both for packaging and verifying), and likely to be included in RHEL 6
b) Upstream rpm now filters out the bogus timestamp differences on verification: as files which differ only by timestamp aren't treated as conflicts, it makes no sense to have verification whine about them either. No reason why this wouldn't make it to RHEL 6. This makes verification output reasonably meaningful on multilib systems again.

FWIW, b) can trivially be backported to RHEL 4+5 too.

[Dmitry (dmitry-redhat-bugs)]

On our systems we get:

EL 5.2 x86_64:

 # rpm -Va | wc -l
 3289

EL 5.2 i686:

 # rpm -Va | wc -l
 21

which makes it absolutely ridiculous. rpm was *the* tool to use for both package maintenance and sanity checking. At this point on 64bit platforms this is not the case and it's really disappointing.

in reply to Comment #11 I'd like to mention that option (b) doesn't look like a good approach unless by bogus you mean that in cases where file belongs to 2 packages one is showing 'correct' timestamp and another doesn't. Otherwise if any timestamp change is considered to be bogus I would like to disagree as it is an important piece of information and could be used for system sanity checks. MD5 sums could be forged (binary crafted in a way to yield same md5 when in fact it's different). That is not to say timestamps are not as vulnerable if not more. But timestamps provide extra piece of information.

Take this comment as another vote for "Fixing" improper behavior of "rpm -V" on x86_64 whatever that means (prelink fix or rpm fix or package re-shuffle so there's no overlapping files).

[Dmitry (dmitry-redhat-bugs)]

forgot to mention that it doesn't make sense for us to use "yum" on servers as they have no outbound connectivity whatsoever. So we need RPM to function as it was intended: with -V providing correct information about changes on the system.

[Denise (denise-redhat-bugs)]

This is fixed in rpm 4.6.0. That version is actually present in 5.4 now. RHEL6 will have at least rpm 4.7. I'm setting this into MODIFIED so we do a sanity-test after the Fedora drop.

[Panu (panu-redhat-bugs)]

(In reply to comment #15)
> This is fixed in rpm 4.6.0. That version is actually present in 5.4 now.

Slight correction: rpm 4.6.0 is obviously not in RHEL 5.4, but the fix *could* easily be backported there too.

[errata-xmlrpc (errata-xmlrpc-redhat-bugs)]

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2009-1371.html

[Jeremy (jeremy-redhat-bugs)]

*** Bug 237171 has been marked as a duplicate of this bug. ***