URPMI has issues with db-5.3.15

[Jeff Johnson]

(from Per Oyvind)
RPM+db53 has issues under URPMI.

The command "urpmi --replacepkgs perl-base" should reproduce one issue.

The other issue previously reported was with the rpmdb conversion
undertaken by URPMI and iurt (and likely jurt) Mandriva build systems.

[Per Øyvind Karlsen (proyvind)]

It's not specific to urpmi, the reproducer was only meant as a oneliner to reproduce it easily (with downloading of packages
etc. taken care of)..

Anyways, I've bumped to db 5.3 again, and here's the actual error encountered:
[root@t61 rpms]# rpm -Uvh python-2.7.2-4-mdv2012.0.x86_64.rpmerror: db3cpget:db3.c:1443: dbcursor->pget(-30999): BDB0063 DB_BUFFER_SMALL: User memory too small for return value
rpm: rpmdb.c:2192: rpmmiNext: Assertion `0' failed.
Avbrutt (SIGABRT) (core dumped)

[Per Øyvind Karlsen (proyvind)]

I notice that this only happens when upgrading packages, ie. 'rpm -Uvh --force foo.rpm' will fail, but if I do 'rpm -E --justdb --nodeps foo; rpm -Uvh foo.rpm' package installation will succeed.

[Jeff Johnson (n3npq)]

You have a mixture of disablers on all the reproducers.

Are there any issues on "production" paths without --force and --nodeps
and all the other gook?

[Jeff Johnson (n3npq)]

And split --force into the specific option that is failing please.
Its likely --replacepkgs (presumably consistent with what urpmi
calls --replacepkgs).

[Jeff Johnson (n3npq)]

Partially reproduced (on i586, and with db-5.2.x in /var/lib/rpm):

D: ========== --- python-2.7.2-3.i586 i586/linux 0x0
D: opening db index /var/lib/rpm/Requirename thread:rdonly:auto_commit mode=0x0
D: Requires: python-elementtree YES (added provide)
D: Requires: python(abi) = 2.7 YES (added provide)
D: Requires: python(abi) = 2.7 YES (added provide)
error: db3cpget:db3.c:1443: dbcursor->pget(-30999): BDB0063 DB_BUFFER_SMALL: User memory too small for return value
rpm: rpmdb.c:2192: rpmmiNext: Assertion `0' failed.

Note the failure on the already installed python package while checking
that erased dependencies are still satisfied while an identical dependency
is checked.

[Per Øyvind Karlsen (proyvind)]

Notice that it happens with regular upgrades as well, ie. the --force used is merely to more easily reproduce the issue..

[Jeff Johnson (n3npq)]

Boring in through
    $ sudo ./rpm -Uvv --replacepkgs --hdrdebug --rpmmidebug python-2.7.2-4-mdv2012.0.i586.rpm
with "debug" added in /usr/lib/rpm/macros
    %_dbi_config_3_Requirename %{_dbi_btconfig} %{?_bt_dupsort} debug
the relevant accesses are

--> rpmmiInit(0x8eaa698, Requirename, 0x8f3eaa0[0]="python(abi)") dbi 0x8f34a30 mi 0x8f34920
<-- rpmmiPrune(0x8f34920, 0x8e9bbe0[1], 1) rc 0 h# 4264
--> rpmmiNext(0x8f34920) dbi 0x8f34a30(Requirename)
<-- db3copen(0x8f34a30,(nil),0x8f3493c,0x0) dbc 0x8f342d0 0x0 rc 0
<-- db3cget(0x8f34a30,0x8f342d0,0xbfb23178,0xbfb231b0,0x1a) rc -30999
 flags: DB_SET
   key: 0x8f33f18[11] 0x0 "python(abi)"
  data: (nil)[9620] 0x800<USERMEM>
<-- db3cpget(0x8f34a30,0x8f342d0,0xbfb23178,0xbfb23194,0xbfb231b0,0x1a) rc 0
 flags: DB_SET
   key: 0x8f33f18[11] 0x0 "python(abi)"
  pkey: 0x8f3ea90[4] 0x0 0x4e010000
  data: 0xb7444000[9620] 0x800<USERMEM>
<-- rpmmiGet(0x8f34a30(Requirename),0x8f342d0,0xbfb23178,0xbfb231b0,0x1a) rc 0
--> h 0x8e9c2d0 ++ 1 headerLoad at header.c:1003
D: Requires: python(abi) = 2.7 YES (added provide)
--> rpmmiNext(0x8f34920) dbi 0x8f34a30(Requirename)
<-- db3cget(0x8f34a30,0x8f342d0,0xbfb23178,0xbfb231b0,0x11) rc -30999
 flags: DB_NEXT_DUP
   key: 0x8f02108[11] 0x0 "python(abi)"
  data: (nil)[22812] 0x800<USERMEM>
<-- db3cpget(0x8f34a30,0x8f342d0,0xbfb23178,0xbfb23194,0xbfb231b0,0x11) rc 0
 flags: DB_NEXT_DUP
   key: 0x8f02108[11] 0x0 "python(abi)"
  pkey: 0x8f3ea90[4] 0x0 0xbd030000
  data: 0xb61ab000[6784] 0x800<USERMEM>
<-- rpmmiGet(0x8f34a30(Requirename),0x8f342d0,0xbfb23178,0xbfb231b0,0x11) rc 0
--> h 0x8e9c2d0 -- 1 miFreeHeader at rpmdb.c:1539
--> h 0x8eaf998 ++ 1 headerLoad at header.c:1003
D: Requires: python(abi) = 2.7 YES (added provide)
--> rpmmiNext(0x8f34920) dbi 0x8f34a30(Requirename)
<-- db3cget(0x8f34a30,0x8f342d0,0xbfb23178,0xbfb231b0,0x11) rc -30999
 flags: DB_NEXT_DUP
   key: 0x8f02108[11] 0x0 "python(abi)"
  data: (nil)[6392] 0x800<USERMEM>
error: db3cpget:db3.c:1443: dbcursor->pget(-30999): BDB0063 DB_BUFFER_SMALL: User memory too small for return value
<-- db3cpget(0x8f34a30,0x8f342d0,0xbfb23178,0xbfb23194,0xbfb231b0,0x11) rc -30999
 flags: DB_NEXT_DUP
   key: 0x8f02108[11] 0x0 "python(abi)"
  pkey: 0x8f3ea90[4] 0x0 0xc1030000
  data: 0xb7445000[7500] 0x800<USERMEM>
<-- rpmmiGet(0x8f34a30(Requirename),0x8f342d0,0xbfb23178,0xbfb231b0,0x11) rc -30999
rpm: rpmdb.c:2192: rpmmiNext: Assertion `0' failed.

In short: db-5.3.15 is failing a repeated lookup of an identical key.

[Jeff Johnson (n3npq)]

The code is attempting the following:

1) Attempt db3cget() with a NULL pointer that returns "rc -30999" but
fills in the size of the blob that needs to be returned.

2) A "anonymous memory" allocation using mmap is allocated.

3) Attempt db3cpget() to retrieve (secondary -> primary) blob.

4) The blob memory is set to PROT_READ.

The DB_SET successful access is claiming a size of 9620 and db3cpget returns same size.

The 1st DB_NEXTDUP success is claiming 22812b are needed but returning 6784b.

The 2nd DB_NEXTDUP failure is claiming 6392b are needed (which is smaller than 6784b).

The testable hypothesis is
    DB_NEXTDUP isn't filling in the needed blob size correctly in db-5.3.15.
Time to tinker.

[Jeff Johnson (n3npq)]

This quick-and-dirty hack to map 10x the needed memory avoids the problem:

Index: rpmdb/rpmdb.c
===================================================================
RCS file: /v/rpm/cvs/rpm/rpmdb/rpmdb.c,v
retrieving revision 1.392.2.9
diff -p -u -w -r1.392.2.9 rpmdb.c
--- rpmdb/rpmdb.c 7 Nov 2011 15:33:01 -0000 1.392.2.9
+++ rpmdb/rpmdb.c 21 Feb 2012 16:02:41 -0000
@@ -2081,7 +2081,7 @@ static int rpmmiGet(dbiIndex dbi, DBC *
  vp->flags |= DB_DBT_USERMEM;
  rc = dbiGet(dbi, dbcursor, kp, vp, flags);
  if (rc == DB_BUFFER_SMALL) {
- size_t uhlen = vp->size;
+ size_t uhlen = 10 * vp->size;
      void * uh = mmap(NULL, uhlen, _prot, _flags, _fdno, _off);
      if (uh == NULL || uh == (void *)-1)
   fprintf(stderr,

I shall add another DB_SET (to attempt to get the blob size accurately) next.

[Jeff Johnson (n3npq)]

Attempting DB_SET did not help. Setting a minimum size of ~64Kb
is likely viable until the real flaw can be identified. Meanwhile there's
a few more things to try first.

[Jeff Johnson (n3npq)]

Hmmm: what rpm is doing SHOULD continue to work.

Read notes about DB_DBT_USERMEM here:
    http://docs.oracle.com/cd/E17076_02/html/api_reference/C/frame_main.html
(you will need a Oracle login)

Meanwhile the only way I can get past the issue is in comment #9.

I tried "= 4 * vp->size" and that was too small. I'll dig deeper when I have brunch
w my brain Sunday morning, sigh.

[Per Øyvind Karlsen (proyvind)]

d'oh, even with the workoing in #9, I'm still hitting this:

[root@t61 rpms]# rpm -Uvh glibc-2.15-1-mdv2012.0.x86_64.rpm
error: db3cpget:db3.c:1443: dbcursor->pget(-30999): BDB0063 DB_BUFFER_SMALL: User memory too small for return value
rpm: rpmdb.c:2192: rpmmiNext: Assertion `0' failed.
Avbrutt (SIGABRT) (core dumped)

[Jeff Johnson (n3npq)]

Then diagnose as I did. 'taint hard ...

[Jeff Johnson (n3npq)]

The workaround is of course dependent on what was returned (which might be random)
Try
    size_t uhlen = 1024 * 1024;
to see whether problem disappears.

[Jeff Johnson (n3npq)]

Dunno what resolution was done in Mandriva: I am seeing
no issues in buildbot's using db-5.3.15 for months.

[Per Øyvind Karlsen (proyvind)]

I suspect difficulty reproducing is related to size of rpmdb, so here's a reproducer for ya with mine:
mkdir -p /tmp/db60reproducer/var/lib/rpm
curl http://proyvind.net/~peroyvind/rpm/db60/Packages.xz | unxz > /tmp/db60reproducer/var/lib/rpm/Packages
rpm --justdb -Uvh --nofsync -vvv --root /tmp/db60reproducer/ http://proyvind.net/~peroyvind/rpm/db60/lib64gcc1-4.8.3_2013.11-1-omv2013.0.x86_64.rpm

[Jeff Johnson (n3npq)]

You are running a heavily patched custom build of RPM5 on
an operating system that I cannot type "make install" from CVS.

I also do not have any access to a a machine or VM on which
to attempt a reproducer, largely because lack of "make install"
prevents meaningful rpm development.

You also have not stated what you expect (or what is wrong), nor
attempted/reported any of the detailed debugging steps that I have
already given you.

And this is db60 not db53. We both know that using db52 works reliably,
which would seem to indicate a change in Berkeley DB, not rpm, is the root
cause of the problem.

[Per Øyvind Karlsen (proyvind)]

You were asking for a reproducer, I've provided one.

The issue applies to both db53 & db60, so whichever being used isn't really relevant.

From what I can see, the issue seems to only be triggered when having a large enough rpmdb (which is what I suspect why it is that you cannot reproduce the issue easily with buildbots), hence why I provided a copy of mine and a sample package to test with in order to reproduce the issue.
AFAIK it shouldn't be dependent on mandriva specific rpm package.

Whether there's a bug in rpm or bdb isn't really that relevant to the matter, it's still a blocker bug for using > db52 with rpm that needs to be fixed (wherever) regardless of..

The workaround provided by Mark Hatle at http://rpm5.org/community/rpm-devel/5393.html seems to be working though.

Dunno what more information or anything that I can provide...

[Jeff Johnson (n3npq)]

You could provide:

1) the same debugging info used before to confirm the same code paths,
2) a statement or two about

You could cease

1) guessing that the cause is too large a database
2) claiming "blocker" for what? your life? your distro? OpenMandriva? Whatever ...
3) asserting irrelevancies like "whether there's a bug in in rpm or bdb isn't really that relevant"

You have been told multiple times based on the previous debugging trace
that this is a regression in Berkeley DB that needs to be fixed in Berkeley DB,
not with hack-o-rounds in rpm.

You already have a fix: use bdb 5.2. (Mark Hatle's patch is a workaround, not a fix).

Quoted from the link you have provided:
    I have a workaround patch, but it's far from a fix.

And you've made a whole lotta noise about an ancient (and still not properly fixed) bug
for no important reason.

[Per Øyvind Karlsen (proyvind)]

For me I've pretty much just not cared about this for quite a while, but since you brought up migration to bdb 60 for mandriva on IRC recently and was unable to reproduce the bug blocking it, I tried providing you with a reproducer (after just having trying it with vanilla rpm-5.4.14, I realize that now rpm segfaults when trying to make use of it, so the reproducer might not be too useful without my set of patches..).

As this bug prevents migration to bdb >= 5.3, I consider it as a blocker bug for what Mandriva is concerned if it weren't for that mandriva rpm package has been patched to use db 5.2.

As vanilla rpm insists on db6.0 and have to be patched to use any other version, I'd consider it as a blocker bug for rpm upstream as well.

If you refuse to care about it, then fine, but an ostrich approach to the problem won't really solve it.

That you bring up the age of this bug is certainly not something making the bug less relevant, but rather makes one worried about how this serious bug have been known and remained unfixed for so long...

Oh well, I merely tried being helpful in order for the bug to be resolved, but as you point out; it works just fine for me with bdb 5.2, so I'll just join you in ignoring this bug (which doesn't affect myself at least).

[Jeff Johnson (n3npq)]

I don't feed your trolls any further.