Botan segfaults using GMP_MPZ

[Jerry Lundström]

Hi,

I'm working on the OpenDNSSEC project and we have SoftHSM that uses Botan.

On Ubuntu Server 10.04.3 running in a virtual environment (VirtualBox and VMware) some checks we do segfaults _ sometimes _, back trace is attached. It works on Debian 6.0.3 and the difference is a slightly newer Botan (1.8.6-2 on Ubuntu, 1.8.9-1 on Debian).

The segfault happens deep inside Botan and GMP and I haven't been able to find out why.

Code to produce the segfault can be found here: http://svn.opendnssec.org/releases/softHSM-1.3.0/checks/ .

Hope you can fix it or update the Botan library.

Cheers,
Jerry

1)
$ lsb_release -rd
Description: Ubuntu 10.04.3 LTS
Release: 10.04

2)
$ apt-cache policy libbotan-1.8.2
libbotan-1.8.2:
  Installed: 1.8.6-2
  Candidate: 1.8.6-2
  Version table:
 *** 1.8.6-2 0
        500 http://nl.archive.ubuntu.com/ubuntu/ lucid/universe Packages
        100 /var/lib/dpkg/status

$ apt-cache policy libgmp3c2
libgmp3c2:
  Installed: 2:4.3.2+dfsg-1ubuntu1
  Candidate: 2:4.3.2+dfsg-1ubuntu1
  Version table:
 *** 2:4.3.2+dfsg-1ubuntu1 0
        500 http://nl.archive.ubuntu.com/ubuntu/ lucid/main Packages
        100 /var/lib/dpkg/status

3)
'make check' / 'checks -z' to succeed in SoftHSM.

4)
'make check' / 'checks -z' segfaulted.