libvirtd randomly crashes on xenial nodes with "realloc(): invalid next size"

[Matt Riedemann]

Seen here:

http://logs.openstack.org/25/386625/2/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/dbd07a2/logs/syslog.txt.gz#_Nov_03_15_27_42

Nov 03 15:27:42 ubuntu-xenial-osic-cloud1-s3700-5180728 libvirtd[14954]: *** Error in `/usr/sbin/libvirtd': realloc(): invalid next size: 0x0000563c6334ced0 ***

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Error%20in%20%60%2Fusr%2Fsbin%2Flibvirtd'%3A%20realloc()%3A%20invalid%20next%20size%5C%22%20AND%20tags%3A%5C%22syslog.txt%5C%22&from=7d

This doesn't hit often, but it has hit in the gate.

[Matt Riedemann (mriedem)]

Removed nova from this bug as there is likely not anything nova can do about this, it's a bug in libvirt 1.3.1 on xenial.

[Christian Ehrhardt  (paelzer)]

I don't know why I got a notification of this today ?!, but well here I am looking at it.

I've found bugs like that (The signatures only slightly differ) reported very sporadically back til 2011. So it seems a long standing bug at least the overall exit path (might have different root causes).
But it was never reproducible enough.

Seeing 16 occurrences last week in your Kibana seems to be closer to reproduce than ever.

Yet I wanted to write what is needed and in terms of progress-already-being-made I found another bug which covered more or less what I wanted to write.
Therefore I wanted to ask - is this a dup of 1643911?

[Christian Ehrhardt  (paelzer)]

Hi (cross post I know),
there is a bit of a "somewhat dup" context around the list of the following bugs:
- bug 1646779
- bug 1643911
- bug 1638982
- bug 1673483

Unfortunately, I’ve never hit these bugs in any of my libvirt/qemu runs and these are literally thousands every day due to all the automated tests. I also checked with our Ubuntu OpenStack Team and they didn't see them so far either. That makes it hard to debug them in depth as you will all understand.

But while the “signature” on each bug is different, they share a lot of things still (lets call it the bug "meta signature"):
- there is no way to recreate yet other than watching gate test crash statistics
- they seem to haunt the openstack gate testing more than anything else
- most are directly or indirectly related to memory corruption

As I’m unable to reproduce any of these bugs myself, I’d like to get some help from anyone that can help to recreate. Therefore I ask all people affected (mostly the same on all these bugs anyway) to test the PPAs I created for bug 1673483. That is the one bug where thanks to the great help of Kashyp, Matt and Dan (and more) at least a potential fix was identified.

That means two ppa's for you to try:
- Backport of the minimal recommended fix: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/2620
- Backport of the full series of related fixes: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/2619

Especially since the potential error of these fixes refers to almost anything from memleak to deadlock there is a chance that they all might be caused by the same root cause.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Clark Boylan (cboylan)]

It should be noted that after switching to using Ubuntu Cloud Archive which includes newer libvirt this issue went away in the gate.