libvirtd crashes with double free or corruption (out)

[Tom Eichhorn]

I saw libvirtd crash with the attached error when someone tried to start a VM via openstack nova.

Anyone with hints howto debug this further?

Thanks,
Tom

[Maik Zumstrull (m-zumstrull)]

I'm pretty sure this bug is actually in libnss-extrausers (see top of stack trace).

libvirtd is trying a getgrouplist, which is calling out to the various NSS modules through glibc.

In libnss-extrausers, this construct in group.c:

static FILE *groupsfile = NULL;

[...]

enum nss_status _nss_extrausers_endgrent(void) {
        if (groupsfile != NULL) {
                fclose(groupsfile);
                groupsfile = NULL;
        }
        return NSS_STATUS_SUCCESS;
}

is obviously not in any way thread safe, even though the module advertises itself as reentrant.

[Christian Ehrhardt  (paelzer)]

Given the pre-analysis I agree with Maik, reassigning to libnss-extrausers

[Maik Zumstrull (m-zumstrull)]

Note that this is https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=831390.

We have been running with a custom build of the package, that adds __thread to the relevant variables as suggested in that bug, and haven't seen any crashes in this module since.

> On 13 Dec 2016, at 16:30, ChristianEhrhardt <email address hidden> wrote:
>
> Given the pre-analysis I agree with Maik, reassigning to libnss-
> extrausers
>
> ** Package changed: libvirt (Ubuntu) => libnss-extrausers (Ubuntu)
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1602264
>
> Title:
> libvirtd crashes with double free or corruption (out)
>
> Status in libnss-extrausers package in Ubuntu:
> New
>
> Bug description:
> I saw libvirtd crash with the attached error when someone tried to
> start a VM via openstack nova.
>
> Anyone with hints howto debug this further?
>
> Thanks,
> Tom
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/libnss-extrausers/+bug/1602264/+subscriptions

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Olaf Seibert (oseibert-sys11)]

It is my opinion at this time that the bug is really in glibc. These functions like _nss_extrausers_endgrent() must share state from call to call, and therefore cannot possibly be thread safe.

Unfortunately, the fix with making the shared state thread-local could probably make problems for callers which try to be careful when using these functions, but still take into account the globally shared state (which isn't global any more with the patch). For instance, if one thread calls _nss_extrausers_setgrent() under some lock, and another thread later calls _nss_extrausers_endgrent() under some lock, this would reasonably be expected to work, but it won't with the thread-local state.

The getgrouplist() is documented as thread safe, while it calls these functions without locking.
I think the fix should be made in glibc by adding appropriate locking there.

I have filed a report for glibc: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1923738