mono: GC_install_header() in libgc/headers.c will segfault if mmap() fails

[Peter Maydell]

If mmap() fails mono can crash with a segfault rather than handling the memory allocation failure cleanly. This cropped up in bug 816791 (where mono is running under qemu; qemu is prone to returning failure from mmap()).

This is the function GC_install_header from libgc/headers.c:

/* Install a header for block h. */
/* The header is uninitialized. */
/* Returns the header or 0 on failure. */
struct hblkhdr * GC_install_header(h)
register struct hblk * h;
{
    hdr * result;

    if (!get_index((word) h)) return(0);
    result = alloc_hdr();
    SET_HDR(h, result);
# ifdef USE_MUNMAP
result -> hb_last_reclaimed = GC_gc_no;
# endif
    return(result);
}

The comment correctly claims we can return 0 on failure, because alloc_hdr() might return 0. However if USE_MUNMAP is defined then before we return 0 we will try to dereference it, and crash instead.

Found with mono 2.6.7-5ubuntu3, but this bug appears to still be present in upstream:
https://github.com/mono/mono/blob/master/libgc/headers.c#L201