gmond crashes when an NFS mount is active

[Dan Shick]

[IMPACT]

 * Current gmond crashes when an NFS mount is active due to stack corruption detection complaining about |
sscanf(procline, "%s %s %s %s ", device, mount, type, mode);| when used with too small buffers.

 * Fixed three years ago upstream (https://github.com/ganglia/monitor-core/commit/883f169353bdd58df10f21fb4e78541f53bc0f17).

 * Red Hat already backported a fix a year ago (https://bugzilla.redhat.com/show_bug.cgi?id=689483).

[TESTCASE]

 * Make sure the ganglia moddisk.so module is loaded.

 * Use NFS for some disk mount

 * Start gmond (`gmond -f`)

 * Ensure no crash for "stack smashing detected"

[Regression Potential]

 * None known...

Previous description:

root@mymachine:/etc/init.d# lsb_release -rd
Description: Ubuntu 11.10
Release: 11.10

root@mymachine# apt-cache policy ganglia-monitor
ganglia-monitor:
  Installed: 3.1.7-2
  Candidate: 3.1.7-2
  Version table:
 *** 3.1.7-2 0
        500 http://us.archive.ubuntu.com/ubuntu/ oneiric/universe i386 Packages
        100 /var/lib/dpkg/status

What I expected to happen: gmond should stay running regardless of whether I mount an NFS mount

What actually happened: gmond crashes when the system has an NFS mount active.

It looks like a manifestation of this behavior:

http://<email address hidden>/msg05118.html

With NFS mount:

root@mymachine# /usr/sbin/gmond -f -c /etc/ganglia/gmond.conf
*** stack smashing detected ***: /usr/sbin/gmond terminated
======= Backtrace: =========
/lib/i386-linux-gnu/libc.so.6(__fortify_fail+0x45)[0xb776b8d5]
/lib/i386-linux-gnu/libc.so.6(+0xe7887)[0xb776b887]
/usr/lib/ganglia/moddisk.so(+0x4844)[0xb73cd844]
/usr/lib/ganglia/moddisk.so(+0x473c)[0xb73cd73c]
[0x746e756f]
======= Memory map: ========
08048000-08055000 r-xp 00000000 08:06 659247 /usr/sbin/gmond
08055000-08056000 r--p 0000c000 08:06 659247 /usr/sbin/gmond
08056000-08057000 rw-p 0000d000 08:06 659247 /usr/sbin/gmond
08057000-08058000 rw-p 00000000 00:00 0
09495000-094b6000 rw-p 00000000 00:00 0 [heap]
b72ad000-b72c9000 r-xp 00000000 08:06 1967034 /lib/i386-linux-gnu/libgcc_s.so.1
b72c9000-b72ca000 r--p 0001b000 08:06 1967034 /lib/i386-linux-gnu/libgcc_s.so.1
b72ca000-b72cb000 rw-p 0001c000 08:06 1967034 /lib/i386-linux-gnu/libgcc_s.so.1
b72e1000-b72ec000 r-xp 00000000 08:06 1967054 /lib/i386-linux-gnu/libnss_files-2.13.so
b72ec000-b72ed000 r--p 0000a000 08:06 1967054 /lib/i386-linux-gnu/libnss_files-2.13.so
b72ed000-b72ee000 rw-p 0000b000 08:06 1967054 /lib/i386-linux-gnu/libnss_files-2.13.so
b72ee000-b72f8000 r-xp 00000000 08:06 1967058 /lib/i386-linux-gnu/libnss_nis-2.13.so
b72f8000-b72f9000 r--p 00009000 08:06 1967058 /lib/i386-linux-gnu/libnss_nis-2.13.so
b72f9000-b72fa000 rw-p 0000a000 08:06 1967058 /lib/i386-linux-gnu/libnss_nis-2.13.so
b72fa000-b730f000 r-xp 00000000 08:06 1967048 /lib/i386-linux-gnu/libnsl-2.13.so
b730f000-b7310000 r--p 00015000 08:06 1967048 /lib/i386-linux-gnu/libnsl-2.13.so
b7310000-b7311000 rw-p 00016000 08:06 1967048 /lib/i386-linux-gnu/libnsl-2.13.so
b7311000-b7313000 rw-p 00000000 00:00 0
b7321000-b7329000 rw-p 00000000 00:00 0
b7329000-b732f000 r-xp 00000000 08:06 787490 /usr/lib/ganglia/modsys.so
b732f000-b7330000 r--p 00005000 08:06 787490 /usr/lib/ganglia/modsys.so
b7330000-b7341000 rw-p 00006000 08:06 787490 /usr/lib/ganglia/modsys.so
b7341000-b7349000 rw-p 00000000 00:00 0
b7349000-b734f000 r-xp 00000000 08:06 787489 /usr/lib/ganglia/modproc.so
b734f000-b7350000 r--p 00005000 08:06 787489 /usr/lib/ganglia/modproc.so
b7350000-b7361000 rw-p 00006000 08:06 787489 /usr/lib/ganglia/modproc.so
b7361000-b7369000 rw-p 00000000 00:00 0
b7369000-b736f000 r-xp 00000000 08:06 787486 /usr/lib/ganglia/modnet.so
b736f000-b7370000 r--p 00005000 08:06 787486 /usr/lib/ganglia/modnet.so
b7370000-b7381000 rw-p 00006000 08:06 787486 /usr/lib/ganglia/modnet.so
b7381000-b7389000 rw-p 00000000 00:00 0
b7389000-b738f000 r-xp 00000000 08:06 787485 /usr/lib/ganglia/modmem.so
b738f000-b7390000 r--p 00005000 08:06 787485 /usr/lib/ganglia/modmem.so
b7390000-b73a1000 rw-p 00006000 08:06 787485 /usr/lib/ganglia/modmem.so
b73a1000-b73a9000 rw-p 00000000 00:00 0
b73a9000-b73af000 r-xp 00000000 08:06 787483 /usr/lib/ganglia/modload.so
b73af000-b73b0000 r--p 00005000 08:06 787483 /usr/lib/ganglia/modload.so
b73b0000-b73c1000 rw-p 00006000 08:06 787483 /usr/lib/ganglia/modload.so
b73c1000-b73c9000 rw-p 00000000 00:00 0
b73c9000-b73cf000 r-xp 00000000 08:06 787484 /usr/lib/ganglia/moddisk.so
b73cf000-b73d0000 r--p 00005000 08:06 787484 /usr/lib/ganglia/moddisk.so
b73d0000-b73e1000 rw-p 00006000 08:06 787484 /usr/lib/ganglia/moddisk.so
b73e1000-b73e9000 rw-p 00000000 00:00 0
b73e9000-b73ef000 r-xp 00000000 08:06 787482 /usr/lib/ganglia/modcpu.so
b73ef000-b73f0000 r--p 00005000 08:06 787482 /usr/lib/ganglia/modcpu.so
b73f0000-b7401000 rw-p 00006000 08:06 787482 /usr/lib/ganglia/modcpu.so
b7401000-b740b000 rw-p 00000000 00:00 0
b740c000-b740d000 r--p 0043a000 08:06 662632 /usr/lib/locale/locale-archive
b740d000-b744d000 r--p 002bd000 08:06 662632 /usr/lib/locale/locale-archive
b744d000-b764d000 r--p 00000000 08:06 662632 /usr/lib/locale/locale-archive
b764d000-b764e000 rw-p 00000000 00:00 0
b764e000-b7651000 r-xp 00000000 08:06 1967024 /lib/i386-linux-gnu/libdl-2.13.so
b7651000-b7652000 r--p 00002000 08:06 1967024 /lib/i386-linux-gnu/libdl-2.13.so
b7652000-b7653000 rw-p 00003000 08:06 1967024 /lib/i386-linux-gnu/libdl-2.13.so
b7653000-b7657000 r-xp 00000000 08:06 1967092 /lib/i386-linux-gnu/libuuid.so.1.3.0
b7657000-b7658000 r--p 00003000 08:06 1967092 /lib/i386-linux-gnu/libuuid.so.1.3.0
b7658000-b7659000 rw-p 00004000 08:06 1967092 /lib/i386-linux-gnu/libuuid.so.1.3.0
b7659000-b765a000 rw-p 00000000 00:00 0
b765a000-b7680000 r-xp 00000000 08:06 1967029 /lib/i386-linux-gnu/libexpat.so.1.5.2
b7680000-b7681000 ---p 00026000 08:06 1967029 /lib/i386-linux-gnu/libexpat.so.1.5.2
b7681000-b7683000 r--p 00026000 08:06 1967029 /lib/i386-linux-gnu/libexpat.so.1.5.2
b7683000-b7684000 rw-p 00028000 08:06 1967029 /lib/i386-linux-gnu/libexpat.so.1.5.2
b7684000-b77fa000 r-xp 00000000 08:06 1967013 /lib/i386-linux-gnu/libc-2.13.so
b77fa000-b77fc000 r--p 00176000 08:06 1967013 /lib/i386-linux-gnu/libc-2.13.so
b77fc000-b77fd000 rw-p 00178000 08:06 1967013 /lib/i386-linux-gnu/libc-2.13.so
b77fd000-b7800000 rw-p 00000000 00:00 0
b7800000-b7817000 r-xp 00000000 08:06 1967073 /lib/i386-linux-gnu/libpthread-2.13.so
b7817000-b7818000 r--p 00016000 08:06 1967073 /lib/i386-linux-gnu/libpthread-2.13.so
b7818000-b7819000 rw-p 00017000 08:06 1967073 /lib/i386-linux-gnu/libpthread-2.13.so
b7819000-b781b000 rw-p 00000000 00:00 0
b781b000-b784e000 r-xp 00000000 08:06 672170 /usr/lib/libapr-1.so.0.4.5
b784e000-b784f000 r--p 00032000 08:06 672170 /usr/lib/libapr-1.so.0.4.5
b784f000-b7850000 rw-p 00033000 08:06 672170 /usr/lib/libapr-1.so.0.4.5
b7850000-b785a000 r-xp 00000000 08:06 672024 /usr/lib/i386-linux-gnu/libconfuse.so.0.0.0
b785a000-b785b000 r--p 00009000 08:06 672024 /usr/lib/i386-linux-gnu/libconfuse.so.0.0.0
b785b000-b785c000 rw-p 0000a000 08:06 672024 /usr/lib/i386-linux-gnu/libconfuse.so.0.0.0
b785c000-b785d000 rw-p 00000000 00:00 0
b785d000-b789a000 r-xp 00000000 08:06 1967070 /lib/i386-linux-gnu/libpcre.so.3.12.1
b789a000-b789b000 r--p 0003c000 08:06 1967070 /lib/i386-linux-gnu/libpcre.so.3.12.1
b789b000-b789c000 rw-p 0003d000 08:06 1967070 /lib/i386-linux-gnu/libpcre.so.3.12.1
b789c000-b78ae000 r-xp 00000000 08:06 668473 /usr/lib/libganglia-3.1.7.so.0.0.0
b78ae000-b78af000 r--p 00011000 08:06 668473 /usr/lib/libganglia-3.1.7.so.0.0.0
b78af000-b78b1000 rw-p 00012000 08:06 668473 /usr/lib/libganglia-3.1.7.so.0.0.0
b78b1000-b78b9000 r-xp 00000000 08:06 1967050 /lib/i386-linux-gnu/libnss_compat-2.13.so
b78b9000-b78ba000 r--p 00007000 08:06 1967050 /lib/i386-linux-gnu/libnss_compat-2.13.so
b78ba000-b78bb000 rw-p 00008000 08:06 1967050 /lib/i386-linux-gnu/libnss_compat-2.13.so
b78bb000-b78c9000 rw-p 00000000 00:00 0
b78c9000-b78ca000 r-xp 00000000 00:00 0 [vdso]
b78ca000-b78e8000 r-xp 00000000 08:06 1967000 /lib/i386-linux-gnu/ld-2.13.so
b78e8000-b78e9000 r--p 0001d000 08:06 1967000 /lib/i386-linux-gnu/ld-2.13.so
b78e9000-b78ea000 rw-p 0001e000 08:06 1967000 /lib/i386-linux-gnu/ld-2.13.so
bfa4b000-bfa6c000 rw-p 00000000 00:00 0 [stack]
Aborted

ProblemType: Bug
DistroRelease: Ubuntu 11.10
Package: ganglia-monitor 3.1.7-2 [modified: usr/share/doc/ganglia-monitor/changelog.Debian.gz]
ProcVersionSignature: Ubuntu 3.0.0-14.23-generic-pae 3.0.9
Uname: Linux 3.0.0-14-generic-pae i686
ApportVersion: 1.23-0ubuntu4
Architecture: i386
Date: Sun Jan 1 16:45:03 2012
InstallationMedia: Ubuntu 11.10 "Oneiric Ocelot" - Release i386 (20111012)
ProcEnviron:
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: ganglia
UpgradeStatus: No upgrade log present (probably fresh install)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Chris Adamson (addo81)]

A fix for this bug was committed to Fedora

https://bugzilla.redhat.com/show_bug.cgi?id=689483

libmetrics/linux/metrics.c

line 1231:

char mount[128], device[128], type[32], mode[128];

change this to

char mount[2048], device[2048], type[32], mode[2048];

in order to accommodate the longer strings in /proc/mounts for NFS and automounts

[teonghan (teonghan)]

Hi all,

If I did an "apt-get upgrade" or specifically, apt-get install libc6, ganglia-monitor (gmond) can be started without problem, even if there is a NFS partition. Can anyone verify this?

Thanks.

[teonghan (teonghan)]

Hi all,

Sorry, should have stated that the version of libc6 I am using is 2.13-20ubuntu5.1.

Thanks.

[dab (dabosuke)]

I got the same problem with Ubuntu 12.04.

[Teppo Sulavuori (teppo-a)]

This has been fixed in upstream since version 3.2.0

https://github.com/ganglia/monitor-core/commit/883f169353bdd58df10f21fb4e78541f53bc0f17

(I'm a bit sad that Precise being an LTS version did not get a newer version of ganglia-monitor since current version 3.1.7 is already over two years old.)

[Mark Asbach (mark-asbach)]

A quick fix for users not wanting to wait for an update or build gmond manually is to change the gmond.conf to not load the disks module.

Comment out the respective sections of /etc/ganglia/gmond.conf as follows:

  module {
    name = "cpu_module"
    path = "/usr/lib/ganglia/modcpu.so"
  }
/*
  module {
    name = "disk_module"
    path = "/usr/lib/ganglia/moddisk.so"
  }
*/
  module {
    name = "load_module"
    path = "/usr/lib/ganglia/modload.so"
  }

[...]

/*
collection_group {
  collect_every = 1800
  time_threshold = 3600
  metric {
    name = "disk_total"
    value_threshold = 1.0
    title = "Total Disk Space"
  }
}

collection_group {
  collect_every = 40
  time_threshold = 180
  metric {
    name = "disk_free"
    value_threshold = 1.0
    title = "Disk Space Available"
  }
  metric {
    name = "part_max_used"
    value_threshold = 1.0
    title = "Maximum Disk Space Used"
  }
}

*/

[Niklas Edmundsson (niklas-edmundsson)]

This should be fixed ASAP, Ganglia is an important component in many server environments which is the target of LTS releases.

[Mike Johnson (mikejohnson)]

I'm having this problem in 12.04 as well.

[Exilist (sandro-grundmann)]

i can confirm this, ganglia/gmond is not starting on 12.04 and nfs-mounts

[Trevor Robinson (scurrilous)]

I hit this issue as well. My workaround was to install the ganglia packages from quantal, which picked up a substantially newer version that includes the upstream fix.

IMHO, the quantal packages should be backported to precise, given its status as an LTS release. Ganglia is widely used to monitor server apps like Hadoop, and those environments often use NFS.

[Reed Loden (reed)]

quantal has 3.3.8, so it's not affected by this issue.

[Reed Loden (reed)]

My attempt at a debdiff for this fix, but I don't think I did it right. :(

[Jamie Strandboge (jdstrand)]

Thanks for the debdiff! There are a few things that need updating:
 * this should target precise-proposed, not precise
 * the version should be 3.1.7-2ubuntu1.1 as per https://wiki.ubuntu.com/StableReleaseUpdates
 * there is an undocumented change to debian/source/options. I see why you want to do this, but it doesn't seem to work because of the upstream Debian changes in debian-changes-3.1.7-2 that add config.guess and config.sub. I've updated debian-changes-3.1.7-2 to not add config.guess and config.sub and this seems to work.

Since the first two were simple and the third less so, I went ahead an made these changes and uploaded to precise-proposed.

[Jamie Strandboge (jdstrand)]

Unsubscribing ubuntu-sponsors.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Clint Byrum (clint-fewbar)]

Hello Dan, or anyone else affected,

Accepted ganglia into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/ganglia/3.1.7-2ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Robert Dyer (psybers)]

Updated to 3.1.7-2ubuntu1.1 from precise-proposed. Re-enabled the disk module. Ganglia now starts up instead of segfaulting and appears to be working normal.

[Joseph Holsten (i4-joseph)]

This is preventing my company from upgrading from 10.04 to 12.04. Thanks for working on this.

[Joseph Holsten (i4-joseph)]

Confirming that the package in -proposed also resolved the issue for us.

[Scott Kitterman (kitterman)]

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ganglia - 3.1.7-2ubuntu1.1

---------------
ganglia (3.1.7-2ubuntu1.1) precise-proposed; urgency=low

  [ Reed Loden ]
  * Fix buffer overflow in moddisk.so. (LP: #910678)
  * debian/source/options: ignore autogenerated files

  [ Jamie Strandboge ]
  * debian/patches/debian-changes-3.1.7-2: remove addition of config.guess
    and config.sub
 -- Reed Loden <email address hidden> Mon, 15 Oct 2012 11:22:10 -0700