gmond crashes when an NFS mount is active
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
| ganglia (Ubuntu) |
Undecided
|
Unassigned | ||
| Precise |
Undecided
|
Reed Loden | ||
| Quantal |
Undecided
|
Unassigned |
Bug Description
[IMPACT]
* Current gmond crashes when an NFS mount is active due to stack corruption detection complaining about |
sscanf(procline, "%s %s %s %s ", device, mount, type, mode);| when used with too small buffers.
* Fixed three years ago upstream (https:/
* Red Hat already backported a fix a year ago (https:/
[TESTCASE]
* Make sure the ganglia moddisk.so module is loaded.
* Use NFS for some disk mount
* Start gmond (`gmond -f`)
* Ensure no crash for "stack smashing detected"
[Regression Potential]
* None known...
Previous description:
root@mymachine:
Description: Ubuntu 11.10
Release: 11.10
root@mymachine# apt-cache policy ganglia-monitor
ganglia-monitor:
Installed: 3.1.7-2
Candidate: 3.1.7-2
Version table:
*** 3.1.7-2 0
500 http://
100 /var/lib/
What I expected to happen: gmond should stay running regardless of whether I mount an NFS mount
What actually happened: gmond crashes when the system has an NFS mount active.
It looks like a manifestation of this behavior:
http://<email address hidden>
With NFS mount:
root@mymachine# /usr/sbin/gmond -f -c /etc/ganglia/
*** stack smashing detected ***: /usr/sbin/gmond terminated
======= Backtrace: =========
/lib/i386-
/lib/i386-
/usr/lib/
/usr/lib/
[0x746e756f]
======= Memory map: ========
08048000-08055000 r-xp 00000000 08:06 659247 /usr/sbin/gmond
08055000-08056000 r--p 0000c000 08:06 659247 /usr/sbin/gmond
08056000-08057000 rw-p 0000d000 08:06 659247 /usr/sbin/gmond
08057000-08058000 rw-p 00000000 00:00 0
09495000-094b6000 rw-p 00000000 00:00 0 [heap]
b72ad000-b72c9000 r-xp 00000000 08:06 1967034 /lib/i386-
b72c9000-b72ca000 r--p 0001b000 08:06 1967034 /lib/i386-
b72ca000-b72cb000 rw-p 0001c000 08:06 1967034 /lib/i386-
b72e1000-b72ec000 r-xp 00000000 08:06 1967054 /lib/i386-
b72ec000-b72ed000 r--p 0000a000 08:06 1967054 /lib/i386-
b72ed000-b72ee000 rw-p 0000b000 08:06 1967054 /lib/i386-
b72ee000-b72f8000 r-xp 00000000 08:06 1967058 /lib/i386-
b72f8000-b72f9000 r--p 00009000 08:06 1967058 /lib/i386-
b72f9000-b72fa000 rw-p 0000a000 08:06 1967058 /lib/i386-
b72fa000-b730f000 r-xp 00000000 08:06 1967048 /lib/i386-
b730f000-b7310000 r--p 00015000 08:06 1967048 /lib/i386-
b7310000-b7311000 rw-p 00016000 08:06 1967048 /lib/i386-
b7311000-b7313000 rw-p 00000000 00:00 0
b7321000-b7329000 rw-p 00000000 00:00 0
b7329000-b732f000 r-xp 00000000 08:06 787490 /usr/lib/
b732f000-b7330000 r--p 00005000 08:06 787490 /usr/lib/
b7330000-b7341000 rw-p 00006000 08:06 787490 /usr/lib/
b7341000-b7349000 rw-p 00000000 00:00 0
b7349000-b734f000 r-xp 00000000 08:06 787489 /usr/lib/
b734f000-b7350000 r--p 00005000 08:06 787489 /usr/lib/
b7350000-b7361000 rw-p 00006000 08:06 787489 /usr/lib/
b7361000-b7369000 rw-p 00000000 00:00 0
b7369000-b736f000 r-xp 00000000 08:06 787486 /usr/lib/
b736f000-b7370000 r--p 00005000 08:06 787486 /usr/lib/
b7370000-b7381000 rw-p 00006000 08:06 787486 /usr/lib/
b7381000-b7389000 rw-p 00000000 00:00 0
b7389000-b738f000 r-xp 00000000 08:06 787485 /usr/lib/
b738f000-b7390000 r--p 00005000 08:06 787485 /usr/lib/
b7390000-b73a1000 rw-p 00006000 08:06 787485 /usr/lib/
b73a1000-b73a9000 rw-p 00000000 00:00 0
b73a9000-b73af000 r-xp 00000000 08:06 787483 /usr/lib/
b73af000-b73b0000 r--p 00005000 08:06 787483 /usr/lib/
b73b0000-b73c1000 rw-p 00006000 08:06 787483 /usr/lib/
b73c1000-b73c9000 rw-p 00000000 00:00 0
b73c9000-b73cf000 r-xp 00000000 08:06 787484 /usr/lib/
b73cf000-b73d0000 r--p 00005000 08:06 787484 /usr/lib/
b73d0000-b73e1000 rw-p 00006000 08:06 787484 /usr/lib/
b73e1000-b73e9000 rw-p 00000000 00:00 0
b73e9000-b73ef000 r-xp 00000000 08:06 787482 /usr/lib/
b73ef000-b73f0000 r--p 00005000 08:06 787482 /usr/lib/
b73f0000-b7401000 rw-p 00006000 08:06 787482 /usr/lib/
b7401000-b740b000 rw-p 00000000 00:00 0
b740c000-b740d000 r--p 0043a000 08:06 662632 /usr/lib/
b740d000-b744d000 r--p 002bd000 08:06 662632 /usr/lib/
b744d000-b764d000 r--p 00000000 08:06 662632 /usr/lib/
b764d000-b764e000 rw-p 00000000 00:00 0
b764e000-b7651000 r-xp 00000000 08:06 1967024 /lib/i386-
b7651000-b7652000 r--p 00002000 08:06 1967024 /lib/i386-
b7652000-b7653000 rw-p 00003000 08:06 1967024 /lib/i386-
b7653000-b7657000 r-xp 00000000 08:06 1967092 /lib/i386-
b7657000-b7658000 r--p 00003000 08:06 1967092 /lib/i386-
b7658000-b7659000 rw-p 00004000 08:06 1967092 /lib/i386-
b7659000-b765a000 rw-p 00000000 00:00 0
b765a000-b7680000 r-xp 00000000 08:06 1967029 /lib/i386-
b7680000-b7681000 ---p 00026000 08:06 1967029 /lib/i386-
b7681000-b7683000 r--p 00026000 08:06 1967029 /lib/i386-
b7683000-b7684000 rw-p 00028000 08:06 1967029 /lib/i386-
b7684000-b77fa000 r-xp 00000000 08:06 1967013 /lib/i386-
b77fa000-b77fc000 r--p 00176000 08:06 1967013 /lib/i386-
b77fc000-b77fd000 rw-p 00178000 08:06 1967013 /lib/i386-
b77fd000-b7800000 rw-p 00000000 00:00 0
b7800000-b7817000 r-xp 00000000 08:06 1967073 /lib/i386-
b7817000-b7818000 r--p 00016000 08:06 1967073 /lib/i386-
b7818000-b7819000 rw-p 00017000 08:06 1967073 /lib/i386-
b7819000-b781b000 rw-p 00000000 00:00 0
b781b000-b784e000 r-xp 00000000 08:06 672170 /usr/lib/
b784e000-b784f000 r--p 00032000 08:06 672170 /usr/lib/
b784f000-b7850000 rw-p 00033000 08:06 672170 /usr/lib/
b7850000-b785a000 r-xp 00000000 08:06 672024 /usr/lib/
b785a000-b785b000 r--p 00009000 08:06 672024 /usr/lib/
b785b000-b785c000 rw-p 0000a000 08:06 672024 /usr/lib/
b785c000-b785d000 rw-p 00000000 00:00 0
b785d000-b789a000 r-xp 00000000 08:06 1967070 /lib/i386-
b789a000-b789b000 r--p 0003c000 08:06 1967070 /lib/i386-
b789b000-b789c000 rw-p 0003d000 08:06 1967070 /lib/i386-
b789c000-b78ae000 r-xp 00000000 08:06 668473 /usr/lib/
b78ae000-b78af000 r--p 00011000 08:06 668473 /usr/lib/
b78af000-b78b1000 rw-p 00012000 08:06 668473 /usr/lib/
b78b1000-b78b9000 r-xp 00000000 08:06 1967050 /lib/i386-
b78b9000-b78ba000 r--p 00007000 08:06 1967050 /lib/i386-
b78ba000-b78bb000 rw-p 00008000 08:06 1967050 /lib/i386-
b78bb000-b78c9000 rw-p 00000000 00:00 0
b78c9000-b78ca000 r-xp 00000000 00:00 0 [vdso]
b78ca000-b78e8000 r-xp 00000000 08:06 1967000 /lib/i386-
b78e8000-b78e9000 r--p 0001d000 08:06 1967000 /lib/i386-
b78e9000-b78ea000 rw-p 0001e000 08:06 1967000 /lib/i386-
bfa4b000-bfa6c000 rw-p 00000000 00:00 0 [stack]
Aborted
ProblemType: Bug
DistroRelease: Ubuntu 11.10
Package: ganglia-monitor 3.1.7-2 [modified: usr/share/
ProcVersionSign
Uname: Linux 3.0.0-14-
ApportVersion: 1.23-0ubuntu4
Architecture: i386
Date: Sun Jan 1 16:45:03 2012
InstallationMedia: Ubuntu 11.10 "Oneiric Ocelot" - Release i386 (20111012)
ProcEnviron:
PATH=(custom, no user)
LANG=en_US.UTF-8
SHELL=/bin/bash
SourcePackage: ganglia
UpgradeStatus: No upgrade log present (probably fresh install)
Related branches
Dan Shick (ubuntu-eleven) wrote : | #1 |
Chris Adamson (addo81) wrote : | #3 |
A fix for this bug was committed to Fedora
https:/
libmetrics/
line 1231:
char mount[128], device[128], type[32], mode[128];
change this to
char mount[2048], device[2048], type[32], mode[2048];
in order to accommodate the longer strings in /proc/mounts for NFS and automounts
teonghan (teonghan) wrote : | #4 |
Hi all,
If I did an "apt-get upgrade" or specifically, apt-get install libc6, ganglia-monitor (gmond) can be started without problem, even if there is a NFS partition. Can anyone verify this?
Thanks.
teonghan (teonghan) wrote : | #5 |
Hi all,
Sorry, should have stated that the version of libc6 I am using is 2.13-20ubuntu5.1.
Thanks.
dab (dabosuke) wrote : | #6 |
I got the same problem with Ubuntu 12.04.
Teppo Sulavuori (sulate) wrote : | #7 |
This has been fixed in upstream since version 3.2.0
https:/
(I'm a bit sad that Precise being an LTS version did not get a newer version of ganglia-monitor since current version 3.1.7 is already over two years old.)
description: | updated |
Mark Asbach (mark-asbach) wrote : | #8 |
A quick fix for users not wanting to wait for an update or build gmond manually is to change the gmond.conf to not load the disks module.
Comment out the respective sections of /etc/ganglia/
module {
name = "cpu_module"
path = "/usr/lib/
}
/*
module {
name = "disk_module"
path = "/usr/lib/
}
*/
module {
name = "load_module"
path = "/usr/lib/
}
[...]
/*
collection_group {
collect_every = 1800
time_threshold = 3600
metric {
name = "disk_total"
value_threshold = 1.0
title = "Total Disk Space"
}
}
collection_group {
collect_every = 40
time_threshold = 180
metric {
name = "disk_free"
value_threshold = 1.0
title = "Disk Space Available"
}
metric {
name = "part_max_used"
value_threshold = 1.0
title = "Maximum Disk Space Used"
}
}
*/
This should be fixed ASAP, Ganglia is an important component in many server environments which is the target of LTS releases.
Mike Johnson (mikejohnson) wrote : | #10 |
I'm having this problem in 12.04 as well.
description: | updated |
Exilist (sandro-grundmann) wrote : | #11 |
i can confirm this, ganglia/gmond is not starting on 12.04 and nfs-mounts
Trevor Robinson (scurrilous) wrote : | #12 |
I hit this issue as well. My workaround was to install the ganglia packages from quantal, which picked up a substantially newer version that includes the upstream fix.
IMHO, the quantal packages should be backported to precise, given its status as an LTS release. Ganglia is widely used to monitor server apps like Hadoop, and those environments often use NFS.
Reed Loden (reed) wrote : | #13 |
quantal has 3.3.8, so it's not affected by this issue.
Changed in ganglia (Ubuntu Quantal): | |
status: | Confirmed → Fix Released |
Reed Loden (reed) wrote : | #14 |
My attempt at a debdiff for this fix, but I don't think I did it right. :(
Jamie Strandboge (jdstrand) wrote : | #15 |
Thanks for the debdiff! There are a few things that need updating:
* this should target precise-proposed, not precise
* the version should be 3.1.7-2ubuntu1.1 as per https:/
* there is an undocumented change to debian/
Since the first two were simple and the third less so, I went ahead an made these changes and uploaded to precise-proposed.
Jamie Strandboge (jdstrand) wrote : | #16 |
Unsubscribing ubuntu-sponsors.
Launchpad Janitor (janitor) wrote : | #17 |
Status changed to 'Confirmed' because the bug affects multiple users.
Changed in ganglia (Ubuntu Precise): | |
status: | New → Confirmed |
Hello Dan, or anyone else affected,
Accepted ganglia into precise-proposed. The package will build now and be available at http://
Please help us by testing this new package. See https:/
If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-
Further information regarding the verification process can be found at https:/
Changed in ganglia (Ubuntu Precise): | |
assignee: | nobody → Reed Loden (reed) |
status: | Confirmed → In Progress |
status: | In Progress → Fix Committed |
tags: | added: verification-needed |
Robert Dyer (psybers) wrote : | #19 |
Updated to 3.1.7-2ubuntu1.1 from precise-proposed. Re-enabled the disk module. Ganglia now starts up instead of segfaulting and appears to be working normal.
tags: |
added: verification-done removed: verification-needed |
Joseph Holsten (i4-joseph) wrote : | #20 |
This is preventing my company from upgrading from 10.04 to 12.04. Thanks for working on this.
Joseph Holsten (i4-joseph) wrote : | #21 |
Confirming that the package in -proposed also resolved the issue for us.
The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.
Launchpad Janitor (janitor) wrote : | #23 |
This bug was fixed in the package ganglia - 3.1.7-2ubuntu1.1
---------------
ganglia (3.1.7-2ubuntu1.1) precise-proposed; urgency=low
[ Reed Loden ]
* Fix buffer overflow in moddisk.so. (LP: #910678)
* debian/
[ Jamie Strandboge ]
* debian/
and config.sub
-- Reed Loden <email address hidden> Mon, 15 Oct 2012 11:22:10 -0700
Changed in ganglia (Ubuntu Precise): | |
status: | Fix Committed → Fix Released |
Status changed to 'Confirmed' because the bug affects multiple users.