Ubuntu
net-snmp package

snmpd reports wrong value for hrStorageSize on large filesystem

Bug #865268 reported by Brian Candler
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
net-snmp (Ubuntu)
Fix Released
Low
Unassigned

Bug Description

Look at hrStorageSize.34 in the below table. Notice that:
(1) it's smaller than hrStorageUsed.34, so it can't possibly be right
(2) it's the same as the previous row, hrStorageSize.33

$ snmpwalk -c public -v2c <hostname> hrStorageTable
HOST-RESOURCES-MIB::hrStorageIndex.1 = INTEGER: 1
HOST-RESOURCES-MIB::hrStorageIndex.3 = INTEGER: 3
HOST-RESOURCES-MIB::hrStorageIndex.6 = INTEGER: 6
HOST-RESOURCES-MIB::hrStorageIndex.7 = INTEGER: 7
HOST-RESOURCES-MIB::hrStorageIndex.10 = INTEGER: 10
HOST-RESOURCES-MIB::hrStorageIndex.31 = INTEGER: 31
HOST-RESOURCES-MIB::hrStorageIndex.32 = INTEGER: 32
HOST-RESOURCES-MIB::hrStorageIndex.33 = INTEGER: 33
HOST-RESOURCES-MIB::hrStorageIndex.34 = INTEGER: 34
HOST-RESOURCES-MIB::hrStorageIndex.35 = INTEGER: 35
HOST-RESOURCES-MIB::hrStorageIndex.36 = INTEGER: 36
HOST-RESOURCES-MIB::hrStorageIndex.37 = INTEGER: 37
HOST-RESOURCES-MIB::hrStorageType.1 = OID: HOST-RESOURCES-TYPES::hrStorageRam
HOST-RESOURCES-MIB::hrStorageType.3 = OID: HOST-RESOURCES-TYPES::hrStorageVirtualMemory
HOST-RESOURCES-MIB::hrStorageType.6 = OID: HOST-RESOURCES-TYPES::hrStorageOther
HOST-RESOURCES-MIB::hrStorageType.7 = OID: HOST-RESOURCES-TYPES::hrStorageOther
HOST-RESOURCES-MIB::hrStorageType.10 = OID: HOST-RESOURCES-TYPES::hrStorageVirtualMemory
HOST-RESOURCES-MIB::hrStorageType.31 = OID: HOST-RESOURCES-TYPES::hrStorageFixedDisk
HOST-RESOURCES-MIB::hrStorageType.32 = OID: HOST-RESOURCES-TYPES::hrStorageFixedDisk
HOST-RESOURCES-MIB::hrStorageType.33 = OID: HOST-RESOURCES-TYPES::hrStorageFixedDisk
HOST-RESOURCES-MIB::hrStorageType.34 = OID: HOST-RESOURCES-TYPES::hrStorageFixedDisk
HOST-RESOURCES-MIB::hrStorageType.35 = OID: HOST-RESOURCES-TYPES::hrStorageFixedDisk
HOST-RESOURCES-MIB::hrStorageType.36 = OID: HOST-RESOURCES-TYPES::hrStorageFixedDisk
HOST-RESOURCES-MIB::hrStorageType.37 = OID: HOST-RESOURCES-TYPES::hrStorageNetworkDisk
HOST-RESOURCES-MIB::hrStorageDescr.1 = STRING: Physical memory
HOST-RESOURCES-MIB::hrStorageDescr.3 = STRING: Virtual memory
HOST-RESOURCES-MIB::hrStorageDescr.6 = STRING: Memory buffers
HOST-RESOURCES-MIB::hrStorageDescr.7 = STRING: Cached memory
HOST-RESOURCES-MIB::hrStorageDescr.10 = STRING: Swap space
HOST-RESOURCES-MIB::hrStorageDescr.31 = STRING: /
HOST-RESOURCES-MIB::hrStorageDescr.32 = STRING: /sys/fs/fuse/connections
HOST-RESOURCES-MIB::hrStorageDescr.33 = STRING: /dev
HOST-RESOURCES-MIB::hrStorageDescr.34 = STRING: /media/data
HOST-RESOURCES-MIB::hrStorageDescr.35 = STRING: /media/data2
HOST-RESOURCES-MIB::hrStorageDescr.36 = STRING: /home/tomi/.gvfs
HOST-RESOURCES-MIB::hrStorageDescr.37 = STRING: /media/monster1
HOST-RESOURCES-MIB::hrStorageAllocationUnits.1 = INTEGER: 1024 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.3 = INTEGER: 1024 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.6 = INTEGER: 1024 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.7 = INTEGER: 1024 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.10 = INTEGER: 1024 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.31 = INTEGER: 4096 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.32 = INTEGER: 4096 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.33 = INTEGER: 4096 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.34 = INTEGER: 4096 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.35 = INTEGER: 4096 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.36 = INTEGER: 0 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.37 = INTEGER: 8192 Bytes
HOST-RESOURCES-MIB::hrStorageSize.1 = INTEGER: 24732776
HOST-RESOURCES-MIB::hrStorageSize.3 = INTEGER: 49887332
HOST-RESOURCES-MIB::hrStorageSize.6 = INTEGER: 24732776
HOST-RESOURCES-MIB::hrStorageSize.7 = INTEGER: 18405520
HOST-RESOURCES-MIB::hrStorageSize.10 = INTEGER: 25154556
HOST-RESOURCES-MIB::hrStorageSize.31 = INTEGER: 116744604
HOST-RESOURCES-MIB::hrStorageSize.32 = INTEGER: 0
HOST-RESOURCES-MIB::hrStorageSize.33 = INTEGER: 3089665
HOST-RESOURCES-MIB::hrStorageSize.34 = INTEGER: 3089665
HOST-RESOURCES-MIB::hrStorageSize.35 = INTEGER: 388630628
HOST-RESOURCES-MIB::hrStorageSize.36 = INTEGER: 0
HOST-RESOURCES-MIB::hrStorageSize.37 = INTEGER: 1763671548
HOST-RESOURCES-MIB::hrStorageUsed.1 = INTEGER: 21206848
HOST-RESOURCES-MIB::hrStorageUsed.3 = INTEGER: 21266872
HOST-RESOURCES-MIB::hrStorageUsed.6 = INTEGER: 99780
HOST-RESOURCES-MIB::hrStorageUsed.7 = INTEGER: 18405520
HOST-RESOURCES-MIB::hrStorageUsed.10 = INTEGER: 60024
HOST-RESOURCES-MIB::hrStorageUsed.31 = INTEGER: 6489199
HOST-RESOURCES-MIB::hrStorageUsed.32 = INTEGER: 0
HOST-RESOURCES-MIB::hrStorageUsed.33 = INTEGER: 162
HOST-RESOURCES-MIB::hrStorageUsed.34 = INTEGER: 53617360
HOST-RESOURCES-MIB::hrStorageUsed.35 = INTEGER: 290113170
HOST-RESOURCES-MIB::hrStorageUsed.36 = INTEGER: 0
HOST-RESOURCES-MIB::hrStorageUsed.37 = INTEGER: 1163419544
brian@monitoring:~$ snmpwalk -c public -v2c recording1 hrStorageTable
HOST-RESOURCES-MIB::hrStorageIndex.1 = INTEGER: 1
HOST-RESOURCES-MIB::hrStorageIndex.3 = INTEGER: 3
HOST-RESOURCES-MIB::hrStorageIndex.6 = INTEGER: 6
HOST-RESOURCES-MIB::hrStorageIndex.7 = INTEGER: 7
HOST-RESOURCES-MIB::hrStorageIndex.10 = INTEGER: 10
HOST-RESOURCES-MIB::hrStorageIndex.31 = INTEGER: 31
HOST-RESOURCES-MIB::hrStorageIndex.32 = INTEGER: 32
HOST-RESOURCES-MIB::hrStorageIndex.33 = INTEGER: 33
HOST-RESOURCES-MIB::hrStorageIndex.34 = INTEGER: 34
HOST-RESOURCES-MIB::hrStorageIndex.35 = INTEGER: 35
HOST-RESOURCES-MIB::hrStorageIndex.36 = INTEGER: 36
HOST-RESOURCES-MIB::hrStorageIndex.37 = INTEGER: 37
HOST-RESOURCES-MIB::hrStorageType.1 = OID: HOST-RESOURCES-TYPES::hrStorageRam
HOST-RESOURCES-MIB::hrStorageType.3 = OID: HOST-RESOURCES-TYPES::hrStorageVirtualMemory
HOST-RESOURCES-MIB::hrStorageType.6 = OID: HOST-RESOURCES-TYPES::hrStorageOther
HOST-RESOURCES-MIB::hrStorageType.7 = OID: HOST-RESOURCES-TYPES::hrStorageOther
HOST-RESOURCES-MIB::hrStorageType.10 = OID: HOST-RESOURCES-TYPES::hrStorageVirtualMemory
HOST-RESOURCES-MIB::hrStorageType.31 = OID: HOST-RESOURCES-TYPES::hrStorageFixedDisk
HOST-RESOURCES-MIB::hrStorageType.32 = OID: HOST-RESOURCES-TYPES::hrStorageFixedDisk
HOST-RESOURCES-MIB::hrStorageType.33 = OID: HOST-RESOURCES-TYPES::hrStorageFixedDisk
HOST-RESOURCES-MIB::hrStorageType.34 = OID: HOST-RESOURCES-TYPES::hrStorageFixedDisk
HOST-RESOURCES-MIB::hrStorageType.35 = OID: HOST-RESOURCES-TYPES::hrStorageFixedDisk
HOST-RESOURCES-MIB::hrStorageType.36 = OID: HOST-RESOURCES-TYPES::hrStorageFixedDisk
HOST-RESOURCES-MIB::hrStorageType.37 = OID: HOST-RESOURCES-TYPES::hrStorageNetworkDisk
HOST-RESOURCES-MIB::hrStorageDescr.1 = STRING: Physical memory
HOST-RESOURCES-MIB::hrStorageDescr.3 = STRING: Virtual memory
HOST-RESOURCES-MIB::hrStorageDescr.6 = STRING: Memory buffers
HOST-RESOURCES-MIB::hrStorageDescr.7 = STRING: Cached memory
HOST-RESOURCES-MIB::hrStorageDescr.10 = STRING: Swap space
HOST-RESOURCES-MIB::hrStorageDescr.31 = STRING: /
HOST-RESOURCES-MIB::hrStorageDescr.32 = STRING: /sys/fs/fuse/connections
HOST-RESOURCES-MIB::hrStorageDescr.33 = STRING: /dev
HOST-RESOURCES-MIB::hrStorageDescr.34 = STRING: /media/data
HOST-RESOURCES-MIB::hrStorageDescr.35 = STRING: /media/data2
HOST-RESOURCES-MIB::hrStorageDescr.36 = STRING: /home/tomi/.gvfs
HOST-RESOURCES-MIB::hrStorageDescr.37 = STRING: /media/monster1
HOST-RESOURCES-MIB::hrStorageAllocationUnits.1 = INTEGER: 1024 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.3 = INTEGER: 1024 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.6 = INTEGER: 1024 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.7 = INTEGER: 1024 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.10 = INTEGER: 1024 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.31 = INTEGER: 4096 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.32 = INTEGER: 4096 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.33 = INTEGER: 4096 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.34 = INTEGER: 4096 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.35 = INTEGER: 4096 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.36 = INTEGER: 0 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.37 = INTEGER: 8192 Bytes
HOST-RESOURCES-MIB::hrStorageSize.1 = INTEGER: 24732776
HOST-RESOURCES-MIB::hrStorageSize.3 = INTEGER: 49887332
HOST-RESOURCES-MIB::hrStorageSize.6 = INTEGER: 24732776
HOST-RESOURCES-MIB::hrStorageSize.7 = INTEGER: 18670248
HOST-RESOURCES-MIB::hrStorageSize.10 = INTEGER: 25154556
HOST-RESOURCES-MIB::hrStorageSize.31 = INTEGER: 116744604
HOST-RESOURCES-MIB::hrStorageSize.32 = INTEGER: 0
HOST-RESOURCES-MIB::hrStorageSize.33 = INTEGER: 3089665
HOST-RESOURCES-MIB::hrStorageSize.34 = INTEGER: 3089665 <<<<< NOTE
HOST-RESOURCES-MIB::hrStorageSize.35 = INTEGER: 388630628
HOST-RESOURCES-MIB::hrStorageSize.36 = INTEGER: 0
HOST-RESOURCES-MIB::hrStorageSize.37 = INTEGER: 1763671548
HOST-RESOURCES-MIB::hrStorageUsed.1 = INTEGER: 21959984
HOST-RESOURCES-MIB::hrStorageUsed.3 = INTEGER: 22020008
HOST-RESOURCES-MIB::hrStorageUsed.6 = INTEGER: 99856
HOST-RESOURCES-MIB::hrStorageUsed.7 = INTEGER: 18670248
HOST-RESOURCES-MIB::hrStorageUsed.10 = INTEGER: 60024
HOST-RESOURCES-MIB::hrStorageUsed.31 = INTEGER: 6534349
HOST-RESOURCES-MIB::hrStorageUsed.32 = INTEGER: 0
HOST-RESOURCES-MIB::hrStorageUsed.33 = INTEGER: 162
HOST-RESOURCES-MIB::hrStorageUsed.34 = INTEGER: 53617360 <<<<< NOTE
HOST-RESOURCES-MIB::hrStorageUsed.35 = INTEGER: 290116334
HOST-RESOURCES-MIB::hrStorageUsed.36 = INTEGER: 0
HOST-RESOURCES-MIB::hrStorageUsed.37 = INTEGER: 1163421564

For comparison, here's what df -k shows on the host:

root@r1:/var/log# df -k
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1 466978416 24961836 418295416 6% /
none 12358660 648 12358012 1% /dev
none 12366388 248 12366140 1% /dev/shm
none 12366388 404 12365984 1% /var/run
none 12366388 0 12366388 0% /var/lock
/dev/sdb1 15747966392 214469440 14733545784 2% /media/data
/dev/sdb2 1554522512 1160578276 314979008 79% /media/data2
192.168.5.119:/109TB-NFS_shared
                     117188587488 9307643872 107880943616 8% /media/monster1

The partition in question is /media/data.

The value for KB used matches with hrStorageUsed (214469440 / 4 = 53617360), but the value for hrStorageSize should be 14733545784 / 4 = 3683386446

Since this value is between 2^31 and 2^32, it's possible that it is being miscalculated as a negative value. But even then, hrStorageSize should not just show an arbitrary value which happens to be the hrStorageSize of the previous partition.

ProblemType: Bug
DistroRelease: Ubuntu 11.04
Package: snmpd 5.4.3~dfsg-2ubuntu1
ProcVersionSignature: Ubuntu 2.6.38-11.48-generic 2.6.38.8
Uname: Linux 2.6.38-11-generic x86_64
Architecture: amd64
Date: Mon Oct 3 12:45:06 2011
InstallationMedia: Ubuntu 11.04 "Natty Narwhal" - Release amd64 (20110427.1)
ProcEnviron:
 LANGUAGE=en_GB:en
 PATH=(custom, no user)
 LANG=en_GB.UTF-8
 SHELL=/bin/bash
SNMPVersion:
 NET-SNMP version: 5.4.3
 Web: http://www.net-snmp.org/
 Email: <email address hidden>
SourcePackage: net-snmp
SyslogSnmptrapd:

UpgradeStatus: No upgrade log present (probably fresh install)
mtime.conffile..etc.snmp.snmpd.conf: 2011-09-26T10:37:39.583350

Revision history for this message
Brian Candler (b-candler) wrote :
Revision history for this message
Brian Candler (b-candler) wrote :

Looks like it could be related to this: https://bugzilla.redhat.com/show_bug.cgi?id=654384

Apparently a newer snmpd would clamp this value to 2^31, and an even newer snmpd would use a fake larger block size so as to be able to report on large filesystems:
http://net-snmp.svn.sourceforge.net/viewvc/net-snmp?view=revision&revision=19941

Workaround is to use the UCD-SNMP-MIB::dskTable instead, which can report dskAvail and dskPercent successfully for this large filesystem.

summary: - snmpd reports wrong value for hrStorageSize
+ snmpd reports wrong value for hrStorageSize on large filesystem
Robie Basak (racb)
Changed in net-snmp (Ubuntu):
importance: Undecided → Low
Revision history for this message
Guangxian Zou (邹光先) (zouguangxian) wrote :

Version: net-snmp 5.7.1

in net-snmp-5.7.1/agent/mibgroup/hardware/fsys/hw_fsys.c:337

should be
    f->avail_32 = f->avail >> factor;
    f->used_32 = f->used >> factor;

instead of
    f->avail_32 = f->avail << factor;
    f->used_32 = f->used << factor;

Revision history for this message
Chuck Short (zulcss) wrote :

This should be fixed in pangolin. Please re-check and if still not fixed please re-open.

Changed in net-snmp (Ubuntu):
status: New → Fix Released
Revision history for this message
Bruc (d769487) wrote :

I can confirm this is not fixed. I'm still having the issue on 12.04.

Revision history for this message
Brian Candler (b-candler) wrote :

Also confirmed, this remains broken in 12.04

$ snmpwalk -v2c -c public storage3 hrStorageTable | grep \\.34HOST-RESOURCES-MIB::hrStorageIndex.34 = INTEGER: 34
HOST-RESOURCES-MIB::hrStorageType.34 = OID: HOST-RESOURCES-TYPES::hrStorageFixedDisk
HOST-RESOURCES-MIB::hrStorageDescr.34 = STRING: /disk/scratch
HOST-RESOURCES-MIB::hrStorageAllocationUnits.34 = INTEGER: 4096 Bytes
HOST-RESOURCES-MIB::hrStorageSize.34 = INTEGER: 393941158
HOST-RESOURCES-MIB::hrStorageUsed.34 = INTEGER: 3959967469

Notice how hrStorageSize is much less than hrStorageUsed.

Here is the real output from df:

$ df -h /disk/scratch
Filesystem Size Used Avail Use% Mounted on
/dev/md127 66T 63T 2.8T 96% /disk/scratch
$ df -k /disk/scratch
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/md127 70295241368 67379477428 2915763940 96% /disk/scratch

But the values reported in hrStorageTable are:

size = 393941158 * 4 = 1575764632 KiB = 1.47 TiB
used = 3959967469 * 4 = 15839869876 KiB = 14.75 TiB

Revision history for this message
Brian Candler (b-candler) wrote :

Looking at the source of net-snmp-5.4.3~dfsg, it appears to make no attempt to scale the block size to avoid 2^32 wrapping - see agent/mibgroup/host/hr_storage.c, var_hrstore()

I have made a simple patch which fixes just this particular problem, so should be suitable for backporting to LTS.

Once this patch has been applied, sane values are returned:

$ snmpwalk -v2c -c public storage3 hrStorageTable | grep '\.34 'HOST-RESOURCES-MIB::hrStorageIndex.34 = INTEGER: 34
HOST-RESOURCES-MIB::hrStorageType.34 = OID: HOST-RESOURCES-TYPES::hrStorageFixedDisk
HOST-RESOURCES-MIB::hrStorageDescr.34 = STRING: /disk/scratch
HOST-RESOURCES-MIB::hrStorageAllocationUnits.34 = INTEGER: 65536 Bytes
HOST-RESOURCES-MIB::hrStorageSize.34 = INTEGER: 1098363146
HOST-RESOURCES-MIB::hrStorageUsed.34 = INTEGER: 1052804334

(Note: the agent modules are actually in libsnmp15, so after dpkg-buildpackage remember to install libsnmp15, libsnmp-base and snmpd)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.