hrStorage Not Available In 5.7.3+dfsg-1ubuntu4.3

[David Ford]

In our environment, we use monitoring checks that utilise SNMP for hard disk space monitoring. This relies upon HOST-RESOURCES-MIB returning all the data and all was working fine in previous releases.

After upgrading to version 5.7.3+dfsg-1ubuntu4.3, hrStorage is no longer available and therefore prevents monitoring of hard disk space via SNMP

For example:

With version 5.7.3+dfsg-1ubuntu4.2 running snmpwalk -v2c -c community hostname hrStorage

HOST-RESOURCES-MIB::hrMemorySize.0 = INTEGER: 8174784 KBytes
HOST-RESOURCES-MIB::hrStorageIndex.1 = INTEGER: 1
HOST-RESOURCES-MIB::hrStorageIndex.3 = INTEGER: 3
HOST-RESOURCES-MIB::hrStorageIndex.6 = INTEGER: 6
HOST-RESOURCES-MIB::hrStorageIndex.7 = INTEGER: 7
HOST-RESOURCES-MIB::hrStorageIndex.8 = INTEGER: 8
HOST-RESOURCES-MIB::hrStorageIndex.10 = INTEGER: 10
HOST-RESOURCES-MIB::hrStorageIndex.31 = INTEGER: 31
HOST-RESOURCES-MIB::hrStorageIndex.32 = INTEGER: 32
HOST-RESOURCES-MIB::hrStorageIndex.37 = INTEGER: 37
HOST-RESOURCES-MIB::hrStorageIndex.39 = INTEGER: 39
HOST-RESOURCES-MIB::hrStorageIndex.40 = INTEGER: 40
HOST-RESOURCES-MIB::hrStorageIndex.41 = INTEGER: 41
HOST-RESOURCES-MIB::hrStorageType.1 = OID: HOST-RESOURCES-TYPES::hrStorageRam
HOST-RESOURCES-MIB::hrStorageType.3 = OID: HOST-RESOURCES-TYPES::hrStorageVirtualMemory
HOST-RESOURCES-MIB::hrStorageType.6 = OID: HOST-RESOURCES-TYPES::hrStorageOther
HOST-RESOURCES-MIB::hrStorageType.7 = OID: HOST-RESOURCES-TYPES::hrStorageOther
HOST-RESOURCES-MIB::hrStorageType.8 = OID: HOST-RESOURCES-TYPES::hrStorageOther
HOST-RESOURCES-MIB::hrStorageType.10 = OID: HOST-RESOURCES-TYPES::hrStorageVirtualMemory
HOST-RESOURCES-MIB::hrStorageType.31 = OID: HOST-RESOURCES-TYPES::hrStorageFixedDisk
HOST-RESOURCES-MIB::hrStorageType.32 = OID: HOST-RESOURCES-TYPES::hrStorageFixedDisk
HOST-RESOURCES-MIB::hrStorageType.37 = OID: HOST-RESOURCES-TYPES::hrStorageFixedDisk
HOST-RESOURCES-MIB::hrStorageType.39 = OID: HOST-RESOURCES-TYPES::hrStorageFixedDisk
HOST-RESOURCES-MIB::hrStorageType.40 = OID: HOST-RESOURCES-TYPES::hrStorageFixedDisk
HOST-RESOURCES-MIB::hrStorageType.41 = OID: HOST-RESOURCES-TYPES::hrStorageFixedDisk
HOST-RESOURCES-MIB::hrStorageDescr.1 = STRING: Physical memory
HOST-RESOURCES-MIB::hrStorageDescr.3 = STRING: Virtual memory
HOST-RESOURCES-MIB::hrStorageDescr.6 = STRING: Memory buffers
HOST-RESOURCES-MIB::hrStorageDescr.7 = STRING: Cached memory
HOST-RESOURCES-MIB::hrStorageDescr.8 = STRING: Shared memory
HOST-RESOURCES-MIB::hrStorageDescr.10 = STRING: Swap space
HOST-RESOURCES-MIB::hrStorageDescr.31 = STRING: /
HOST-RESOURCES-MIB::hrStorageDescr.32 = STRING: /var
HOST-RESOURCES-MIB::hrStorageDescr.37 = STRING: /run
HOST-RESOURCES-MIB::hrStorageDescr.39 = STRING: /dev/shm
HOST-RESOURCES-MIB::hrStorageDescr.40 = STRING: /run/lock
HOST-RESOURCES-MIB::hrStorageDescr.41 = STRING: /sys/fs/cgroup
HOST-RESOURCES-MIB::hrStorageAllocationUnits.1 = INTEGER: 1024 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.3 = INTEGER: 1024 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.6 = INTEGER: 1024 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.7 = INTEGER: 1024 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.8 = INTEGER: 1024 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.10 = INTEGER: 1024 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.31 = INTEGER: 4096 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.32 = INTEGER: 4096 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.37 = INTEGER: 4096 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.39 = INTEGER: 4096 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.40 = INTEGER: 4096 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.41 = INTEGER: 4096 Bytes
HOST-RESOURCES-MIB::hrStorageSize.1 = INTEGER: 8174784
HOST-RESOURCES-MIB::hrStorageSize.3 = INTEGER: 8174784
HOST-RESOURCES-MIB::hrStorageSize.6 = INTEGER: 8174784
HOST-RESOURCES-MIB::hrStorageSize.7 = INTEGER: 560348
HOST-RESOURCES-MIB::hrStorageSize.8 = INTEGER: 80764
HOST-RESOURCES-MIB::hrStorageSize.10 = INTEGER: 0
HOST-RESOURCES-MIB::hrStorageSize.31 = INTEGER: 2177843
HOST-RESOURCES-MIB::hrStorageSize.32 = INTEGER: 2142582
HOST-RESOURCES-MIB::hrStorageSize.37 = INTEGER: 204370
HOST-RESOURCES-MIB::hrStorageSize.39 = INTEGER: 1021848
HOST-RESOURCES-MIB::hrStorageSize.40 = INTEGER: 1280
HOST-RESOURCES-MIB::hrStorageSize.41 = INTEGER: 1021848
HOST-RESOURCES-MIB::hrStorageUsed.1 = INTEGER: 1372788
HOST-RESOURCES-MIB::hrStorageUsed.3 = INTEGER: 1372788
HOST-RESOURCES-MIB::hrStorageUsed.6 = INTEGER: 34596
HOST-RESOURCES-MIB::hrStorageUsed.7 = INTEGER: 560348
HOST-RESOURCES-MIB::hrStorageUsed.8 = INTEGER: 80764
HOST-RESOURCES-MIB::hrStorageUsed.10 = INTEGER: 0
HOST-RESOURCES-MIB::hrStorageUsed.31 = INTEGER: 738146
HOST-RESOURCES-MIB::hrStorageUsed.32 = INTEGER: 640542
HOST-RESOURCES-MIB::hrStorageUsed.37 = INTEGER: 20096
HOST-RESOURCES-MIB::hrStorageUsed.39 = INTEGER: 0
HOST-RESOURCES-MIB::hrStorageUsed.40 = INTEGER: 0
HOST-RESOURCES-MIB::hrStorageUsed.41 = INTEGER: 0

And with the latest version we get

HOST-RESOURCES-MIB::hrMemorySize.0 = INTEGER: 8174784 KBytes

This has been confirmed on multiple systems in our environment and rollback of snnpd, libsnnp30 and snmp packages to the previous version restores functionality

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[David Ford (atheros1200)]

Adding

view systemonly included .1.3.6.1.2.1.25.2

To /etc/snmp/snmp.conf allows for some storage devices to be returned

HOST-RESOURCES-MIB::hrStorageDescr.1 = STRING: Physical memory
HOST-RESOURCES-MIB::hrStorageDescr.3 = STRING: Virtual memory
HOST-RESOURCES-MIB::hrStorageDescr.6 = STRING: Memory buffers
HOST-RESOURCES-MIB::hrStorageDescr.7 = STRING: Cached memory
HOST-RESOURCES-MIB::hrStorageDescr.8 = STRING: Shared memory
HOST-RESOURCES-MIB::hrStorageDescr.10 = STRING: Swap space

But physical drives are not returned. Looking at the index numbers, physical drives appear to start from index 31 (which would be 32 if starting at 0). Don't know if that is significant

HOST-RESOURCES-MIB::hrStorageDescr.31 = STRING: /
HOST-RESOURCES-MIB::hrStorageDescr.32 = STRING: /var
HOST-RESOURCES-MIB::hrStorageDescr.37 = STRING: /run
HOST-RESOURCES-MIB::hrStorageDescr.39 = STRING: /dev/shm
HOST-RESOURCES-MIB::hrStorageDescr.40 = STRING: /run/lock
HOST-RESOURCES-MIB::hrStorageDescr.41 = STRING: /sys/fs/cgroup
HOST-RESOURCES-MIB::hrStorageDescr.59 = STRING: /run/user/2128

[David Ford (atheros1200)]

Ignore the comment above, looks like something in the AutoFS patch is detecting physical file systems as autofs

We are running Ubuntu 16.04 and does not work on both 4.4 and 4.15 Kernel

[Joseph Gooch (whitemystic)]

Some behavior seen here, on Xenial but also on Bionic @ 5.7.3+dfsg-1.8ubuntu3.2

[Joseph Gooch (whitemystic)]

This part looks questionable:
@@ -839,6 +839,27 @@ Check_HR_FileSys_NFS (void)
     return 0; /* no NFS file system */
 }

+/* This function checks whether current file system is an AutoFs
+ * HRFS_entry must be valid prior to calling this function
+ * return 1 if AutoFs, 0 otherwise
+ */
+int
+Check_HR_FileSys_AutoFs(void)
+{
+#if HAVE_GETFSSTAT
+ if (HRFS_entry->HRFS_type != NULL &&
+#if defined(MNTTYPE_AUTOFS)
+ !strcmp(HRFS_entry->HRFS_type, MNTTYPE_AUTOFS)
+#else
+ !strcmp(HRFS_entry->HRFS_type, "autofs")
+#endif
+ )
+#endif /* HAVE_GETFSSTAT */
+ return 1; /* AUTOFS */
+
+ return 0; /* no AUTOFS */
+}
+

if HAVE_GETFSSTAT doesn't pass, we end up with:
return 1;
return 0;

In effect, ALL filesystems are autofs.

[Joseph Gooch (whitemystic)]

While that's wrong it must be more than that because changing it didn't fix it.

[Peter Newman (peternewman)]

I don't know how you changed it Joseph, but shouldn't it be (entirely untested):
/* This function checks whether current file system is an AutoFs
 * HRFS_entry must be valid prior to calling this function
 * return 1 if AutoFs, 0 otherwise
 */
int
Check_HR_FileSys_AutoFs(void)
{
#if HAVE_GETFSSTAT
 if (HRFS_entry->HRFS_type != NULL &&
#if defined(MNTTYPE_AUTOFS)
 !strcmp(HRFS_entry->HRFS_type, MNTTYPE_AUTOFS)
#else
 !strcmp(HRFS_entry->HRFS_type, "autofs")
#endif
 )
  return 1; /* AUTOFS */
 else
#endif /* HAVE_GETFSSTAT */
  return 0; /* no AUTOFS */
}

Do you know what your config.h status is for HAVE_GETFSSTAT and MNTTYPE_AUTOFS?

[Peter Newman (peternewman)]

Which I guess in their style would be (and I suspect this is probably what you tried):
/* This function checks whether current file system is an AutoFs
 * HRFS_entry must be valid prior to calling this function
 * return 1 if AutoFs, 0 otherwise
 */
int
Check_HR_FileSys_AutoFs(void)
{
#if HAVE_GETFSSTAT
    if (HRFS_entry->HRFS_type != NULL &&
#if defined(MNTTYPE_AUTOFS)
        !strcmp(HRFS_entry->HRFS_type, MNTTYPE_AUTOFS)
#else
        !strcmp(HRFS_entry->HRFS_type, "autofs")
#endif
        )
        return 1; /* AUTOFS */
#endif /* HAVE_GETFSSTAT */
    return 0; /* no AUTOFS */
}

[Joseph Gooch (whitemystic)]

I removed
#if HAVE_GETFSSTAT
And
#endif /* HAVE_GETFSSTAT */

Based on looking at the Check_HR_FileSys_NFS function earlier in the file, it was structured like:
#if HAVE_GETFSSTAT
  something that looks nothing like the autofs function above with different constants
#else
  something that looks just like the autofs function above
#endif

BUT, I'm not sure if there should be a HAVE_GETFSSTAT section in this function, and if so what it should be. I'm basically poking around and compiling and doing a lot of trial and error. I played enough in GDB to determine that netsnmp_fsys_arch_load IS finding all the correct mountpoints, but I couldn't directly determine at what point the physical ones were being thrown away. The constants all over the code made it very difficult to debug with GDB as it couldn't resolve the precompiler directives, and I very quickly got sucked into a rabbit hole of header spelunking.

I pinned the previous release on my servers to fix the immediate issue. I may have more time to look later, but it's very likely it'll take the original patch author, or someone who has any experience working in the daemon far less time to resolve :)

[Joseph Gooch (whitemystic)]

Or not, I think I found it.

This line:
        /* Skip AUTOFS entries */
        if (entry->type == NETSNMP_FS_TYPE_AUTOFS)
            continue;

Looks really innocuous until you run it through gcc -E:
        if (entry->type == 7 | 0x1000 | 0x2000)
            continue;

At which point you realize the | is going to make this if TRUE all the time.

I need to find the define and add ()'s around it.

[David Ford (atheros1200)]

Could be wrong...but is this what you are looking for

+Index: net-snmp-5.7.3+dfsg/include/net-snmp/agent/hardware/fsys.h
+===================================================================
+--- net-snmp-5.7.3+dfsg.orig/include/net-snmp/agent/hardware/fsys.h
++++ net-snmp-5.7.3+dfsg/include/net-snmp/agent/hardware/fsys.h
+@@ -41,6 +41,7 @@ typedef struct netsnmp_fsys_info_s netsn
+ #define NETSNMP_FS_TYPE_SYSFS 4 | _NETSNMP_FS_TYPE_LOCAL | _NETSNMP_FS_TYPE_SKIP_BIT
+ #define NETSNMP_FS_TYPE_TMPFS 5 | _NETSNMP_FS_TYPE_LOCAL
+ #define NETSNMP_FS_TYPE_USBFS 6 | _NETSNMP_FS_TYPE_LOCAL
++#define NETSNMP_FS_TYPE_AUTOFS 7 | _NETSNMP_FS_TYPE_LOCAL | _NETSNMP_FS_TYPE_SKIP_BIT
+
+ #define NETSNMP_FS_FLAG_ACTIVE 0x01
+ #define NETSNMP_FS_FLAG_REMOTE 0x02

[Joseph Gooch (whitemystic)]

Patch provided on upstream bugreport
https://sourceforge.net/p/net-snmp/patches/1350/

Here's the quilt version of the chatter...

[Joseph Gooch (whitemystic)]

Dev is apparently happening on this ticket FYI
https://bugs.launchpad.net/ubuntu/+source/net-snmp/+bug/1843036