SNMP stopped running all of sudden (snmpd 5.8+dfsg-2)

[ankur]

[Impact]

When the user requests a bulkget operation using the snmpv3 protocol, and this operation errors out, snmpd will experience a double free error and will abort. This will obviously be very annoying to the user, because from the client side it is not possible to determine why the server suddenly stopped responding.

Upstream has fixed this problem already, albeit it took several patches to get everything right. The fix involved redesigning how a specific struct is manipulated internally, including the addition of a refcount mechanism.

[Test Case]

To reproduce the issue, one can do:

$ lxc launch ubuntu-daily:focal net-snmp-bug1877027-focal
$ lxc shell net-snmp-bug1877027-focal
# apt update
# apt install -y snmpd snmp
# systemctl stop snmpd.service
# cat >> /var/lib/snmp/snmpd.conf << __EOF__
createUser testuser SHA "testpass" AES "testpass"
__EOF__
# cat >> /etc/snmp/snmpd.conf << __EOF__
rwuser testuser
__EOF__
# systemct start snmpd.service
# snmpbulkget -v3 -Cn1 -Cr1472 -l authPriv -u testuser -a SHA -A testpass -x AES -X testpass 127.0.0.1 1.3.6.1.2.1.1.5 1.3.6.1.2.1.1.7

You can check that snmpd crashed by doing:

# systemctl status snmpd.service

[Regression Potential]

Due to the fact that several patches needed to be backported to fix this issue, and that parts of the code had to be redesign in order to guarantee that nothing is being freed twice, there exists the possibility of encountering a regression when dealing with snmpv3, because most of the code that was touched was there to deal with security states of the protocol.

I took a non-trivial amount of time to review the patches and make sure that they are as contained as possible, and I am satisfied with the approach taken by upstream.

[Other Info]

It is interesting to note that when the bug is fixed, the "snmpbulkget" command will fail with:

  Error in packet.
  Reason: (genError) A general failure occured

This is expected, and upstream also shows this error.

[Original Description]

Logs:

Apr 30 19:39:11 snmpd[1895]: Cannot adopt OID in UCD-SNMP-MIB: laIndex ::= { laEntry 1 }

Apr 30 19:39:11 snmpd[1895]: /etc/snmp/snmpd.conf: line 157: Warning: Unknown token: defaultMonitors.

Apr 30 19:39:11 snmpd[1895]: /etc/snmp/snmpd.conf: line 160: Warning: Unknown token: linkUpDownNotifications.

Apr 30 19:39:11 snmpd[1895]: /var/lib/snmp/snmpd.conf: line 34: Warning: Unknown token: _mteTTable.

Apr 30 19:39:11 snmpd[1895]: /var/lib/snmp/snmpd.conf: line 35: Warning: Unknown token: _mteTTable.

Apr 30 19:39:11 snmpd[1895]: /var/lib/snmp/snmpd.conf: line 36: Warning: Unknown token: _mteTTable.

Apr 30 19:39:11 snmpd[1895]: /var/lib/snmp/snmpd.conf: line 37: Warning: Unknown token: _mteTTable.

Apr 30 19:39:11 snmpd[1895]: /var/lib/snmp/snmpd.conf: line 38: Warning: Unknown token: _mteTTable.

Apr 30 19:39:11 snmpd[1895]: /var/lib/snmp/snmpd.conf: line 39: Warning: Unknown token: _mteTTable.

Apr 30 19:39:11 snmpd[1895]: /var/lib/snmp/snmpd.conf: line 40: Warning: Unknown token: _mteTTable.

Apr 30 19:39:11 snmpd[1895]: Error opening specified endpoint "udp:161"

Apr 30 19:39:11 snmpd[1895]: Server Exiting with code 1

[ankur (kulhar007)]

snmpd 5.8+dfsg-2 "double free or corruption"

openat(AT_FDCWD, "/proc/net/ipv6_route", O_RDONLY) = 14
fstat(14, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
read(14, "fe800000000000000000000000000000"..., 1024) = 900
read(14, "", 1024) = 0
close(14) = 0
clock_gettime(CLOCK_MONOTONIC, {501384, 839231761}) = 0
clock_gettime(CLOCK_MONOTONIC, {501384, 839412979}) = 0
writev(2, [
{"*double free or corruption* (fastt"..., 35}

, {"\n", 1}], 2) = 36
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f39a2e10000
rt_sigprocmask(SIG_UNBLOCK, [ABRT], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1], [], 8) = 0
getpid() = 17928
gettid() = 17928
tgkill(17928, 17928, SIGABRT) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0

                SIGABRT {si_signo=SIGABRT, si_code=SI_TKILL, si_pid=17928, si_uid=111} ---*
                +++ killed by SIGABRT +++

[Paride Legovini (paride)]

Thank you for taking the time to report this bug and helping to make Ubuntu better.

There isn't really enough information here for a developer to confirm this issue is a bug, or to begin working on it, so I am marking this bug Incomplete for now.

If you can provide exact steps so that a developer can reproduce the original problem, like a minimal snmpd.conf which causes the problem, then please add them to this bug and change the status back to New. Please specify if this is happening on Focal al Groovy (the current development release).

Thanks!

[ankur (kulhar007)]

@Paride Legovini (legovini)

This issue is super easy to reproduce, you can take any ubuntu version including ubuntu 20.04 with snmpd 5.8+dfsg-2, configure snmpv3 and run following command

snmpbulkget -v3 -Cn1 -Cr1472 -l authPriv -u testuser -a SHA -A <pass> -x AES -X <pass> 127.0.0.1 1.3.6.1.2.1.1.5 1.3.6.1.2.1.1.7

It will immediately crash.

[ankur (kulhar007)]

net-snmp has given a patch in https://sourceforge.net/p/net-snmp/bugs/2923/

Is there any plan to release a new veersion of SNMP?

[ankur (kulhar007)]

Hi
Updated the bug, please have a look.

Regards
Ankur kulhar

On Thu, May 7, 2020 at 10:11 PM Paride Legovini <
<email address hidden>> wrote:

> Thank you for taking the time to report this bug and helping to make
> Ubuntu better.
>
> There isn't really enough information here for a developer to confirm
> this issue is a bug, or to begin working on it, so I am marking this bug
> Incomplete for now.
>
> If you can provide exact steps so that a developer can reproduce the
> original problem, like a minimal snmpd.conf which causes the problem,
> then please add them to this bug and change the status back to New.
> Please specify if this is happening on Focal al Groovy (the current
> development release).
>
> Thanks!
>
> ** Changed in: net-snmp (Ubuntu)
> Status: New => Incomplete
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1877027
>
> Title:
> SNMP stopped running all of sudden (snmpd 5.8+dfsg-2)
>
> Status in net-snmp package in Ubuntu:
> Incomplete
>
> Bug description:
> Logs:
>
> Apr 30 19:39:11 snmpd[1895]: Cannot adopt OID in UCD-SNMP-MIB: laIndex
> ::= { laEntry 1 }
>
> Apr 30 19:39:11 snmpd[1895]: /etc/snmp/snmpd.conf: line 157: Warning:
> Unknown token: defaultMonitors.
>
> Apr 30 19:39:11 snmpd[1895]: /etc/snmp/snmpd.conf: line 160: Warning:
> Unknown token: linkUpDownNotifications.
>
> Apr 30 19:39:11 snmpd[1895]: /var/lib/snmp/snmpd.conf: line 34:
> Warning: Unknown token: _mteTTable.
>
> Apr 30 19:39:11 snmpd[1895]: /var/lib/snmp/snmpd.conf: line 35:
> Warning: Unknown token: _mteTTable.
>
> Apr 30 19:39:11 snmpd[1895]: /var/lib/snmp/snmpd.conf: line 36:
> Warning: Unknown token: _mteTTable.
>
> Apr 30 19:39:11 snmpd[1895]: /var/lib/snmp/snmpd.conf: line 37:
> Warning: Unknown token: _mteTTable.
>
> Apr 30 19:39:11 snmpd[1895]: /var/lib/snmp/snmpd.conf: line 38:
> Warning: Unknown token: _mteTTable.
>
> Apr 30 19:39:11 snmpd[1895]: /var/lib/snmp/snmpd.conf: line 39:
> Warning: Unknown token: _mteTTable.
>
> Apr 30 19:39:11 snmpd[1895]: /var/lib/snmp/snmpd.conf: line 40:
> Warning: Unknown token: _mteTTable.
>
> Apr 30 19:39:11 snmpd[1895]: Error opening specified endpoint
> "udp:161"
>
> Apr 30 19:39:11 snmpd[1895]: Server Exiting with code 1
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/net-snmp/+bug/1877027/+subscriptions
>

[Paride Legovini (paride)]

Thanks for following up and for linking to the upstream work. To recap:

Upstream bug: https://sourceforge.net/p/net-snmp/bugs/2923/
Upstream patch: https://sourceforge.net/p/net-snmp/patches/1388/

The patch has been applied upstream [1] to the V5-8-patches branch as commit

adc9b71aba9168ec64149345ea37a1acc11875c6

but was not perfect, the patch has been amended by:

7384a8b550d4ed4a00e41b72229cfcc124926b06
a6ba68ff77ca60260d5f5b007f409d08ce75c3b1

Not really a "patch on a table" scenario, but we do have a bug and a patch.

[1] https://git.code.sf.net/p/net-snmp/code

[ankur (kulhar007)]

By when ubuntu will officially release a Debian package with this fix ?

[Ken Mix (kmix)]

I can reliably reproduce this on my 20.04 (vmWare 6.7 amd64 guest) systems running the following:

sudo snmpbulkget -v3 -Cn0 -Cr50 -l authPriv -u testuser -a SHA -A <pass> -x DES -X <pass> 127.0.0.1 iso.3.6.1.2.1.4.22.1.4

A gdb backtrace looks like the following:

double free or corruption (fasttop)

Program received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50 return ret;
(gdb) bt
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x00007ffff7979859 in __GI_abort () at abort.c:79
#2 0x00007ffff79e43ee in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7ffff7b0e285 "%s\n") at ../sysdeps/posix/libc_fatal.c:155
#3 0x00007ffff79ec47c in malloc_printerr (str=str@entry=0x7ffff7b10628 "double free or corruption (fasttop)") at malloc.c:5347
#4 0x00007ffff79edde5 in _int_free (av=0x7ffff7b3fb80 <main_arena>, p=0x55555586ea10, have_lock=0) at malloc.c:4266
#5 0x00007ffff7bc2a14 in usm_free_usmStateReference () from /lib/x86_64-linux-gnu/libnetsnmp.so.35
#6 0x00007ffff7bc7c8d in usm_generate_out_msg () from /lib/x86_64-linux-gnu/libnetsnmp.so.35
#7 0x00007ffff7bc8609 in usm_secmod_generate_out_msg () from /lib/x86_64-linux-gnu/libnetsnmp.so.35
#8 0x00007ffff7b7e05a in snmpv3_packet_build () from /lib/x86_64-linux-gnu/libnetsnmp.so.35
#9 0x00007ffff7b80087 in snmp_build () from /lib/x86_64-linux-gnu/libnetsnmp.so.35
#10 0x00007ffff7b805ab in netsnmp_build_packet () from /lib/x86_64-linux-gnu/libnetsnmp.so.35
#11 0x00007ffff7b807cf in _build_initial_pdu_packet () from /lib/x86_64-linux-gnu/libnetsnmp.so.35
#12 0x00007ffff7f85a18 in netsnmp_wrap_up_request () from /lib/x86_64-linux-gnu/libnetsnmpagent.so.35
#13 0x00007ffff7f88beb in netsnmp_handle_request () from /lib/x86_64-linux-gnu/libnetsnmpagent.so.35
#14 0x00007ffff7f88f1b in handle_snmp_packet () from /lib/x86_64-linux-gnu/libnetsnmpagent.so.35
#15 0x00007ffff7b8882f in ?? () from /lib/x86_64-linux-gnu/libnetsnmp.so.35
#16 0x00007ffff7b898c6 in _sess_read () from /lib/x86_64-linux-gnu/libnetsnmp.so.35
#17 0x00007ffff7b8a3fd in snmp_sess_read2 () from /lib/x86_64-linux-gnu/libnetsnmp.so.35
#18 0x00007ffff7b8a44b in snmp_read2 () from /lib/x86_64-linux-gnu/libnetsnmp.so.35
#19 0x0000555555559ca2 in ?? ()
#20 0x000055555555910d in main ()

I haven't been able to reproduce running the command remotely, but it does seem to reliably crash snmpd when running locally. However, we use SolarWinds Orion NPM to monitor our internal infrastructure, and it runs similar bulkget commands that will reliably crash snmpd remotely on our 20.04 test servers.

Please let me know if there's any other information I can provide.

[ankur (kulhar007)]

Any update on this bug? we are still waiting for the fix.

[Sergio Durigan Junior (sergiodj)]

On Tuesday, June 23 2020, ankur wrote:

> Any update on this bug? we are still waiting for the fix.

Yes, I think I finally have found the right patches to backport in order
to solve the segfault.

Before I go ahead with the merge proposal, I'd like to ask you to give
it a try with this PPA:

  https://launchpad.net/~sergiodj/+archive/ubuntu/net-snmp-bug1877027

Can you let me know if it solves the issue you're seeing?

Thanks,

--
Sergio
GPG key ID: E92F D0B3 6B14 F1F4 D8E0 EB2F 106D A1C8 C3CB BF14

[Ken Mix (kmix)]

Hello,

Running the reproducer with the build from your PPA now just shows a "genError" error instead of crashing the service:

$ sudo snmpbulkget -v3 -Cn0 -Cr50 -l authPriv -u testuser -a SHA -A <pass> -x DES -X <pass> 127.0.0.1 iso.3.6.1.2.1.4.22.1.4
Error in packet.
Reason: (genError) A general failure occured

Standard SNMP gets and bulk gets work as expected before and after the reproducer above, so this seems to fix the issue from my perspective.

I notice your PPA was for groovy -- will these patches be backported into focal, as well?

Thanks!

Ken Mix

[Sergio Durigan Junior (sergiodj)]

On Tuesday, June 23 2020, Ken Mix wrote:

> Hello,
>
> Running the reproducer with the build from your PPA now just shows a
> "genError" error instead of crashing the service:
>
> $ sudo snmpbulkget -v3 -Cn0 -Cr50 -l authPriv -u testuser -a SHA -A <pass> -x DES -X <pass> 127.0.0.1 iso.3.6.1.2.1.4.22.1.4
> Error in packet.
> Reason: (genError) A general failure occured
>
> Standard SNMP gets and bulk gets work as expected before and after the
> reproducer above, so this seems to fix the issue from my perspective.

Cool. That's what I'm seeing as well. I also confirmed that the
general failure reported by the command above also happens upstream.

> I notice your PPA was for groovy -- will these patches be backported
> into focal, as well?

Yes, I will SRU this for focal too.

Thanks,

--
Sergio
GPG key ID: E92F D0B3 6B14 F1F4 D8E0 EB2F 106D A1C8 C3CB BF14

[Andreas Hasenack (ahasenack)]

@security team, this looks like a remote DoS that can be triggered by authenticated users

[Steve Beattie (sbeattie)]

Andreas, agreed, I think (speaking from the Ubuntu Security Team's perspective), this should go to focal-security.

[Steve Beattie (sbeattie)]

FYI, this was assigned CVE-2019-20892.

[Launchpad Janitor (janitor)]

This bug was fixed in the package net-snmp - 5.8+dfsg-2ubuntu2.1

---------------
net-snmp (5.8+dfsg-2ubuntu2.1) focal-security; urgency=medium

  * SECURITY UPDATE: Fix segmentation fault that happens when using the
    snmpv3 protocol with snmpbulkget. (LP: #1877027)
    - d/p/move-securityStateRef-into-free_securityStateRef.patch:
      Consolidate the check of the securityStateRef pointer into the
      free_securityStateRef function.
    - d/p/prevent-snmpv3-bulkget-errors-double-free.patch:
      Prevent snmpv3 bulkget errors from becoming resulting in a
      double free.
    - d/p/fix-usmStateReference-free.patch:
      Fix typo on usm_free_usmStateReference from last patch.
    - d/p/unexport-struct-usmStateReference.patch:
      Unexport struct usmStateReference and to prevent ABI breakages,
      since it will be necessary to add a reference count to it.
    - d/p/introduce-refcount-usmStateReference.patch:
      Introduce refcount in the struct usmStateReference, and adjust
      code to properly use the field.
    - CVE-2019-20892

 -- Sergio Durigan Junior <email address hidden> Tue, 23 Jun 2020 14:57:12 -0400

[Launchpad Janitor (janitor)]

This bug was fixed in the package net-snmp - 5.8+dfsg-2ubuntu3

---------------
net-snmp (5.8+dfsg-2ubuntu3) groovy; urgency=medium

  * SECURITY UPDATE: Fix segmentation fault that happens when using the
    snmpv3 protocol with snmpbulkget. (LP: #1877027)
    - d/p/move-securityStateRef-into-free_securityStateRef.patch:
      Consolidate the check of the securityStateRef pointer into the
      free_securityStateRef function.
    - d/p/prevent-snmpv3-bulkget-errors-double-free.patch:
      Prevent snmpv3 bulkget errors from becoming resulting in a
      double free.
    - d/p/fix-usmStateReference-free.patch:
      Fix typo on usm_free_usmStateReference from last patch.
    - d/p/unexport-struct-usmStateReference.patch:
      Unexport struct usmStateReference and to prevent ABI breakages,
      since it will be necessary to add a reference count to it.
    - d/p/introduce-refcount-usmStateReference.patch:
      Introduce refcount in the struct usmStateReference, and adjust
      code to properly use the field.
    - CVE-2019-20892

 -- Sergio Durigan Junior <email address hidden> Tue, 23 Jun 2020 14:37:42 -0400