Comment 4 for bug 1742671

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

While running the commands that triggers the server hang I see a bunch of connects, but the same is true for the working walks.

Once that is triggered snmpd falls into a busy loop (I'd assume some sort of spinning deadlock / infinite loop)

on Strace I see the following repeating:
     0.000085 openat(AT_FDCWD, "/proc/sys/net/ipv4/conf/all/forwarding", O_RDONLY) = 11 <0.000026>
     0.000057 fstat(11, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 <0.000014>
     0.000043 read(11, "1\n", 1024) = 2 <0.000014>
     0.000044 close(11) = 0 <0.000014>
     [...]

Also installing a bunch of dbg symbols to look at where it spins:
$ apt install snmp-dbgsym snmpd-dbgsym libsnmp30-dbg libsensors4-dbgsym libpci3-dbgsym zlib1g-dbg libudev1-dbgsym libpcre3-dbg

An example backtrace:
#0 0x00007fe4de8fad0e in __libc_open64 (file=0x7fe4defd29c0 <ipfw_name> "/proc/sys/net/ipv4/conf/all/forwarding", oflag=0)
    at ../sysdeps/unix/sysv/linux/open64.c:46
#1 0x00007fe4de87ad83 in __GI__IO_file_open (fp=fp@entry=0x558c3bd609a0, filename=<optimized out>, posix_mode=<optimized out>, prot=prot@entry=438,
    read_write=8, is32not64=is32not64@entry=1) at fileops.c:229
#2 0x00007fe4de87b055 in _IO_new_file_fopen (fp=fp@entry=0x558c3bd609a0,
    filename=filename@entry=0x7fe4defd29c0 <ipfw_name> "/proc/sys/net/ipv4/conf/all/forwarding", mode=<optimized out>, mode@entry=0x7fe4defaa952 "r",
    is32not64=is32not64@entry=1) at fileops.c:334
#3 0x00007fe4de86da24 in __fopen_internal (filename=0x7fe4defd29c0 <ipfw_name> "/proc/sys/net/ipv4/conf/all/forwarding", mode=0x7fe4defaa952 "r", is32=1)
    at iofopen.c:86
#4 0x00007fe4de86da6a in _IO_new_fopen (filename=<optimized out>, mode=<optimized out>) at iofopen.c:97
#5 0x00007fe4def7f593 in netsnmp_arch_ip_scalars_ipForwarding_get (value=value@entry=0x7ffe83c37990) at ip-mib/data_access/scalars_linux.c:29
#6 0x00007fe4def36460 in handle_ipForwarding (handler=<optimized out>, reginfo=<optimized out>, reqinfo=0x558c3bec8740, requests=0x558c3bee4b10)
    at ip-mib/ip_scalars.c:112
#7 0x00007fe4df35425d in netsnmp_call_handler (requests=0x558c3bee4b10, reqinfo=0x558c3bec8740, reginfo=0x558c3be69640, next_handler=0x558c3be695d0)
    at agent_handler.c:526
#8 netsnmp_call_next_handler (current=current@entry=0x558c3be69710, reginfo=reginfo@entry=0x558c3be69640, reqinfo=reqinfo@entry=0x558c3bec8740,
    requests=requests@entry=0x558c3bee4b10) at agent_handler.c:640
#9 0x00007fe4df343370 in netsnmp_instance_helper_handler (handler=0x558c3be69710, reginfo=0x558c3be69640, reqinfo=0x558c3bec8740, requests=0x558c3bee4b10)
    at helpers/instance.c:775
#10 0x00007fe4df35425d in netsnmp_call_handler (requests=0x558c3bee4b10, reqinfo=0x558c3bec8740, reginfo=0x558c3be69640, next_handler=0x558c3be69710)
    at agent_handler.c:526
#11 netsnmp_call_next_handler (current=current@entry=0x558c3be69780, reginfo=reginfo@entry=0x558c3be69640, reqinfo=reqinfo@entry=0x558c3bec8740,
    requests=requests@entry=0x558c3bee4b10) at agent_handler.c:640
#12 0x00007fe4df345e7d in netsnmp_scalar_helper_handler (handler=0x558c3be69780, reginfo=0x558c3be69640, reqinfo=0x558c3bec8740, requests=0x558c3bee4b10)
    at helpers/scalar.c:188
#13 0x00007fe4df35425d in netsnmp_call_handler (requests=0x558c3bee4b10, reqinfo=0x558c3bec8740, reginfo=0x558c3be69640, next_handler=0x558c3be69780)
    at agent_handler.c:526
#14 netsnmp_call_next_handler (current=current@entry=0x558c3be697f0, reginfo=reginfo@entry=0x558c3be69640, reqinfo=reqinfo@entry=0x558c3bec8740,
    requests=requests@entry=0x558c3bee4b10) at agent_handler.c:640
#15 0x00007fe4df3466dd in netsnmp_serialize_helper_handler (handler=0x558c3be697f0, reginfo=0x558c3be69640, reqinfo=0x558c3bec8740, requests=<optimized out>)
    at helpers/serialize.c:66
#16 0x00007fe4df353d17 in netsnmp_call_handler (requests=0x558c3bee4b10, reqinfo=0x558c3bec8740, reginfo=0x558c3be69640, next_handler=0x558c3be697f0)
    at agent_handler.c:526
#17 netsnmp_call_handlers (reginfo=0x558c3be69640, reqinfo=0x558c3bec8740, requests=0x558c3bee4b10) at agent_handler.c:611
#18 0x00007fe4df361d1f in handle_var_requests (asp=asp@entry=0x558c3bec88e0) at snmp_agent.c:2676
#19 0x00007fe4df362180 in handle_getnext_loop (asp=asp@entry=0x558c3bec88e0) at snmp_agent.c:3122
#20 0x00007fe4df362a87 in handle_pdu (asp=asp@entry=0x558c3bec88e0) at snmp_agent.c:3451
#21 0x00007fe4df362c59 in netsnmp_handle_request (asp=asp@entry=0x558c3bec88e0, status=status@entry=0) at snmp_agent.c:3281
#22 0x00007fe4df3630da in handle_snmp_packet (op=<optimized out>, session=<optimized out>, reqid=<optimized out>, pdu=0x558c3bee4da0, magic=<optimized out>)
    at snmp_agent.c:1987
---Type <return> to continue, or q <return> to quit---
#23 0x00007fe4dec1461a in _sess_process_packet (sessp=sessp@entry=0x558c3bec5eb0, sp=sp@entry=0x558c3bec5f50, isp=isp@entry=0x558c3bec5ee0,
    transport=transport@entry=0x558c3bec5c50, opaque=0x558c3bedfba0, olength=<optimized out>,
    packetptr=0x558c3becf170 "0\201\215\002\001\003\060\021\002\004'\025d\311\002\003", length=144) at snmp_api.c:5441
#24 0x00007fe4dec151af in _sess_read (sessp=sessp@entry=0x558c3bec5eb0, fdset=fdset@entry=0x7ffe83c37da0) at snmp_api.c:5876
#25 0x00007fe4dec15cd9 in snmp_sess_read2 (sessp=sessp@entry=0x558c3bec5eb0, fdset=fdset@entry=0x7ffe83c37da0) at snmp_api.c:5908
#26 0x00007fe4dec15d2b in snmp_read2 (fdset=0x7ffe83c37da0) at snmp_api.c:5504
#27 0x0000558c3a4b09b1 in receive () at snmpd.c:1328
#28 main (argc=<optimized out>, argv=<optimized out>) at snmpd.c:1115

Looking for a loop there:
It seems to loop from handle_getnext_loop, but I don't see yet how the limited_vw config would influence that.

With that so far in mind I found [1] and [2].

Which both seem very related.
One explains that some OIDs would have dependent OIDs that without snmpwalk would cause just such a hang. for that [3]

I wonder if these fixes actually went in already - and if not if they would be applicable for a test. Could you check if you agree that this seems to be the same issue, essentially "limiting OID visibility triggers snmpwalk busy loop".

[1]: https://sourceforge.net/p/net-snmp/mailman/message/35379818/
[2]: https://sourceforge.net/p/net-snmp/bugs/2529/
[3]: https://github.com/fenner/net-snmp/commits/V5-7-fix-view-filtering