While running the commands that triggers the server hang I see a bunch of connects, but the same is true for the working walks.
Once that is triggered snmpd falls into a busy loop (I'd assume some sort of spinning deadlock / infinite loop)
on Strace I see the following repeating:
0.000085 openat(AT_FDCWD, "/proc/sys/net/ipv4/conf/all/forwarding", O_RDONLY) = 11 <0.000026>
0.000057 fstat(11, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 <0.000014>
0.000043 read(11, "1\n", 1024) = 2 <0.000014>
0.000044 close(11) = 0 <0.000014>
[...]
Also installing a bunch of dbg symbols to look at where it spins:
$ apt install snmp-dbgsym snmpd-dbgsym libsnmp30-dbg libsensors4-dbgsym libpci3-dbgsym zlib1g-dbg libudev1-dbgsym libpcre3-dbg
An example backtrace:
#0 0x00007fe4de8fad0e in __libc_open64 (file=0x7fe4defd29c0 <ipfw_name> "/proc/sys/net/ipv4/conf/all/forwarding", oflag=0)
at ../sysdeps/unix/sysv/linux/open64.c:46
#1 0x00007fe4de87ad83 in __GI__IO_file_open (fp=fp@entry=0x558c3bd609a0, filename=<optimized out>, posix_mode=<optimized out>, prot=prot@entry=438,
read_write=8, is32not64=is32not64@entry=1) at fileops.c:229
#2 0x00007fe4de87b055 in _IO_new_file_fopen (fp=fp@entry=0x558c3bd609a0,
filename=filename@entry=0x7fe4defd29c0 <ipfw_name> "/proc/sys/net/ipv4/conf/all/forwarding", mode=<optimized out>, mode@entry=0x7fe4defaa952 "r",
is32not64=is32not64@entry=1) at fileops.c:334
#3 0x00007fe4de86da24 in __fopen_internal (filename=0x7fe4defd29c0 <ipfw_name> "/proc/sys/net/ipv4/conf/all/forwarding", mode=0x7fe4defaa952 "r", is32=1)
at iofopen.c:86
#4 0x00007fe4de86da6a in _IO_new_fopen (filename=<optimized out>, mode=<optimized out>) at iofopen.c:97
#5 0x00007fe4def7f593 in netsnmp_arch_ip_scalars_ipForwarding_get (value=value@entry=0x7ffe83c37990) at ip-mib/data_access/scalars_linux.c:29
#6 0x00007fe4def36460 in handle_ipForwarding (handler=<optimized out>, reginfo=<optimized out>, reqinfo=0x558c3bec8740, requests=0x558c3bee4b10)
at ip-mib/ip_scalars.c:112
#7 0x00007fe4df35425d in netsnmp_call_handler (requests=0x558c3bee4b10, reqinfo=0x558c3bec8740, reginfo=0x558c3be69640, next_handler=0x558c3be695d0)
at agent_handler.c:526
#8 netsnmp_call_next_handler (current=current@entry=0x558c3be69710, reginfo=reginfo@entry=0x558c3be69640, reqinfo=reqinfo@entry=0x558c3bec8740,
requests=requests@entry=0x558c3bee4b10) at agent_handler.c:640
#9 0x00007fe4df343370 in netsnmp_instance_helper_handler (handler=0x558c3be69710, reginfo=0x558c3be69640, reqinfo=0x558c3bec8740, requests=0x558c3bee4b10)
at helpers/instance.c:775
#10 0x00007fe4df35425d in netsnmp_call_handler (requests=0x558c3bee4b10, reqinfo=0x558c3bec8740, reginfo=0x558c3be69640, next_handler=0x558c3be69710)
at agent_handler.c:526
#11 netsnmp_call_next_handler (current=current@entry=0x558c3be69780, reginfo=reginfo@entry=0x558c3be69640, reqinfo=reqinfo@entry=0x558c3bec8740,
requests=requests@entry=0x558c3bee4b10) at agent_handler.c:640
#12 0x00007fe4df345e7d in netsnmp_scalar_helper_handler (handler=0x558c3be69780, reginfo=0x558c3be69640, reqinfo=0x558c3bec8740, requests=0x558c3bee4b10)
at helpers/scalar.c:188
#13 0x00007fe4df35425d in netsnmp_call_handler (requests=0x558c3bee4b10, reqinfo=0x558c3bec8740, reginfo=0x558c3be69640, next_handler=0x558c3be69780)
at agent_handler.c:526
#14 netsnmp_call_next_handler (current=current@entry=0x558c3be697f0, reginfo=reginfo@entry=0x558c3be69640, reqinfo=reqinfo@entry=0x558c3bec8740,
requests=requests@entry=0x558c3bee4b10) at agent_handler.c:640
#15 0x00007fe4df3466dd in netsnmp_serialize_helper_handler (handler=0x558c3be697f0, reginfo=0x558c3be69640, reqinfo=0x558c3bec8740, requests=<optimized out>)
at helpers/serialize.c:66
#16 0x00007fe4df353d17 in netsnmp_call_handler (requests=0x558c3bee4b10, reqinfo=0x558c3bec8740, reginfo=0x558c3be69640, next_handler=0x558c3be697f0)
at agent_handler.c:526
#17 netsnmp_call_handlers (reginfo=0x558c3be69640, reqinfo=0x558c3bec8740, requests=0x558c3bee4b10) at agent_handler.c:611
#18 0x00007fe4df361d1f in handle_var_requests (asp=asp@entry=0x558c3bec88e0) at snmp_agent.c:2676
#19 0x00007fe4df362180 in handle_getnext_loop (asp=asp@entry=0x558c3bec88e0) at snmp_agent.c:3122
#20 0x00007fe4df362a87 in handle_pdu (asp=asp@entry=0x558c3bec88e0) at snmp_agent.c:3451
#21 0x00007fe4df362c59 in netsnmp_handle_request (asp=asp@entry=0x558c3bec88e0, status=status@entry=0) at snmp_agent.c:3281
#22 0x00007fe4df3630da in handle_snmp_packet (op=<optimized out>, session=<optimized out>, reqid=<optimized out>, pdu=0x558c3bee4da0, magic=<optimized out>)
at snmp_agent.c:1987
---Type <return> to continue, or q <return> to quit---
#23 0x00007fe4dec1461a in _sess_process_packet (sessp=sessp@entry=0x558c3bec5eb0, sp=sp@entry=0x558c3bec5f50, isp=isp@entry=0x558c3bec5ee0,
transport=transport@entry=0x558c3bec5c50, opaque=0x558c3bedfba0, olength=<optimized out>,
packetptr=0x558c3becf170 "0\201\215\002\001\003\060\021\002\004'\025d\311\002\003", length=144) at snmp_api.c:5441
#24 0x00007fe4dec151af in _sess_read (sessp=sessp@entry=0x558c3bec5eb0, fdset=fdset@entry=0x7ffe83c37da0) at snmp_api.c:5876
#25 0x00007fe4dec15cd9 in snmp_sess_read2 (sessp=sessp@entry=0x558c3bec5eb0, fdset=fdset@entry=0x7ffe83c37da0) at snmp_api.c:5908
#26 0x00007fe4dec15d2b in snmp_read2 (fdset=0x7ffe83c37da0) at snmp_api.c:5504
#27 0x0000558c3a4b09b1 in receive () at snmpd.c:1328
#28 main (argc=<optimized out>, argv=<optimized out>) at snmpd.c:1115
Looking for a loop there:
It seems to loop from handle_getnext_loop, but I don't see yet how the limited_vw config would influence that.
With that so far in mind I found [1] and [2].
Which both seem very related.
One explains that some OIDs would have dependent OIDs that without snmpwalk would cause just such a hang. for that [3]
I wonder if these fixes actually went in already - and if not if they would be applicable for a test. Could you check if you agree that this seems to be the same issue, essentially "limiting OID visibility triggers snmpwalk busy loop".
While running the commands that triggers the server hang I see a bunch of connects, but the same is true for the working walks.
Once that is triggered snmpd falls into a busy loop (I'd assume some sort of spinning deadlock / infinite loop)
on Strace I see the following repeating: sys/net/ ipv4/conf/ all/forwarding" , O_RDONLY) = 11 <0.000026> S_IFREG| 0644, st_size=0, ...}) = 0 <0.000014>
0.000085 openat(AT_FDCWD, "/proc/
0.000057 fstat(11, {st_mode=
0.000043 read(11, "1\n", 1024) = 2 <0.000014>
0.000044 close(11) = 0 <0.000014>
[...]
Also installing a bunch of dbg symbols to look at where it spins:
$ apt install snmp-dbgsym snmpd-dbgsym libsnmp30-dbg libsensors4-dbgsym libpci3-dbgsym zlib1g-dbg libudev1-dbgsym libpcre3-dbg
An example backtrace: d29c0 <ipfw_name> "/proc/ sys/net/ ipv4/conf/ all/forwarding" , oflag=0) unix/sysv/ linux/open64. c:46 entry=0x558c3bd 609a0, filename=<optimized out>, posix_mode= <optimized out>, prot=prot@ entry=438, is32not64@ entry=1) at fileops.c:229 entry=0x558c3bd 609a0, filename@ entry=0x7fe4def d29c0 <ipfw_name> "/proc/ sys/net/ ipv4/conf/ all/forwarding" , mode=<optimized out>, mode@entry= 0x7fe4defaa952 "r", is32not64@ entry=1) at fileops.c:334 0x7fe4defd29c0 <ipfw_name> "/proc/ sys/net/ ipv4/conf/ all/forwarding" , mode=0x7fe4defaa952 "r", is32=1) <optimized out>, mode=<optimized out>) at iofopen.c:97 arch_ip_ scalars_ ipForwarding_ get (value= value@entry= 0x7ffe83c37990) at ip-mib/ data_access/ scalars_ linux.c: 29 0x558c3bec8740, requests= 0x558c3bee4b10) ip_scalars. c:112 call_handler (requests= 0x558c3bee4b10, reqinfo= 0x558c3bec8740, reginfo= 0x558c3be69640, next_handler= 0x558c3be695d0) call_next_ handler (current= current@ entry=0x558c3be 69710, reginfo= reginfo@ entry=0x558c3be 69640, reqinfo= reqinfo@ entry=0x558c3be c8740, requests@ entry=0x558c3be e4b10) at agent_handler.c:640 instance_ helper_ handler (handler= 0x558c3be69710, reginfo= 0x558c3be69640, reqinfo= 0x558c3bec8740, requests= 0x558c3bee4b10) instance. c:775 call_handler (requests= 0x558c3bee4b10, reqinfo= 0x558c3bec8740, reginfo= 0x558c3be69640, next_handler= 0x558c3be69710) call_next_ handler (current= current@ entry=0x558c3be 69780, reginfo= reginfo@ entry=0x558c3be 69640, reqinfo= reqinfo@ entry=0x558c3be c8740, requests@ entry=0x558c3be e4b10) at agent_handler.c:640 scalar_ helper_ handler (handler= 0x558c3be69780, reginfo= 0x558c3be69640, reqinfo= 0x558c3bec8740, requests= 0x558c3bee4b10) scalar. c:188 call_handler (requests= 0x558c3bee4b10, reqinfo= 0x558c3bec8740, reginfo= 0x558c3be69640, next_handler= 0x558c3be69780) call_next_ handler (current= current@ entry=0x558c3be 697f0, reginfo= reginfo@ entry=0x558c3be 69640, reqinfo= reqinfo@ entry=0x558c3be c8740, requests@ entry=0x558c3be e4b10) at agent_handler.c:640 serialize_ helper_ handler (handler= 0x558c3be697f0, reginfo= 0x558c3be69640, reqinfo= 0x558c3bec8740, requests=<optimized out>) serialize. c:66 call_handler (requests= 0x558c3bee4b10, reqinfo= 0x558c3bec8740, reginfo= 0x558c3be69640, next_handler= 0x558c3be697f0) call_handlers (reginfo= 0x558c3be69640, reqinfo= 0x558c3bec8740, requests= 0x558c3bee4b10) at agent_handler.c:611 entry=0x558c3be c88e0) at snmp_agent.c:2676 entry=0x558c3be c88e0) at snmp_agent.c:3122 entry=0x558c3be c88e0) at snmp_agent.c:3451 handle_ request (asp=asp@ entry=0x558c3be c88e0, status= status@ entry=0) at snmp_agent.c:3281 packet (sessp= sessp@entry= 0x558c3bec5eb0, sp=sp@entry= 0x558c3bec5f50, isp=isp@ entry=0x558c3be c5ee0, transport@ entry=0x558c3be c5c50, opaque= 0x558c3bedfba0, olength=<optimized out>, 0x558c3becf170 "0\201\ 215\002\ 001\003\ 060\021\ 002\004' \025d\311\ 002\003" , length=144) at snmp_api.c:5441 sessp@entry= 0x558c3bec5eb0, fdset=fdset@ entry=0x7ffe83c 37da0) at snmp_api.c:5876 sessp@entry= 0x558c3bec5eb0, fdset=fdset@ entry=0x7ffe83c 37da0) at snmp_api.c:5908 0x7ffe83c37da0) at snmp_api.c:5504
#0 0x00007fe4de8fad0e in __libc_open64 (file=0x7fe4def
at ../sysdeps/
#1 0x00007fe4de87ad83 in __GI__IO_file_open (fp=fp@
read_write=8, is32not64=
#2 0x00007fe4de87b055 in _IO_new_file_fopen (fp=fp@
filename=
is32not64=
#3 0x00007fe4de86da24 in __fopen_internal (filename=
at iofopen.c:86
#4 0x00007fe4de86da6a in _IO_new_fopen (filename=
#5 0x00007fe4def7f593 in netsnmp_
#6 0x00007fe4def36460 in handle_ipForwarding (handler=<optimized out>, reginfo=<optimized out>, reqinfo=
at ip-mib/
#7 0x00007fe4df35425d in netsnmp_
at agent_handler.c:526
#8 netsnmp_
requests=
#9 0x00007fe4df343370 in netsnmp_
at helpers/
#10 0x00007fe4df35425d in netsnmp_
at agent_handler.c:526
#11 netsnmp_
requests=
#12 0x00007fe4df345e7d in netsnmp_
at helpers/
#13 0x00007fe4df35425d in netsnmp_
at agent_handler.c:526
#14 netsnmp_
requests=
#15 0x00007fe4df3466dd in netsnmp_
at helpers/
#16 0x00007fe4df353d17 in netsnmp_
at agent_handler.c:526
#17 netsnmp_
#18 0x00007fe4df361d1f in handle_var_requests (asp=asp@
#19 0x00007fe4df362180 in handle_getnext_loop (asp=asp@
#20 0x00007fe4df362a87 in handle_pdu (asp=asp@
#21 0x00007fe4df362c59 in netsnmp_
#22 0x00007fe4df3630da in handle_snmp_packet (op=<optimized out>, session=<optimized out>, reqid=<optimized out>, pdu=0x558c3bee4da0, magic=<optimized out>)
at snmp_agent.c:1987
---Type <return> to continue, or q <return> to quit---
#23 0x00007fe4dec1461a in _sess_process_
transport=
packetptr=
#24 0x00007fe4dec151af in _sess_read (sessp=
#25 0x00007fe4dec15cd9 in snmp_sess_read2 (sessp=
#26 0x00007fe4dec15d2b in snmp_read2 (fdset=
#27 0x0000558c3a4b09b1 in receive () at snmpd.c:1328
#28 main (argc=<optimized out>, argv=<optimized out>) at snmpd.c:1115
Looking for a loop there: getnext_ loop, but I don't see yet how the limited_vw config would influence that.
It seems to loop from handle_
With that so far in mind I found [1] and [2].
Which both seem very related.
One explains that some OIDs would have dependent OIDs that without snmpwalk would cause just such a hang. for that [3]
I wonder if these fixes actually went in already - and if not if they would be applicable for a test. Could you check if you agree that this seems to be the same issue, essentially "limiting OID visibility triggers snmpwalk busy loop".
[1]: https:/ /sourceforge. net/p/net- snmp/mailman/ message/ 35379818/ /sourceforge. net/p/net- snmp/bugs/ 2529/ /github. com/fenner/ net-snmp/ commits/ V5-7-fix- view-filtering
[2]: https:/
[3]: https:/