SNMP traps use src IP of nodes mgmt network

[Jorge Saffe]

Brief Description
------------------
when using SNMP traps in my (subcloud) cluster, I notice that the source IP of the traps
is an IP address from the mgmt networks. The exact IP depends is from that node where the ns-snmp pod runs.

Severity
---------
<Major: SNMP traps should not use cluster-internal IP addresses

Steps to Reproduce
-------------------
Follow documentation to install/run SNMP with external trap destination. Trigger some alarms. Check traps on destination with tcpdump.

tcpdump running at trap destination:
15:04:49.974555 IP 10.87.0.43.51650 > 10.88.0.140.snmptrap: C="testcommunity" V2Trap(478) system.sysUpTime.0=1109294 S:1.1.4.1.0=E:731.1.1.1.1.0.2 E:731.1.1.1.1.1.1.3.0="100.108" E:731.1.1.1.1.1.1.4.0="region=regional-cloud-sd.system=regional-cloud-sd.host=worker6.port=eno2" E:731.1.1.1.1.1.1.5.0="2022-05-11,15:04:49.0,+0:0" E:731.1.1.1.1.1.1.6.0=3 E:731.1.1.1.1.1.1.5.0="'MGMT' port failed" E:731.1.1.1.1.1.1.8.0=7 E:731.1.1.1.1.1.1.9.0=0 E:731.1.1.1.1.1.1.10.0="Check cabling and far-end port configuration and status on adjacent equipment." E:731.1.1.1.1.1.1.11.0=1 E:731.1.1.1.1.1.1.12.0=1

Expected Behavior
------------------
Traps should use public cluster IP e.g. from OAM network.

Actual Behavior
----------------
Traps use internal IP from mgmt network.

Reproducibility
----------------
Reproducible

Same behavior can be seen with any pod-generated traffic to cluster-external destination (e.g. ping external IP)

System Configuration
---------------------
Subcloud with two controllers and 6 workers. In a test the sn-snmp pod was running on worker3:

Last Pass
-----------
Never

Timestamp/Logs
--------------
N/A

Alarms
------
PN/A

Test Activity
--------------
Evaluation

Workaround
-----------
N/A

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/stx-puppet/+/849452

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/stx-puppet/+/849452
Committed: https://opendev.org/starlingx/stx-puppet/commit/638e2292b52256943306d448f654baad15dd4b15
Submitter: "Zuul (22348)"
Branch: master

commit 638e2292b52256943306d448f654baad15dd4b15
Author: Jorge Saffe <email address hidden>
Date: Tue Jul 12 02:33:19 2022 -0400

    Apply NAT on k8s outgoing pkgs for all protocols

    By default the platform firewall for kubernetes traffic
    is configured to apply NAT on outgoing traffic generated
    inside the K8s cluster only for TCP ports.

    The outgoing traffic of other protocolos (e.g. UDP, ICMP)
    are leaving the system without NAT. When the traffic is
    generated on workers nodes the packages leaves the system
    with the internal management IP address.

    Test Plan:
    ----------
    PASS: CENTOS Standard fresh install.
    PASS: Verify that the iptables "Kubernetes post-routing
    rule" is updated to accept all protocols.
    PASS: Verify src IP of outgoing traffic of UDP protocol
    from worker nodes (e.g. sending SNMP traps).

    Closes-Bug: 1981405
    Signed-off-by: Jorge Saffe <email address hidden>
    Change-Id: Id6d9465f318a8360ff5459d16255e8ce49dbdee9