Packets incorrectly marked as martian

[Harry Kominos]

Problem:
The following behaviour is observed:

The deployment has 2 provider networks. One of the them is the public one and another is
getting outside but through NAT. This second one is the one that they hypervisors use and this is what we have as "openstack public" (10.20.6.X). The VMs that are launched are attached to the fabric on the 10.10 public network. Therefore that network is not present on the hypervisor NICS. What we observe is that the switch is sending ARP requests (correctly) from the .250 active standby IP but the kernel is marking them as Martian despite the fact that neutron knows this network.

System:
triple-O Based Rocky Deployment . VXLAN tunneling, DVR enabled with Bond interfaces on 2 switches. (Open vSwitch) 2.11.0, Neutron 13.0.5
kernel: 3.10.0-957.21.3.el7.x86_64
Host OS: CentOS
Switches: Arista

------------------ --------------- ---------------
| SWITCH | | HYPERVISOR | | VM |
| 10.10.91.250 | --- | 10.20.6.X | --- | 10.10.X.Y/23 |
------------------ --------------- ---------------

Subnet details :

+-------------------+--------------------------------------+
| Field | Value |
+-------------------+--------------------------------------+
| allocation_pools | 10.10.90.10-10.10.91.240 |
| cidr | 10.10.90.0/23 |
| created_at | 2019-09-24T08:43:54Z |
| description | |
| dns_nameservers | 10.10.D.Y |
| enable_dhcp | True |
| gateway_ip | 10.10.91.254 |
| host_routes | |
| id | f91d725a-89d1-4a32-97b5-95409177e8eb |
| ip_version | 4 |
| ipv6_address_mode | None |
| ipv6_ra_mode | None |
| name | public-subnet |
| network_id | a1a3280b-9c78-4e5f-883a-9b4bc4e72b1f |
| project_id | ec9851ba91854e10bb8d5e752260f5fd |
| revision_number | 14 |
| segment_id | None |
| service_types | |
| subnetpool_id | None |
| tags | |
| updated_at | 2020-03-03T14:34:26Z |
+-------------------+--------------------------------------+

cat openvswitch_agent.ini
[agent]
l2_population=True
arp_responder=True
enable_distributed_routing=True
drop_flows_on_start=False
extensions=qos
tunnel_csum=False
tunnel_types=vxlan
vxlan_udp_port=4789

[securitygroup]
firewall_driver=iptables_hybrid

Expected output:
No Martian Packets observed

Actual output:
Since The extra provider network is configured I would expect that the linux kernel would not mark the incoming packets as martian.

However,

Mar 9 10:45:41 compute0 kernel: IPv4: martian source 10.10.90.74 from 10.10.91.250 on dev qbrff08c591-e2
Mar 9 10:45:41 compute0 kernel: ll header: 00000000: ff ff ff ff ff ff 98 5d 82 a1 a6 cd 08 06 .......]......
Mar 9 10:45:42 compute0 kernel: IPv4: martian source 10.10.90.74 from 10.10.91.250 on dev qbrff08c591-e2
Mar 9 10:45:42 compute0 kernel: ll header: 00000000: ff ff ff ff ff ff 98 5d 82 a1 a6 cd 08 06 .......]......
Mar 9 10:45:43 compute0 kernel: IPv4: martian source 10.10.91.203 from 10.10.91.250 on dev qbrff08c591-e2
Mar 9 10:45:43 compute0 kernel: ll header: 00000000: ff ff ff ff ff ff 98 5d 82 a1 a6 cd 08 06 .......]......
Mar 9 10:45:44 compute0 kernel: IPv4: martian source 10.10.90.74 from 10.10.91.250 on dev qbrff08c591-e2
Mar 9 10:45:44 compute0 kernel: ll header: 00000000: ff ff ff ff ff ff 98 5d 82 a1 a6 cd 08 06 .......]......
Mar 9 10:45:44 compute0 kernel: IPv4: martian source 10.10.91.203 from 10.10.91.250 on dev qbrff08c591-e2
Mar 9 10:45:44 compute0 kernel: ll header: 00000000: ff ff ff ff ff ff 98 5d 82 a1 a6 cd 08 06 .......]......

Perceived severity:
Minor annoyance since /var/log/messages is flooded.
Minor security vulnerability since disabling martian packet logging will hide a real case of martian packet flooding

[Bernard Cafarelli (bcafarel)]

Ah these pesky martian packets... Set priority according to setup specifics and VMs actually working fine

[Bence Romsics (bence-romsics)]

I'm not sure what's happening here, but there's a workaround you may want to try. Since you're already using ovs you could change the 'firewall_driver' from 'iptables' to 'ovs'. Likely the firewall performance would improve and the qbr bridges would be gone therefore the martian detection would be gone too.

One thing I find very strange in the logs is that you receive packets on qbr interfaces. I would only expect packets in the interfaces enslaved into the qbr bridge (tap and qvb IIRC), not on the qbr itself.

[Harry Kominos (hkominos)]

Thank you for the suggestion Bence.

I have indeed switched to openvswitch for the firewall driver and most of the Martial Packets are gone from the logs.

Regarding your second part of the comment, It is true that we only see Martian packets on the Linux Bridge only and not on any other module of the networking path.

[Brian Haley (brian-haley)]

I am going to close this since moving to the OVS firewall driver has helped, and I'm not sure anyone will take the time to investigate further as OVN is now the default driver. Someone can re-open if they intend on working on it.