floating ip portforwarding from external not working

[Maximilian Stinsky]

We are running neutron victoria 17.4.1. The floating ip port forwarding extention doesn't seem to work with external traffic.

After debugging it looks like the neutron port forwarding extention doesn't create a necessary iptables rule responsible for setting the 0x4000000/0xffff0000 mark.

(public ip addresses replaced with *.*.*.)
Reproduction:
`openstack floating ip port forwarding create --internal-ip-address 10.0.0.227 --port dfba05b5-31ba-466b-80d9-79df7e053e7f --internal-protocol-port 80 --external-protocol-port 80 --protocol tcp *.*.*.172`

We see a created dnat rule:
Chain neutron-l3-agent-pf-b6eaee1f (1 references)
 pkts bytes target prot opt in out source destination
    0 0 DNAT tcp -- * * 0.0.0.0/0 *.*.*.172 tcp dpt:80 to:10.0.0.227:80

But there is no rule in the mangle table that sets the required marker to pass the DROP rule inside the neutron-l3-agent-scope chain.

Speaking for the described suspicion is a functioning port forwarding from the internal network of the vm. Those packets are received by the qr interface of the router before hitting the DROP rule.
This is the mangle rule which seems to work for internal traffic:
Chain neutron-l3-agent-scope (1 references)
 pkts bytes target prot opt in out source destination
75895 11M MARK all -- qr-61cfbe6d-89 * 0.0.0.0/0 0.0.0.0/0 MARK xset 0x4000000/0xffff0000

If I see this correctly the port forwarding extention needs to create a mangle chain/rule like all other floating ips get, example floating ip chain:
Chain neutron-l3-agent-floatingip (1 references)
 pkts bytes target prot opt in out source destination
42728 4400K MARK all -- * * 0.0.0.0/0 *.*.*.89 MARK xset 0x4000000/0xffff0000

[Elvira García Ruiz (elviragr)]

Hi!
If I understand correctly, a floating ip is always supposed to be an external net IP. Therefore, I'm sorry but I don't understand what you mean here by internal/external.

[Maximilian Stinsky (mstinsky)]

What I mean is from where I am testing the port forwarding rule.

With external I mean that I test the rule from outside of the vm's private network, so directly from the internet.
A http package is first received by the qg interface of the router. From there it goes through all the iptables rules and gets dropped by the drop rules inside the neutron-l3-agent-scope FORWARD chain. The reason for this as far as I can see is that it does not get the required mark set by any iptables rule.

So the packet flow until we drop the port forwarding traffic looks something like:
-> http packet received by qg interface
-> iptables mangle tables
-> neutron-l3-agent-PREROUTING
-> neutron-l3-agent-scope - as we come from a qg interface our packet gets marked with 0x4010000/0xffff0000
-> iptables nat tables
-> neutron-l3-agent-PREROUTING
-> neutron-l3-agent-fip-pf - we apply the dnat rule for the port forwarding
-> iptables "normal" table (FORWARDING)
-> neutron-l3-agent-FORWARD
-> neutron-l3-agent-scope
-> packet gets dropped by mark match ! 0x4000000/0xffff0000 rule.

The internal test means that I test the rule from inside of the vm's private network which is the destination of the port forwarding rule. In this case the port forwarding works and the reason for this is that the traffic does not hit the router on the qg interface first but comes from the qr interface as its the internal network. Because the traffic first hits the router from the qr interface and there is a rule that those packets are all getting marked the port forwarding traffic is not dropped by the neutron-l3-agent-scope FORWARD chain.

The packet flow for this case looks something like:
-> http packet received by the qr interface of the router
-> neutron-l3-agent-PREROUTING
-> neutron-l3-agent-scope - because we come from a qr interface we get mark 0x4000000/0xffff0000 applied
-> iptables nat tables
-> neutron-l3-agent-PREROUTING
-> neutron-l3-agent-fip-pf - we apply the dnat rule for the port forwarding
-> iptables "normal" table (FORWARDING)
-> neutron-l3-agent-FORWARD
-> neutron-l3-agent-scope
-> packet gets through the drop rule which failed in the other case because we got the 0x4000000/0xffff0000 mark through the qr interface rule.

---
For "normal" floating ip use cases without port forwarding the attachment of a floating ip create the following rule:
Chain neutron-l3-agent-floatingip (1 references)
 pkts bytes target prot opt in out source destination
  164 11008 MARK all -- * * 0.0.0.0/0 *.*.*.89 MARK xset 0x4000000/0xffff0000

With that floating ip traffic passes through the drop rule and this is what I think is missing for port forwarding rules.