ipv6 prefix delegated subnets are not accessable external of the router they are attached.

[Matthew Thode]

currently ip6tables in the qrouter namespace has the following rule. This causes unmarked packets to drop.

-A neutron-l3-agent-scope -o qr-ca9ffa4f-fd -m mark ! --mark 0x4010000/0xffff0000 -j DROP

It seems that prefix delegated subnets don't get that mark set on incoming trafic from the gateway port, I had to add my own rule to do that.

ip6tables -t mangle -A neutron-l3-agent-scope -i qg-ac290c4b-4f -j MARK --set-xmark 0x4010000/0xffff0000

At the moment that is probably too permissive, it should likely be limited based on the prefix delegated. with a '-d dead:beef:cafe::/64' or whatever the delegation is (tested this and it does work).

[Sean M. Collins (scollins)]

I guess the work done as part of the address-scopes blueprint broke this.

[Matthew Thode (prometheanfire)]

also, it seems like there are no address-scopes, which could be why this happens, though the subnet was created with a subnet-pool

neutron subnet-create ipv6-pd --name ipv6-pd-1 --ip_version 6 --ipv6_ra_mode slaac --ipv6_address_mode slaac --use_default_subnetpool

| subnetpool_id | prefix_delegation |

[Matthew Thode (prometheanfire)]

Is anyone working on this? It'd be nice to not have a broken feature two releases in a row :(

[Jorge (correajl)]

I've a deployment with IPv6 and this is very annoying because periodically the rules are refreshed (it seems like is this) and the manually inserted rules are removed.

I'm using a crontab script to check each qrouter and apply the rule manually.

---
#!/bin/bash

for qrouter in `ip netns | grep ^qrouter`; do
        interface_qg=`ip netns exec $qrouter ifconfig | grep ^qg | awk '{print $1}'`

        qtd_mark=`ip netns exec $qrouter ip6tables -n -L -v -t mangle | grep "MARK xset" | wc -l`
        if [ $qtd_mark -lt 2 ]; then
                echo "APLICAR ==> /bin/ip netns exec $qrouter ip6tables -t mangle -A neutron-l3-agent-scope -i $interface_qg -j MARK --set-xmark 0x4000000/0xffff0000"
                /bin/ip netns exec $qrouter ip6tables -t mangle -A neutron-l3-agent-scope -i $interface_qg -j MARK --set-xmark 0x4000000/0xffff0000
        else
                echo -e "[OK] Regra ip6tables"
        fi
done
---

[Matthew Thode (prometheanfire)]

That script is good if you only have one prefix delegated subnet. the mark changes for each subnet (my second one was 0x4010000).

[Carl Baldwin (carl-baldwin)]

I think this has skipped our notice because of the importance being marked as Medium. I've just raised it to High for visibility. Sorry this fell through the cracks. It was never really brought to my attention.

[Matthew Thode (prometheanfire)]

thanks for raising it to medium, I'm willing to test patches if necessary and would have written a patch myself but didn't know where to start for this particular issue.

[Matthew Thode (prometheanfire)]

I'm not sure if more info is needed, but this setup is ml2(linuxbridge)/vxlan

[Matthew Thode (prometheanfire)]

I can confirm this still exists in newton (and newton didn't start radvd as it should have...)

[Matthew Thode (prometheanfire)]

upon multiple removals and re-adds of the v6-pd subnet to the router I haven't been able to reproduce the radvd failure to start but will open a bug for it if I can reproduce / get more info.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/407025

[John Davidge (john-davidge)]

I've posted a WIP patch to address this issue: https://review.openstack.org/#/c/407025

Those affected, please check it out and confirm/deny if it works for you.

[Matthew Thode (prometheanfire)]

since this is the l3 code that's patch, can I test this on just my l3 node? or do I need to deploy it on lb-agent nodes too?

[John Davidge (john-davidge)]

@Matthew We already discussed this offline, but for anybody else interested, only l3-agent restart is needed.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/407025
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=cd38886d20c0900788f2c15157be1a016cb475be
Submitter: Jenkins
Branch: master

commit cd38886d20c0900788f2c15157be1a016cb475be
Author: John Davidge <email address hidden>
Date: Mon Dec 5 12:32:19 2016 +0000

    Fix iptables rules for Prefix Delegated subnets

    Make sure the correct iptables rule is added when the router gets
    an interface on a PD-enabled subnet. This will allow traffic on PD
    subnets to reach the external network.

    Includes a unit test for the new function, and modifies an
    existing test to verify the adding and removal of the rule.

    Change-Id: I42f8f42995e9809e5bda2b29726f7244c052ca1c
    Closes-Bug: #1570122

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 10.0.0.0rc1 release candidate.

[Matthew Thode (prometheanfire)]

I was just able to reproduce this on ocata :(

manually adding the rule allows traffic to pass. Haven't investigated further than that.

[John Davidge (john-davidge)]

@Matthew Thanks for the heads up, trying to reproduce locally.

[Matthew Thode (prometheanfire)]

so we narrowed it down to reboots (or l3agent restarts) causing this for me, not sure why yet... Gonna do more print statements to see why it isn't doing what it should do

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/444838

[Ihar Hrachyshka (ihar-hrachyshka)]

Matthew, if it still fails for you in Ocata, please reopen the bug.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/444838
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=bebedc42e51d636c2c061e02069080cd6c0db646
Submitter: Jenkins
Branch: stable/newton

commit bebedc42e51d636c2c061e02069080cd6c0db646
Author: John Davidge <email address hidden>
Date: Mon Dec 5 12:32:19 2016 +0000

    Fix iptables rules for Prefix Delegated subnets

    Make sure the correct iptables rule is added when the router gets
    an interface on a PD-enabled subnet. This will allow traffic on PD
    subnets to reach the external network.

    Includes a unit test for the new function, and modifies an
    existing test to verify the adding and removal of the rule.

    Conflicts:
         neutron/agent/l3/router_info.py
         neutron/tests/unit/agent/l3/test_agent.py

    Change-Id: I42f8f42995e9809e5bda2b29726f7244c052ca1c
    Closes-Bug: #1570122
    (cherry picked from commit cd38886d20c0900788f2c15157be1a016cb475be)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 9.3.0 release.

[Lars Erik Pedersen (pedersen-larserik)]

When will the neutron 9.3.0 release appear in the Cloud Archive repo for Xenial?

[Christian Ehrhardt  (paelzer)]

Added neturon/UCA tasks so it shows up more obviously for the Triagers of those components

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[James Page (james-page)]

Newton UCA has 9.4.0 - Marking UCA and Ubuntu tasks fix released.

[Lars Erik Pedersen (pedersen-larserik)]

Hi, we are still able to reproduce this bug in Queens. neutron-l3-agent 2:12.0.2-0ubuntu1~cloud0

[Lars Erik Pedersen (pedersen-larserik)]

It works initally, but if we reboot the l3-agent that has the router, the ip6tables rule disappears when the router is re-created on a new l3-agent.

After adding this manually:

ip6tables -t mangle -A neutron-l3-agent-scope -i qg-28f7e259-d2 -j MARK --set-xmark 0x4000000/0xffff0000

it works again.