No dhcp IPv6 assigned (slaac/slaac) with interface-add after VM boot

[Ritesh Anand]

Instance does not get IP from dhcp ipv6 subnet (slaac/slaac) with DVR, when router interface is added after VM creation.

Instance does get IP when it is booted after interface to the subnet has already been added to the DVR.
This ordering issue is not observed with centralized router.

Easy to recreate.

On compute:
----------------------

NOT getting IP, when router interface is added after VM has been created:
$ ifconfig
eth0 Link encap:Ethernet HWaddr FA:16:3E:9C:15:B7
          inet6 addr: fe80::f816:3eff:fe9c:15b7/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:14 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1116 (1.0 KiB) TX bytes:1138 (1.1 KiB)

lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:16436 Metric:1
          RX packets:12 errors:0 dropped:0 overruns:0 frame:0
          TX packets:12 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1020 (1020.0 B) TX bytes:1020 (1020.0 B)

Gets IP when router interface is added before VM is booted.
$
$ ifconfig
eth0 Link encap:Ethernet HWaddr FA:16:3E:9C:15:B7
          inet6 addr: 4001:db8::f816:3eff:fe9c:15b7/64 Scope:Global
          inet6 addr: fe80::f816:3eff:fe9c:15b7/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:15 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1226 (1.1 KiB) TX bytes:1138 (1.1 KiB)

lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:16436 Metric:1
          RX packets:12 errors:0 dropped:0 overruns:0 frame:0
          TX packets:12 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1020 (1020.0 B) TX bytes:1020 (1020.0 B)

$

Subnet:
stack@osctrlr:~/devstack$ neutron subnet-show ipv62s1
+-------------------+------------------------------------------------------------------+
| Field | Value |
+-------------------+------------------------------------------------------------------+
| allocation_pools | {"start": "4001:db8::2", "end": "4001:db8::ffff:ffff:ffff:ffff"} |
| cidr | 4001:db8::/64 |
| dns_nameservers | |
| enable_dhcp | True |
| gateway_ip | 4001:db8::1 |
| host_routes | |
| id | 2b24b126-f618-4daa-a3a8-24ea8720a0db |
| ip_version | 6 |
| ipv6_address_mode | slaac |
| ipv6_ra_mode | slaac |
| name | ipv62s1 |
| network_id | d9a71eed-0768-46b7-be0e-74664211f28b |
| subnetpool_id | |
| tenant_id | 9fbdd2326fe34e949ece2bef8c8f8c8c |
+-------------------+------------------------------------------------------------------+
stack@osctrlr:~/devstack$

Router:
stack@osctrlr:~/devstack$ neutron router-show dvr
+-----------------------+--------------------------------------+
| Field | Value |
+-----------------------+--------------------------------------+
| admin_state_up | True |
| distributed | True |
| external_gateway_info | |
| ha | False |
| id | 3512b48b-a1a8-4923-9a4b-0720dfd71baf |
| name | dvr |
| routes | |
| status | ACTIVE |
| tenant_id | 9fbdd2326fe34e949ece2bef8c8f8c8c |
+-----------------------+--------------------------------------+
stack@osctrlr:~/devstack$

[Armando Migliaccio (armando-migliaccio)]

@Swami can you triage this one, please?

[Swaminathan Vasudevan (swaminathan-vasudevan)]

Ok let me triage it.

[Ritesh Anand (ritesh-anand)]

Steps:
On a multi node setup with 1 ctrlr+network node and 2 compute nodes.
1 create a netowrk:
neutron net-create ipv62
 2. create an IPv6 dhcp enabled slaac/slaac subnet:
    neutron subnet-create ipv62 4001:db8::/64 --name ipv62s1 --enable-dhcp --ip-version 6 --ipv6-ra-mode slaac --ipv6-address-mode slaac
3. create a dvr:
    neutron router-delete dvr

  287 neutron router-interface-add dvr subnet=2b24b126-f618-4daa-a3a8-24ea8720a0db

[Ritesh Anand (ritesh-anand)]

Please ignore previous comment.

Steps:
On a multi node setup with 1 ctrlr+network node and 2 compute nodes.
1 create a netowrk:
    neutron net-create ipv62
 2. create an IPv6 dhcp enabled slaac/slaac subnet:
    neutron subnet-create ipv62 4001:db8::/64 --name ipv62s1 --enable-dhcp --ip-version 6 --ipv6-ra-mode slaac --ipv6-address-mode slaac
3. create a dvr:
    neutron router-delete dvr
4. Onstantiate a VM with nic on ipv62 network
    nova boot --flavor 1 --image 81df6d9a-e3b2-4a32-bd53-6b5e148d3844 --nic net-id=d9a71eed-0768-46b7-be0e-74664211f28b testv6vm1
5. add router interface for the subnet ipv62s1
   neutron router-interface-add dvr subnet=2b24b126-f618-4daa-a3a8-24ea8720a0db

Observe that VM does not get IP.
If we swap steps 4 and 5, it behaves as expected.

[Swaminathan Vasudevan (swaminathan-vasudevan)]

Do you see any RA adv going from the router to the VM.

[Ritesh Anand (ritesh-anand)]

I do see RAs going out...

stack@osctrlr:~/devstack$ sudo ip netns exec qrouter-3512b48b-a1a8-4923-9a4b-0720dfd71baf tcpdump -vv -i qr-e014e036-68
tcpdump: WARNING: qr-e014e036-68: no IPv4 address assigned
tcpdump: listening on qr-e014e036-68, link-type EN10MB (Ethernet), capture size 65535 bytes
^C09:20:33.523705 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 56) fe80::f816:3eff:fece:3887 > ip6-allnodes: [icmp6 sum ok] ICMP6, router advertisement, length 56
        hop limit 64, Flags [none], pref medium, router lifetime 30s, reachable time 0s, retrans time 0s
          prefix info option (3), length 32 (4): 4001:db8::/64, Flags [onlink, auto], valid time 86400s, pref. time 14400s
            0x0000: 40c0 0001 5180 0000 3840 0000 0000 4001
            0x0010: 0db8 0000 0000 0000 0000 0000 0000
          source link-address option (1), length 8 (1): fa:16:3e:ce:38:87
            0x0000: fa16 3ece 3887
09:20:40.258282 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 56) fe80::f816:3eff:fece:3887 > ip6-allnodes: [icmp6 sum ok] ICMP6, router advertisement, length 56
        hop limit 64, Flags [none], pref medium, router lifetime 30s, reachable time 0s, retrans time 0s
          prefix info option (3), length 32 (4): 4001:db8::/64, Flags [onlink, auto], valid time 86400s, pref. time 14400s
            0x0000: 40c0 0001 5180 0000 3840 0000 0000 4001
            0x0010: 0db8 0000 0000 0000 0000 0000 0000
          source link-address option (1), length 8 (1): fa:16:3e:ce:38:87
            0x0000: fa16 3ece 3887
09:20:47.727934 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 56) fe80::f816:3eff:fece:3887 > ip6-allnodes: [icmp6 sum ok] ICMP6, router advertisement, length 56
        hop limit 64, Flags [none], pref medium, router lifetime 30s, reachable time 0s, retrans time 0s
          prefix info option (3), length 32 (4): 4001:db8::/64, Flags [onlink, auto], valid time 86400s, pref. time 14400s
            0x0000: 40c0 0001 5180 0000 3840 0000 0000 4001
            0x0010: 0db8 0000 0000 0000 0000 0000 0000
          source link-address option (1), length 8 (1): fa:16:3e:ce:38:87
            0x0000: fa16 3ece 3887
09:20:55.535328 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 56) fe80::f816:3eff:fece:3887 > ip6-allnodes: [icmp6 sum ok] ICMP6, router advertisement, length 56
        hop limit 64, Flags [none], pref medium, router lifetime 30s, reachable time 0s, retrans time 0s
          prefix info option (3), length 32 (4): 4001:db8::/64, Flags [onlink, auto], valid time 86400s, pref. time 14400s
            0x0000: 40c0 0001 5180 0000 3840 0000 0000 4001
            0x0010: 0db8 0000 0000 0000 0000 0000 0000
          source link-address option (1), length 8 (1): fa:16:3e:ce:38:87
            0x0000: fa16 3ece 3887

4 packets captured
4 packets received by filter
0 packets dropped by kernel
stack@osctrlr:~/devstack$

stack@osctrlr:~/devstack$ sudo ip netns exec qrouter-3512b48b-a1a8-4923-9a4b-0720dfd71baf ifconfig
lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:65536 Metric:1
          RX packet...

[Swaminathan Vasudevan (swaminathan-vasudevan)]

Ritesh it is better to add any packet traces to the paste bin.

[Swaminathan Vasudevan (swaminathan-vasudevan)]

I have verified the bug.
I can reproduce the same problem on a single node devstack.

The steps to reproduce is captured here.

http://paste.openstack.org/show/475223/

The IP Address is not assigned to the VM's interface. But the VM port still has a valid ip.

[Swaminathan Vasudevan (swaminathan-vasudevan)]

On further investigation, it seems that the Router Advertisement does not get to the VM , when DVR routers are in place.

I could see the "Router solicitation" message and "neighbor solicitation" message coming from the VM to the Router Namespace, but I don't see any RA reaching the VM.

This might be the problem, why we are not seeing an IP address assignment when configured for SLAAC , dhcp with DVR.

[Nell Jerram (neil-jerram)]

It's certainly correct that the VM will not set up an SLAAC IPv6 address, if it does not receive any RA, or if its processing of RAs is disabled. So, if the RA is not reaching the VM, that is sufficient to explain what is being observed.

What is in the data path between the radvd and the VM? It should just be a matter of bisecting that path with tcpdump, to find out if the RA is being dropped somewhere.

Another possibility is whether the VM processes it. When my project was first testing IPv4, we needed a few tweaks in the guest (Ubuntu) OS:

    In /etc/network/interfaces.d/eth0.cfg, add:

    iface eth0 inet6 dhcp
            accept_ra 1

    In /sbin/dhclient-script, add at the start of the script:

    new_ip6_prefixlen=128

    In /etc/sysctl.d, create a file named 30-eth0-rs-delay.conf with contents:

    net.ipv6.conf.eth0.router_solicitation_delay = 10

(This is from http://docs.projectcalico.org/en/latest/ipv6.html.)

However, I can't immediately see why the presence or absence of any of those would make a difference between the VM processing an RA at boot time, and the VM processing an RA at some later time.

But maybe seeing this might trigger some thoughts?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/237403

[Swaminathan Vasudevan (swaminathan-vasudevan)]

Provider security group rule set on the input chain for the VM port to all the RA packets is missing for the DVR Routers.

A neutron-openvswi-i08bad1ab-e -s fe80::f816:3eff:fee1:f8d4/128 -p ipv6-icmp -m icmp6 --icmpv6-type 134 -j RETURN

It is only addressed for the "DEVICE_OWNER_ROUTER_INTF" and not for "DEVICE_OWNER_ROUTER_DISTRIBUTED_INTF".

I have a patch and it fixes the issue seen. Will push it in.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/237403
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a7dc528c1860f6a325a36bb7c633c60b635fc9a6
Submitter: Jenkins
Branch: master

commit a7dc528c1860f6a325a36bb7c633c60b635fc9a6
Author: Swaminathan Vasudevan <email address hidden>
Date: Mon Oct 19 22:02:38 2015 -0700

    Set security group provider rule for icmpv6 RA in DVR

    Security group provider rules for RA is set for the VM ports
    when a router interface is added or updated after the VM
    instance is created.

    In the case of DVR Routers the security group provider rule
    to allow the RA packets to flow through the VM port input
    chain was missing and so the VM was not able to get a
    SLAAC/DHCP address when associated with a DVR Router.

    This fix will add the security group rule to the VM port input
    chain to allow the RA packets to flow into the VM and hence
    the VM will obtain an IP address assigned by the Router.

    Closes-Bug: #1501969

    Change-Id: Ib0b3499d9c880fe1462734b2d4092debf4819509

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/238990

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/238990
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=765c7481645dd1ce389dd2b04a0021cd5bd93110
Submitter: Jenkins
Branch: stable/liberty

commit 765c7481645dd1ce389dd2b04a0021cd5bd93110
Author: Swaminathan Vasudevan <email address hidden>
Date: Mon Oct 19 22:02:38 2015 -0700

    Set security group provider rule for icmpv6 RA in DVR

    Security group provider rules for RA is set for the VM ports
    when a router interface is added or updated after the VM
    instance is created.

    In the case of DVR Routers the security group provider rule
    to allow the RA packets to flow through the VM port input
    chain was missing and so the VM was not able to get a
    SLAAC/DHCP address when associated with a DVR Router.

    This fix will add the security group rule to the VM port input
    chain to allow the RA packets to flow into the VM and hence
    the VM will obtain an IP address assigned by the Router.

    Closes-Bug: #1501969

    Change-Id: Ib0b3499d9c880fe1462734b2d4092debf4819509
    (cherry picked from commit a7dc528c1860f6a325a36bb7c633c60b635fc9a6)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/240933

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/240933
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1578dcbb8d3b210a283f7b188d3788582462f968
Submitter: Jenkins
Branch: stable/kilo

commit 1578dcbb8d3b210a283f7b188d3788582462f968
Author: Swaminathan Vasudevan <email address hidden>
Date: Mon Oct 19 22:02:38 2015 -0700

    Set security group provider rule for icmpv6 RA in DVR

    Security group provider rules for RA is set for the VM ports
    when a router interface is added or updated after the VM
    instance is created.

    In the case of DVR Routers the security group provider rule
    to allow the RA packets to flow through the VM port input
    chain was missing and so the VM was not able to get a
    SLAAC/DHCP address when associated with a DVR Router.

    This fix will add the security group rule to the VM port input
    chain to allow the RA packets to flow into the VM and hence
    the VM will obtain an IP address assigned by the Router.

    Closes-Bug: #1501969

    Conflicts:
     neutron/db/securitygroups_rpc_base.py

    Change-Id: Ib0b3499d9c880fe1462734b2d4092debf4819509
    (cherry picked from commit a7dc528c1860f6a325a36bb7c633c60b635fc9a6)

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/neutron 8.0.0.0b1 development milestone.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/neutron 7.0.1 release.