Base config is on Ubuntu 14.04 running one controller node and two compute nodes, using Linux bridges for L2, l2pop enabled, IPv4 addresses on the private and external networks, IPv6 on the management network and setting astara_auto_add_resources to False (Note this problem occurs with this set to True).
After building a management network, a private network and a external network. I see the following:
root@controller:~# neutron net-list
+--------------------------------------+---------+---------------------------------------------------------------+
| id | name | subnets |
+--------------------------------------+---------+---------------------------------------------------------------+
| b86539ef-5258-491a-83bb-35e288de4824 | private | f2583811-72a1-4cb4-8731-dc748773d750 10.2.0.0/24 |
| d7375936-fcde-4901-8121-06d57daf8923 | mgt | 6ebddcce-6a68-4737-b950-d1b3e9129846 fdca:3ba5:a17a:acda::/64 |
| d9b77a3c-e791-4c7b-a02a-6c30143dc7b2 | public | e408cebc-bf12-45bc-afc3-7fc880d59e32 172.16.0.0/24 |
+--------------------------------------+---------+---------------------------------------------------------------+
root@controller:~# neutron subnet-list
+--------------------------------------+----------------+--------------------------+---------------------------------------------------------------------------------------+
| id | name | cidr | allocation_pools |
+--------------------------------------+----------------+--------------------------+---------------------------------------------------------------------------------------+
| 6ebddcce-6a68-4737-b950-d1b3e9129846 | mgt-subnet | fdca:3ba5:a17a:acda::/64 | {"start": "fdca:3ba5:a17a:acda::2", "end": "fdca:3ba5:a17a:acda:ffff:ffff:ffff:ffff"} |
| e408cebc-bf12-45bc-afc3-7fc880d59e32 | public-subnet | 172.16.0.0/24 | {"start": "172.16.0.2", "end": "172.16.0.254"} |
| f2583811-72a1-4cb4-8731-dc748773d750 | private-subnet | 10.2.0.0/24 | {"start": "10.2.0.2", "end": "10.2.0.254"} |
+--------------------------------------+----------------+--------------------------+---------------------------------------------------------------------------------------+
after booting a VM I see:
root@controller:~# nova list --all-tenants
+--------------------------------------+------------------------------------------------+----------------------------------+--------+------------+-------------+---------------------------------------------+
| ID | Name | Tenant ID | Status | Task State | Power State | Networks |
+--------------------------------------+------------------------------------------------+----------------------------------+--------+------------+-------------+---------------------------------------------+
| ed8258fd-4433-4d3b-a89e-2cada573008b | One | 334f5c6be07e4f70b1bc6efa91b8b17a | ACTIVE | - | Running | private=10.2.0.4 |
| e83c5dca-e2e1-4716-ad0e-76d69a204cc3 | ak-router-30335389-9624-49ba-bf6f-b70955e408ad | 9f94398dfd4c484382f2b7f881ab7400 | ACTIVE | - | Running | mgt=fdca:3ba5:a17a:acda:f816:3eff:feaf:68f8 |
+--------------------------------------+------------------------------------------------+----------------------------------+--------+------------+-------------+---------------------------------------------+
and
root@controller:~# neutron port-list
+--------------------------------------+--------------------------------------------------+-------------------+----------------------------------------------------------------------------------------------------------------+
| id | name | mac_address | fixed_ips |
+--------------------------------------+--------------------------------------------------+-------------------+----------------------------------------------------------------------------------------------------------------+
| 2b70914e-9e0f-424d-817e-9bfca6e9822a | ASTARA:VRRP:30335389-9624-49ba-bf6f-b70955e408ad | fa:16:3e:66:47:6d | {"subnet_id": "00000000-0000-0000-0000-000000000000", "ip_address": "fe80::f816:3eff:fe66:476d"} |
| 382e0d2f-b55a-4853-be88-b8a1822427ca | | fa:16:3e:28:d7:09 | {"subnet_id": "f2583811-72a1-4cb4-8731-dc748773d750", "ip_address": "10.2.0.4"} |
| 3959a90c-1b00-4fc2-a69b-36f7632b2683 | ASTARA:RUG:EXTERNAL | fa:16:3e:fa:0f:b0 | {"subnet_id": "f2583811-72a1-4cb4-8731-dc748773d750", "ip_address": "10.2.0.3"} |
| 85cdf4f0-34ea-41b8-8e7b-76152098534d | ASTARA:RUG:SERVICE | fa:16:3e:fe:2f:9e | {"subnet_id": "6ebddcce-6a68-4737-b950-d1b3e9129846", "ip_address": "fdca:3ba5:a17a:acda:f816:3eff:fefe:2f9e"} |
| 9bad8577-c72a-4fef-88c4-4e9a18b82db5 | | fa:16:3e:47:44:8b | {"subnet_id": "f2583811-72a1-4cb4-8731-dc748773d750", "ip_address": "10.2.0.1"} |
| d185d777-ab30-4f89-850c-cbd6358375c2 | ASTARA:VRRP:30335389-9624-49ba-bf6f-b70955e408ad | fa:16:3e:f7:8f:46 | {"subnet_id": "00000000-0000-0000-0000-000000000000", "ip_address": "fe80::f816:3eff:fef7:8f46"} |
| d4d11112-f090-4152-8a6e-f88d8fd30979 | ASTARA:MGT:30335389-9624-49ba-bf6f-b70955e408ad | fa:16:3e:af:68:f8 | {"subnet_id": "6ebddcce-6a68-4737-b950-d1b3e9129846", "ip_address": "fdca:3ba5:a17a:acda:f816:3eff:feaf:68f8"} |
| d7320222-7593-42cc-a448-3130d84b28ef | | fa:16:3e:a1:e5:5b | {"subnet_id": "e408cebc-bf12-45bc-afc3-7fc880d59e32", "ip_address": "172.16.0.2"} |
+--------------------------------------+--------------------------------------------------+-------------------+----------------------------------------------------------------------------------------------------------------+
logging into the router VM:
astara@ak-334f5c6be07e4f70b1bc6efa91b8b17a:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1280 qdisc pfifo_fast state UP group default qlen 1000
link/ether fa:16:3e:af:68:f8 brd ff:ff:ff:ff:ff:ff
inet6 fdca:3ba5:a17a:acda:f816:3eff:feaf:68f8/64 scope global
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:feaf:68f8/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether fa:16:3e:f7:8f:46 brd ff:ff:ff:ff:ff:ff
inet 172.16.0.2/24 brd 172.16.0.255 scope global eth1
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fef7:8f46/64 scope link
valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether fa:16:3e:66:47:6d brd ff:ff:ff:ff:ff:ff
inet 10.2.0.1/24 brd 10.2.0.255 scope global eth2
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe66:476d/64 scope link
valid_lft forever preferred_lft forever
The problem here is that the MAC address that astara uses for the subnet interface and the router gateway does not match what neutron thinks it should be using. Astara uses the interface ASTARA:VRRP:30335389-9624-49ba-bf6f-b70955e408ad as the subnet interface (subnet gateway) but neutron created the port id 9bad8577-c72a-4fef-88c4-4e9a18b82db5 with the assigned IP of 10.2.0.1 as the subnet interface (gateway). Similarly, the router external gateway interface and the one built by astara are different.
The VMs are created on different VMs and becused of the different ports being used the l2population entries on each compute nodes are wrong and the VM and router cannot reach each other (pings), since that ARP requests do not succeed>
on compute 1 where the router VM exists:
root@compute:~# ip n
fdca:3ba5:a17a:acda:f816:3eff:fefe:2f9e dev vxlan-77 lladdr fa:16:3e:fe:2f:9e PERMANENT
10.2.0.4 dev vxlan-33 lladdr fa:16:3e:28:d7:09 PERMANENT <--- correct entry for the VM
10.2.0.3 dev vxlan-33 lladdr fa:16:3e:fa:0f:b0 PERMANENT <--- I don't see this IP used anywhere
172.16.0.2 dev eth2 lladdr fa:16:3e:51:a5:5d STALE <--- the MAC address on the router VM for this IP is fa:16:3e:f7:8f:46
On the second compute node where the VM is located:
root@compute2:~# ip n
fe80::f816:3eff:fe66:476d dev vxlan-33 lladdr fa:16:3e:66:47:6d PERMANENT
10.2.0.3 dev vxlan-33 lladdr fa:16:3e:fa:0f:b0 PERMANENT <--- I don't see this IP used anywhere
no entry for the subnet interface(gateway) of 10.2.0.1 or for the router gateway of 172.16.0.2.
If I manually create the proper arp table entries to generate a local arp response using ip neighbour everything works.
Note - the VM must be on a different compute node from the router VM to see this problem.
Phil