neutron metadata- after adding a network port to a neutron router, static route is still sent to new VMs

[Alex Stafeyev]

 Metaadata isolated configured. Network and subnet created - ( no --no-gateway option but the network is not connected to a router - there is no router). VM booted - metadata is OK and we have a static route passed to the VM.
Created a router and attached a network port to it ( same network) . Created a new VM and still qdhcp will handle the metadata even though we have the network configured with DG ( /var/lib/neutron/dhcp/XYZ/opts has a static route for 169.254.169.254 and passes it to the new VMs- This is OK ONLY before we added the network to the router) . After dhcp agent reboot the route to qdhcp is removed from /var/lib/neutron/dhcp/XYZ/opts and router will handle metadata.

I strongly believe that if we made no change in the dhcp-agent.ini file, agent restart ( systemctl restart neutron-dhcp-agent) should not affect behavior in any way.

Additionally,
The static route should be removed from /var/lib/neutron/dhcp/XYZ/opts right after the network "XYZ" was added to the router.

BR
Alex

Version- Kilo

[songqianxia (songlihui1947)]

I think it is because the sequence of the static metadata route and the network route.
Please remove the route in metadata, if the network has been set route info.

[Alex Stafeyev (astafeye)]

If I reboot the dhcp agent the static route is not sent to the newly booted VM. This is a way to bypass this issue. May be several. But this is not a fix for the scenario.

The qdhcp should remove the static route to 169.254.169.254 from /var/lib/neutron/dhcp/XYZ/opts and should not send it to the VMs after the network was added to a router.

For the exeisting VMs prior adding the network to the router we need to remove the static route from the ip route table ( not critical because qdhcp can take care of its metedata requests). What bothers is me is that the VMs which created after adding the network to the router are updated by qdhcp with the static route

[John Schwarz (jschwarz)]

What Alex is looking for is that the behaviour before and after restarting the dhcp-agent process will remain the same (because there was no change in configuration).

This is currently not the case - since neutron-server doesn't notify the dhcp-agent whenever it attaches/detaches a router from a previously-isolated network, this is discovered when the dhcp-agent is restarted but not when it actually happens. What we need is some event sent from neutron-server to dhcp-agent on such an occasion.

[John Schwarz (jschwarz)]

@Mohammed, Please only assign to yourself bugs you actually intend to fix. Looking at the list of your assigned bugs [1] you have 16 different bugs assigned to you, but not a single patch under your user. Over-assigning bugs to yourself (without fixing them) means bugs don't get triaged or fixed at all.

[Sreenivas (sreenivas-pothukanoori)]

Alex, I am trying to reproduce the issue but I have the following queries. Could you please check answer to the below queries.

Created a router and attached a network port to it ( same network) .

Q: When same network with no gateway option is attached to router, What is the value given for the interface that is connected to the router. I am using horizon to attach same subnet to router, however add interface fails as the subnet configured did not have a gateway IP.

Q. Do I have to modify the subnet with gateway option before attaching it to the subnet? or I can assign any IP from subnet with no gateway option?

My observation is the subnet has a gateway in dhcp namespace. If a IP from subnet range is configured at router interface. DHCP would never come to know about this gateway and always the default route at VM would poing to gateway in dhcp name space.
Ex
The subnet I created is 172.17.17.0/24. DHCP interface has 172.17.17.1. This is the gateway at VM. However if I cannot give the same gateway for router interface.

With a different IP (same subnet)configured as router interface, even with dhcp agent restart static route still remains in opts.

Could you also please provide the exact steps to reproduce the issue with configuration changes.

[Alex Stafeyev (astafeye)]

Hi
answer to :
Q: When same network with no gateway option is attached to router, What is the value given for the interface that is connected to the router. I am using horizon to attach same subnet to router, however add interface fails as the subnet configured did not have a gateway IP.

Usually the first address in the created network is reserved for the GW and will be after the network is attached to a router.
When you executed with " no-gateway" option the address is not reserved. This is because attaching a "no gateway" network is negative testing action.
In this bug I AM NOT USING "no-gateway" option .

answer to :
Q. Do I have to modify the subnet with gateway option before attaching it to the subnet? or I can assign any IP from subnet with no gateway option?

I prefer you to create a subnet without "no gateway" option from the beginning.

Those 2 answer should also answer the example u gave.
The VM which were created prior adding a router, will use the dhcp agent ( due to its static route in routing table) but vms which created after adding a router should not have that static route and use the router for metadata requests and for exiting the network.

The steps:
Create net and subnet ( no "no gateway" option needed). See option for the subnet- the static route will be there.
After adding the router the static route should be removed from options ( after dhcp agent restart ) . The bug is that it is still there.

I hope I was helpful
BR
Alex

[Hong Hui Xiao (xiaohhui)]

I am going to close this bug, because the behavior in the description

"After dhcp agent reboot the route to qdhcp is removed from /var/lib/neutron/dhcp/XYZ/opts and router will handle metadata."

is fixed by this bug: bug 1554825.

With the patch in that bug, when router interface is added/deleted, the dhcp agent will update the route of metadata service. So, no need to restart the dhcp agent now, the router will take/leave metadata service.

[Alex Stafeyev (astafeye)]

Cool tnx

[Sreenivas (sreenivas-pothukanoori)]

Hong Hui,
I was able to reproduce this issue without your patch on kilo version. Could you please let me know how to get the patch(bug 1554825) to test the fix on my environment(Kilo version) and confirm it.?

[Sreenivas (sreenivas-pothukanoori)]

Today, I tried to reproduce this issue on Liberty version, but I have observed different behavior.

After adding network interface to the router, the static entry from /var/lib/neutron/dhcp/XYZ/opts is not removing before and after restarting the dhcp-agent service. Is it a new issue?.

do you have any plans to fix it, please confirm.

[Anseela M M (anseela-m00)]

I have created test case for verifying the bug. Please execute these test case once the fix is done.

Test case id: OPNST_Neutron Bug: neutron metadata- after adding a network port to a neutron router, static route is still sent to new VMs
Metric: 30 minutes
Test purpose: neutron metadata- after adding a network port to a neutron router, static route is still sent to new VMs
Configuration: An OpenStack environment with at least one controller (with keystone, glance), one compute node and one network node installed.
Test tool: tempest
References: Bug_ 1549860: neutron metadata- after adding a network port to a neutron router, static route is still sent to new VMs
Applicability: Test can be adding a network port to a neutron router, static route is still sent to new VMs
Pre-test conditions: Openstack should be installed and all the services up and running
Step Description:
1. Create an instance
2. set enable_isolated_metadata = True in /etc/quantum/dhcp_agent.ini
3. create n/w without no-gateway option
4. check the opts file in /var/lib/neutron/dhcp/XYZ/opts
5. create a router and attach the network which u have created in previous step to the router
6. check the opts file in /var/lib/neutron/dhcp/XYZ/opts
7. restart the dhcp-agent
Result:
1. User should be able to create an instance and the instance should be in active state
2. User should be able to set the value
3. User should be able to create a network
4. Opts file should have the network entry
5. User should be able to create the router and able to attach the n/w to the router
6. Entry should be removed from the opts file
7. Dhcp-agent restart should not have any impact
Test verdict: Test case pass only if the instance should communicate to the metadata through qrouter once the router added to the network

[Sreenivas (sreenivas-pothukanoori)]

While working on this bug, “Hong Hui Xiao” has marked it as duplicate of 1554825 which is in Fix Released state.

Could you please let me know to which status I have to set this bug.

Also, please provide the patch for bug #1554825 . We need the patch to check and confirm that this patch has resolved the issue reported in the 1549860.

[John Schwarz (jschwarz)]

Sreenivas, I think https://review.openstack.org/#/c/290216/ is the patch that resolved bug #1554825.