metadata proxy won't start in dhcp namespace when network(subnet) is removed from router

Bug #1655605 reported by Perry on 2017-01-11
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
neutron
High
Unassigned

Bug Description

When adding network(subnet) into router immediately after creating network(subnet), there is no metadata proxy process created in dhcp namespace to listen on port 80. It causes problem when deleted network(subnet) from router: it won't call metadata service successfully until restarting dhcp service. Restarting dhcp service is just a workaround and is not acceptable as solution.

This problem is introduced in Newton release. When adding network, it will check whether the network has isolated ipv4 subnet. It queries all ports belonging to the network, and see whether there is any port used as gateway. if yes, then it thinks the subnet is not isolated. If we add subnet to router immediately after creating subnet, the process of network creation( creating metadata proxy) and the process of adding subnet to interface happens at the same time. The seconds process creates port as gateway quickly and then the first process checks and treats it no isolated, and then will kill metadata proxy created soon earlier.

# /etc/neutron/dhcp_agent.ini
enable_isolated_metadata = True
enable_metadata_network = True

#execute the following commands in batch without interruption.
neutron net-create network_1
neutron subnet-create --name subnet_1 network_1 172.16.255.0/24
neutron router-interface-add default subnet_1

# there is no 80 port.
 ip netns exec qdhcp-c5791b7d-ec3e-4e96-9a32-b9d1217ed330 netstat -tunlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 172.16.255.2:53         0.0.0.0:*               LISTEN      16926/dnsmasq
tcp        0      0 169.254.169.254:53      0.0.0.0:*               LISTEN      16926/dnsmasq
tcp6       0      0 fe80::f816:3eff:fe80:53 :::*                    LISTEN      16926/dnsmasq
udp        0      0 172.16.255.2:53         0.0.0.0:*                           16926/dnsmasq
udp        0      0 169.254.169.254:53      0.0.0.0:*                           16926/dnsmasq
udp        0      0 0.0.0.0:67              0.0.0.0:*                           16926/dnsmasq
udp6       0      0 :::547                  :::*                                16926/dnsmasq
udp6       0      0 fe80::f816:3eff:fe80:53 :::*                                16926/dnsmasq

Perry (panxia6679) on 2017-01-11
description: updated
Boden R (boden) wrote :

If there are any relevant log (snippets), please attach them.

Also, once this is fixed it'd be nice to have a functional test for the issue.

Changed in neutron:
status: New → Triaged
importance: Undecided → High
Perry (panxia6679) wrote :

I observed the following log fyi.

journalctl -l -u neutron-dhcp-agent.service -f

#create metadata proxy process 15975
Jan 11 05:16:07 perry1-controller-1 sudo[15977]: neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/neutron-rootwrap /etc/neutron/rootwrap.conf ip netns exec qdhcp-125373bc-aae2-4ff0-a484-b551ce557c94 neutron-ns-metadata-proxy --pid_file=/var/lib/neutron/external/pids/125373bc-aae2-4ff0-a484-b551ce557c94.pid --metadata_proxy_socket=/var/lib/neutron/metadata_proxy --network_id=125373bc-aae2-4ff0-a484-b551ce557c94 --state_path=/var/lib/neutron --metadata_port=80 --metadata_proxy_user=982 --metadata_proxy_group=979 --debug --log-file=neutron-ns-metadata-proxy-125373bc-aae2-4ff0-a484-b551ce557c94.log --log-dir=/var/log/neutron

Jan 11 05:16:08 perry1-controller-1 sudo[16000]: neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/neutron-rootwrap /etc/neutron/rootwrap.conf ip netns exec qdhcp-125373bc-aae2-4ff0-a484-b551ce557c94 ip addr show ns-8c9996eb-f7

#it checked and thoughts the subnet is not isolated, so it tried to kill the process of metadata proxy 15975
Jan 11 05:16:08 perry1-controller-1 sudo[16007]: neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/neutron-rootwrap /etc/neutron/rootwrap.conf kill -HUP 15975
Jan 11 05:16:09 perry1-controller-1 dnsmasq[15975]: read /var/lib/neutron/dhcp/125373bc-aae2-4ff0-a484-b551ce557c94/addn_hosts - 3 addresses
Jan 11 05:16:09 perry1-controller-1 dnsmasq-dhcp[15975]: read /var/lib/neutron/dhcp/125373bc-aae2-4ff0-a484-b551ce557c94/host
Jan 11 05:16:09 perry1-controller-1 dnsmasq-dhcp[15975]: read /var/lib/neutron/dhcp/125373bc-aae2-4ff0-a484-b551ce557c94/opts
Jan 11 05:16:09 perry1-controller-1 sudo[16013]: neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/neutron-rootwrap /etc/neutron/rootwrap.conf ip netns exec qdhcp-125373bc-aae2-4ff0-a484-b551ce557c94 ip route list dev ns-8c9996eb-f7
Jan 11 05:16:09 perry1-controller-1 sudo[16019]: neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/neutron-rootwrap /etc/neutron/rootwrap.conf kill -9 15999
Jan 11 05:16:09 perry1-controller-1 sudo[16023]: neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/neutron-rootwrap /etc/neutron/rootwrap.conf ip netns exec qdhcp-125373bc-aae2-4ff0-a484-b551ce557c94 ip addr show ns-8c9996eb-f7
Jan 11 05:16:10 perry1-controller-1 sudo[16028]: neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/neutron-rootwrap /etc/neutron/rootwrap.conf kill -HUP 15975
Jan 11 05:16:10 perry1-controller-1 dnsmasq[15975]: read /var/lib/neutron/dhcp/125373bc-aae2-4ff0-a484-b551ce557c94/addn_hosts - 3 addresses
Jan 11 05:16:10 perry1-controller-1 dnsmasq-dhcp[15975]: read /var/lib/neutron/dhcp/125373bc-aae2-4ff0-a484-b551ce557c94/host
Jan 11 05:16:10 perry1-controller-1 dnsmasq-dhcp[15975]: read /var/lib/neutron/dhcp/125373bc-aae2-4ff0-a484-b551ce557c94/opts

Perry (panxia6679) wrote :

The killing metadata process is handled by update_isolated_metadata_proxy. It was added to ensure to spawn metadata proxy process for network with ipv4 subnet added later after ipv6 subnet. However it also kills any metadata proxy process in dhcp namespace for network without isolated network. However the method doesn't get called when network(subnet) is deleted from router. As result, there is no metadata proxy in dhcp namespace to provide VM with metadata service in the scenario described in description. Please correct me if I am wrong. Thanks.

https://github.com/openstack/neutron/blob/bd193cba73bd4de41bd3a5c5a693f86797e56b8b/neutron/agent/dhcp/agent.py#L476-L488

Assaf Muller (amuller) wrote :

I believe this (Referring to the original bug report) never worked and this is essentially an RFE.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers