| 2014-09-01 02:59:38 |
Han Zhou |
description |
When running in virtual env, neutron-ns-metadata-proxy on DHCP node cannot be disabled properly when deleting a network.
Consequences:
1. Corresponding folder and files under /var/lib/neutron/dhcp/ are not cleaned up.
2. If neutron-dhcp-agent service is restarted in this unclean state, dnsmasq processes are restarted periodically (resync interval), because the failure of removing the deleted networks. (And because of bug 1345947, huge amount of NAK logs flood for dnsmasq.)
Root cause:
Killing neutron-ns-metadata-proxy is failed because of rootwrapper filter not matched in dhcp_filter.conf:
Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'kill', '-9', '65832']
Exit code: 99
Stdout: ''
Stderr: '/usr/bin/neutron-rootwrap: Unauthorized command: kill -9 65832 (no filter matched)\n'
neutron-ns-metadata-proxy is started via ProcessManager, and in Virtual Environment, the python binary under virtual env is used instead of /usr/bin/python*.
However, in dhcp_file.conf absolute path is specified for KillerFiler for neutron-ns-metadata-proxy:
kill_metadata: KillFilter, root, /usr/bin/python, -9
kill_metadata7: KillFilter, root, /usr/bin/python2.7, -9
kill_metadata6: KillFilter, root, /usr/bin/python2.6, -9
Proposed solution:
Do not specify absolute path in filter: use python (or python2.7, or python2.6) instead
============================================================================
diff --git a/etc/neutron/rootwrap.d/dhcp.filters b/etc/neutron/rootwrap.d/dhcp.filters
index 88d61e8..26c2ffa 100644
--- a/etc/neutron/rootwrap.d/dhcp.filters
+++ b/etc/neutron/rootwrap.d/dhcp.filters
@@ -29,9 +29,9 @@ metadata_proxy_quantum: CommandFilter, quantum-ns-metadata-proxy, root
metadata_proxy_local: CommandFilter, /usr/local/bin/neutron-ns-metadata-proxy, root
metadata_proxy_local_quantum: CommandFilter, /usr/local/bin/quantum-ns-metadata-proxy, root
# RHEL invocation of the metadata proxy will report /usr/bin/python
-kill_metadata: KillFilter, root, /usr/bin/python, -9
-kill_metadata7: KillFilter, root, /usr/bin/python2.7, -9
-kill_metadata6: KillFilter, root, /usr/bin/python2.6, -9
+kill_metadata: KillFilter, root, python, -9
+kill_metadata7: KillFilter, root, python2.7, -9
+kill_metadata6: KillFilter, root, python2.6, -9
# ip_lib
ip: IpFilter, ip, root |
== Problem ==
When running in virtual env, neutron-ns-metadata-proxy on DHCP node cannot be disabled properly when deleting a network.
== Impact ==
* Corresponding folder and files under /var/lib/neutron/dhcp/ are not cleaned up.
* If neutron-dhcp-agent service is restarted in this unclean state, dnsmasq processes are restarted periodically (resync interval), because the failure of removing the deleted networks. (And because of bug 1345947, huge amount of NAK logs flood for dnsmasq.)
== Root cause ==
Killing neutron-ns-metadata-proxy is failed because of rootwrapper filter not matched in dhcp_filter.conf. Related logs are:
{{{
Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'kill', '-9', '65832']
Exit code: 99
Stdout: ''
Stderr: '/usr/bin/neutron-rootwrap: Unauthorized command: kill -9 65832 (no filter matched)\n'
}}}
neutron-ns-metadata-proxy is started via ProcessManager, and in Virtual Environment, the python binary under virtual env is used instead of /usr/bin/python*.
However, in dhcp_file.conf absolute path is specified for KillerFiler for neutron-ns-metadata-proxy:
{{{
kill_metadata: KillFilter, root, /usr/bin/python, -9
kill_metadata7: KillFilter, root, /usr/bin/python2.7, -9
kill_metadata6: KillFilter, root, /usr/bin/python2.6, -9
}}}
== Proposed solution ==
Do not specify absolute path in filter: use python (or python2.7, or python2.6) instead
======================================================================
{{{
diff --git a/etc/neutron/rootwrap.d/dhcp.filters b/etc/neutron/rootwrap.d/dhcp.filters
index 88d61e8..26c2ffa 100644
--- a/etc/neutron/rootwrap.d/dhcp.filters
+++ b/etc/neutron/rootwrap.d/dhcp.filters
@@ -29,9 +29,9 @@ metadata_proxy_quantum: CommandFilter, quantum-ns-metadata-proxy, root
metadata_proxy_local: CommandFilter, /usr/local/bin/neutron-ns-metadata-proxy, root
metadata_proxy_local_quantum: CommandFilter, /usr/local/bin/quantum-ns-metadata-proxy, root
# RHEL invocation of the metadata proxy will report /usr/bin/python
-kill_metadata: KillFilter, root, /usr/bin/python, -9
-kill_metadata7: KillFilter, root, /usr/bin/python2.7, -9
-kill_metadata6: KillFilter, root, /usr/bin/python2.6, -9
+kill_metadata: KillFilter, root, python, -9
+kill_metadata7: KillFilter, root, python2.7, -9
+kill_metadata6: KillFilter, root, python2.6, -9
# ip_lib
ip: IpFilter, ip, root
}}} |
|
| 2014-09-01 03:04:16 |
Han Zhou |
description |
== Problem ==
When running in virtual env, neutron-ns-metadata-proxy on DHCP node cannot be disabled properly when deleting a network.
== Impact ==
* Corresponding folder and files under /var/lib/neutron/dhcp/ are not cleaned up.
* If neutron-dhcp-agent service is restarted in this unclean state, dnsmasq processes are restarted periodically (resync interval), because the failure of removing the deleted networks. (And because of bug 1345947, huge amount of NAK logs flood for dnsmasq.)
== Root cause ==
Killing neutron-ns-metadata-proxy is failed because of rootwrapper filter not matched in dhcp_filter.conf. Related logs are:
{{{
Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'kill', '-9', '65832']
Exit code: 99
Stdout: ''
Stderr: '/usr/bin/neutron-rootwrap: Unauthorized command: kill -9 65832 (no filter matched)\n'
}}}
neutron-ns-metadata-proxy is started via ProcessManager, and in Virtual Environment, the python binary under virtual env is used instead of /usr/bin/python*.
However, in dhcp_file.conf absolute path is specified for KillerFiler for neutron-ns-metadata-proxy:
{{{
kill_metadata: KillFilter, root, /usr/bin/python, -9
kill_metadata7: KillFilter, root, /usr/bin/python2.7, -9
kill_metadata6: KillFilter, root, /usr/bin/python2.6, -9
}}}
== Proposed solution ==
Do not specify absolute path in filter: use python (or python2.7, or python2.6) instead
======================================================================
{{{
diff --git a/etc/neutron/rootwrap.d/dhcp.filters b/etc/neutron/rootwrap.d/dhcp.filters
index 88d61e8..26c2ffa 100644
--- a/etc/neutron/rootwrap.d/dhcp.filters
+++ b/etc/neutron/rootwrap.d/dhcp.filters
@@ -29,9 +29,9 @@ metadata_proxy_quantum: CommandFilter, quantum-ns-metadata-proxy, root
metadata_proxy_local: CommandFilter, /usr/local/bin/neutron-ns-metadata-proxy, root
metadata_proxy_local_quantum: CommandFilter, /usr/local/bin/quantum-ns-metadata-proxy, root
# RHEL invocation of the metadata proxy will report /usr/bin/python
-kill_metadata: KillFilter, root, /usr/bin/python, -9
-kill_metadata7: KillFilter, root, /usr/bin/python2.7, -9
-kill_metadata6: KillFilter, root, /usr/bin/python2.6, -9
+kill_metadata: KillFilter, root, python, -9
+kill_metadata7: KillFilter, root, python2.7, -9
+kill_metadata6: KillFilter, root, python2.6, -9
# ip_lib
ip: IpFilter, ip, root
}}} |
== Problem ==
When running in virtual env, neutron-ns-metadata-proxy on DHCP node cannot be disabled properly when deleting a network.
== Impact ==
* Corresponding folder and files under /var/lib/neutron/dhcp/ are not cleaned up.
* If neutron-dhcp-agent service is restarted in this unclean state, dnsmasq processes are restarted periodically (resync interval), because the failure of removing the deleted networks. (And because of bug 1345947, huge amount of NAK logs flood for dnsmasq.)
== Root cause ==
Killing neutron-ns-metadata-proxy is failed because of rootwrapper filter not matched in dhcp_filter.conf. Related logs are:
Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'kill', '-9', '65832']
Exit code: 99
Stdout: ''
Stderr: '/usr/bin/neutron-rootwrap: Unauthorized command: kill -9 65832 (no filter matched)\n'
neutron-ns-metadata-proxy is started via ProcessManager, and in Virtual Environment, the python binary under virtual env is used instead of /usr/bin/python*.
However, in dhcp_file.conf absolute path is specified for KillerFiler for neutron-ns-metadata-proxy:
kill_metadata: KillFilter, root, /usr/bin/python, -9
kill_metadata7: KillFilter, root, /usr/bin/python2.7, -9
kill_metadata6: KillFilter, root, /usr/bin/python2.6, -9
== Proposed solution ==
Do not specify absolute path in filter: use python (or python2.7, or python2.6) instead
======================================================================
diff --git a/etc/neutron/rootwrap.d/dhcp.filters b/etc/neutron/rootwrap.d/dhcp.filters
index 88d61e8..26c2ffa 100644
--- a/etc/neutron/rootwrap.d/dhcp.filters
+++ b/etc/neutron/rootwrap.d/dhcp.filters
@@ -29,9 +29,9 @@ metadata_proxy_quantum: CommandFilter, quantum-ns-metadata-proxy, root
metadata_proxy_local: CommandFilter, /usr/local/bin/neutron-ns-metadata-proxy, root
metadata_proxy_local_quantum: CommandFilter, /usr/local/bin/quantum-ns-metadata-proxy, root
# RHEL invocation of the metadata proxy will report /usr/bin/python
-kill_metadata: KillFilter, root, /usr/bin/python, -9
-kill_metadata7: KillFilter, root, /usr/bin/python2.7, -9
-kill_metadata6: KillFilter, root, /usr/bin/python2.6, -9
+kill_metadata: KillFilter, root, python, -9
+kill_metadata7: KillFilter, root, python2.7, -9
+kill_metadata6: KillFilter, root, python2.6, -9
# ip_lib
ip: IpFilter, ip, root |
|
