neutron

Metering agent: failed to run ip netns command

Bug #1786213 reported by Dongcan Ye on 2018-08-09

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	neutron	Fix Released	High	Dongcan Ye

Bug Description

While neutron-metering-agent start, it fails as following:
2018-08-09 03:15:08.504 10637 INFO oslo.privsep.daemon [req-6dd24e96-e82b-49a5-9d33-80e1ff572502 - - - - -] Running privsep helper: ['sudo', 'privsep-helper', '--config-file', '/usr/share/neutron/neutron-dist.conf', '--config-file', '/etc/neutron/neutron.conf', '--config-file', '/etc/neutron/metering_agent.ini', '--config-dir', '/etc/neutron/conf.d/neutron-metering-agent', '--privsep_context', 'neutron.privileged.default', '--privsep_sock_path', '/tmp/tmp_JYWYs/privsep.sock']
2018-08-09 03:15:08.525 10637 WARNING oslo.privsep.daemon [-] privsep log:
2018-08-09 03:15:08.526 10637 WARNING oslo.privsep.daemon [-] privsep log: We trust you have received the usual lecture from the local System
2018-08-09 03:15:08.527 10637 WARNING oslo.privsep.daemon [-] privsep log: Administrator. It usually boils down to these three things:
2018-08-09 03:15:08.527 10637 WARNING oslo.privsep.daemon [-] privsep log:
2018-08-09 03:15:08.528 10637 WARNING oslo.privsep.daemon [-] privsep log: #1) Respect the privacy of others.
2018-08-09 03:15:08.528 10637 WARNING oslo.privsep.daemon [-] privsep log: #2) Think before you type.
2018-08-09 03:15:08.528 10637 WARNING oslo.privsep.daemon [-] privsep log: #3) With great power comes great responsibility.
2018-08-09 03:15:08.529 10637 WARNING oslo.privsep.daemon [-] privsep log:
2018-08-09 03:15:08.531 10637 WARNING oslo.privsep.daemon [-] privsep log: sudo: no tty present and no askpass program specified
2018-08-09 03:15:08.544 10637 CRITICAL oslo.privsep.daemon [req-6dd24e96-e82b-49a5-9d33-80e1ff572502 - - - - -] privsep helper command exited non-zero (1)

2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task [req-6dd24e96-e82b-49a5-9d33-80e1ff572502 - - - - -] Error during MeteringAgentWithStateReport._sync_routers_task: FailedToDropPrivileges: privsep helper command exited non-zero (1)
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task Traceback (most recent call last):
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task task(self, context)
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/neutron/services/metering/agents/metering_agent.py", line 189, in _sync_routers_task
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task self._update_routers(context, routers)
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/neutron/services/metering/agents/metering_agent.py", line 212, in _update_routers
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task 'update_routers')
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task return f(*args, **kwargs)
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/neutron/services/metering/agents/metering_agent.py", line 166, in _invoke_driver
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task return getattr(self.metering_driver, func_name)(context, meterings)
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_log/helpers.py", line 67, in wrapper
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task return method(*args, **kwargs)
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/neutron/services/metering/drivers/iptables/iptables_driver.py", line 151, in update_routers
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task self._process_associate_metering_label(router)
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/neutron/services/metering/drivers/iptables/iptables_driver.py", line 249, in _process_associate_metering_label
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task self._update_router(router)
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/neutron/services/metering/drivers/iptables/iptables_driver.py", line 118, in _update_router
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task RouterWithMetering(self.conf, router))
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/neutron/services/metering/drivers/iptables/iptables_driver.py", line 94, in __init__
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task if ip_lib.network_namespace_exists(self.ns_name):
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 1057, in network_namespace_exists
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task output = list_network_namespaces(**kwargs)
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 1046, in list_network_namespaces
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task return privileged.list_netns(**kwargs)
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 206, in _wrap
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task self.start()
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 217, in start
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task channel = daemon.RootwrapClientChannel(context=self)
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 327, in __init__
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task raise FailedToDropPrivileges(msg)
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task FailedToDropPrivileges: privsep helper command exited non-zero (1)
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task

It seems missing setup_privsep() call in metering agent init.

Tags:

Dongcan Ye (hellochosen) on 2018-08-09

Changed in neutron:
assignee:	nobody → Dongcan Ye (hellochosen)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-08-09: Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/590215

Changed in neutron:
status:	New → In Progress

Brian Haley (brian-haley) on 2018-08-09

Changed in neutron:
importance:	Undecided → High

Bernard Cafarelli (bcafarel) on 2018-08-09

summary:

- Metering agent: failed to running ip netns commad
+ Metering agent: failed to run ip netns command

Bernard Cafarelli (bcafarel) on 2018-08-10

tags:

added: metering

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-08-10: Fix merged to neutron (master)

Reviewed: https://review.openstack.org/590215
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=40d92d40ec0953f82a040c2197304934efa3e9e1
Submitter: Zuul
Branch: master

commit 40d92d40ec0953f82a040c2197304934efa3e9e1
Author: Dongcan Ye <email address hidden>
Date: Thu Aug 9 10:05:36 2018 +0000

Config privsep in the metering agent

Enable privsep to execute ip netns or other commands.

Change-Id: I4e20a1e92c0d154b76615437efe5eced4e0cc6bd
Closes-Bug: #1786213

Changed in neutron:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-08-16: Fix proposed to neutron (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/592353

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-08-16: Fix proposed to neutron (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/592354

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-08-16: Fix proposed to neutron (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/592355

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-08-17: Change abandoned on neutron (stable/pike)

Change abandoned by Dongcan Ye (<email address hidden>) on branch: stable/pike
Review: https://review.openstack.org/592355

Bernard Cafarelli (bcafarel) on 2018-08-17

tags:	added: neutron-proactive-backport-potential
tags:	added: neutron-easy-proactive-backport-potential

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-08-23: Fix merged to neutron (stable/rocky)

Reviewed: https://review.openstack.org/592353
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=811cb63b2121a0d54f7cbf244032966b0ba3cf15
Submitter: Zuul
Branch: stable/rocky

commit 811cb63b2121a0d54f7cbf244032966b0ba3cf15
Author: Dongcan Ye <email address hidden>
Date: Thu Aug 9 10:05:36 2018 +0000

Config privsep in the metering agent

Enable privsep to execute ip netns or other commands.

    Change-Id: I4e20a1e92c0d154b76615437efe5eced4e0cc6bd
    Closes-Bug: #1786213
    (cherry picked from commit 40d92d40ec0953f82a040c2197304934efa3e9e1)