neutron L3-agent fail to start interfaces

Bug #1310926 reported by Andrew Woodward on 2014-04-22
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Critical
Unassigned
Mirantis OpenStack
Critical
Sergey Kolekonov
5.1.x
Critical
Sergey Kolekonov
6.0.x
Critical
Sergey Kolekonov

Bug Description

iso 120

deployed ubuntu neutron vlan
3 controller + 2 compute / ceph
ceph for images, volumes, instance, and object

after deployment found that neutron-l3-agent was running but no addresses where plumbed. Found that moving vip around would not bring up neutron routers. Finally was able to start router by un-managing neutron-l3-agent and running it on one host, but still was un-able to ping floating ip.

when agent wasn't starting it was usually raising tracebacks while attempting to plumb the net namespace

Logs are useless as they are nearly empty, I will have to pull some files by hand.

Andrew Woodward (xarses) wrote :
description: updated
Mike Scherbakov (mihgen) on 2014-04-22
Changed in fuel:
assignee: nobody → Fuel Library Team (fuel-library)
Vladimir Kuklin (vkuklin) wrote :

Andrew, this snapshot contains no logs at all, unfortunately. We cannot debug anything without it.

Vladimir Kuklin (vkuklin) wrote :

If it is snapshot-generation tool that is broken - please, pull the log files from master and slave nodes and attach them to the bug.

Changed in fuel:
status: New → Incomplete
Vladimir Kuklin (vkuklin) wrote :

Also, it may be a duplicate of https://bugs.launchpad.net/fuel/+bug/1309087 as it depends on newer upstart script for ovs-agent in Ubuntu.

Andrew Woodward (xarses) wrote :

There is no useful information in https://bugs.launchpad.net/fuel/+bug/1309087 to make it appear to be same.

Changed in fuel:
status: Incomplete → New
Andrew Woodward (xarses) wrote :

From /var/log/neutron/l3-agent.log

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/eventlet/greenpool.py", line 80, in _spawn_n_impl
    func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/neutron/agent/l3_agent.py", line 438, in process_router
    p['ip_cidr'], p['mac_address'])
  File "/usr/lib/python2.7/dist-packages/neutron/agent/l3_agent.py", line 707, in internal_network_added
    prefix=INTERNAL_DEV_PREFIX)
  File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/interface.py", line 195, in plug
    namespace_obj = ip.ensure_namespace(namespace)
  File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 136, in ensure_namespace
    ip = self.netns.add(name)
  File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 446, in add
    self._as_root('add', name, use_root_namespace=True)
  File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 217, in _as_root
    kwargs.get('use_root_namespace', False))
  File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 70, in _as_root
    namespace)
  File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 81, in _execute
    root_helper=root_helper)
  File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 76, in execute
    raise RuntimeError(m)
RuntimeError:
Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'add', 'qrouter-27429c22-0ec8-46dc-83e2-13baf13e6241']
Exit code: 1
Stdout: ''
Stderr: 'Could not create /var/run/netns/qrouter-27429c22-0ec8-46dc-83e2-13baf13e6241: File exists\n'

Andrew Woodward (xarses) wrote :
Vladimir Kuklin (vkuklin) wrote :

Andrew, the issue I referenced was related to OVS agent not creating ports due to agent not functioning on Ubuntu nodes.

Changed in fuel:
status: New → Confirmed
summary: - ubuntu neutron fails to start l3-agent
+ neutron agents fail to start interfaces
Changed in fuel:
importance: High → Critical
summary: - neutron agents fail to start interfaces
+ neutron L3-agent fail to start interfaces
Sergey Vasilenko (xenolog) wrote :

Neutron call 'ip netns list' without using rootwrap:

2014-04-23 16:15:43.806 28240 DEBUG neutron.agent.linux.utils [req-d0f812f6-d987-45f5-9cff-11f1fa52fed6 None] Running command: ['ip', '-o', 'netns', 'list'] create_process /usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py:48
2014-04-23 16:15:43.828 28240 DEBUG neutron.agent.linux.utils [req-d0f812f6-d987-45f5-9cff-11f1fa52fed6 None]
Command: ['ip', '-o', 'netns', 'list']
Exit code: 0
Stdout: ''
Stderr: '' execute /usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py:74
2014-04-23 16:15:43.829 28240 DEBUG neutron.agent.linux.utils [req-d0f812f6-d987-45f5-9cff-11f1fa52fed6 None] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'add', 'qrouter-b582586e-70e3-4a38-8b19-039f30ce87a9'] create_process /usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py:48
2014-04-23 16:15:43.909 28240 DEBUG neutron.agent.linux.utils [req-d0f812f6-d987-45f5-9cff-11f1fa52fed6 None]
Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'add', 'qrouter-b582586e-70e3-4a38-8b19-039f30ce87a9']
Exit code: 0
Stdout: ''

Sergey Vasilenko (xenolog) wrote :

HOTFIX (CENTOS):
# vim /usr/lib/python2.6/site-packages/neutron/agent/linux/ip_lib.py
line #469:
    def exists(self, name):
        output = self._parent._execute('o', 'netns', ['list'])

Should be changed to:
    def exists(self, name):
        output = self._parent._execute('o', 'netns', ['list'], root_helper=self._parent.root_helper)

Changed in fuel:
status: Confirmed → In Progress

Reviewed: https://review.openstack.org/89872
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=c0e3d76a8a96d17d2f20695a36d170824cd581ca
Submitter: Jenkins
Branch: master

commit c0e3d76a8a96d17d2f20695a36d170824cd581ca
Author: Sergey Vasilenko <email address hidden>
Date: Wed Apr 23 19:02:41 2014 +0400

    kill all proceses inside all l3-agent's net.namespaces,

    that using ip protocol. When l3 agent start/stop.
    Also remove unnided namespaces.

    Change-Id: I7e5203f6cec646a8b70609f2bf3b785066068aeb
    Closes-Bug: #1310926

Changed in fuel:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/90221
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=1e290c398712683566a11c78ceea532719e1f72e
Submitter: Jenkins
Branch: stable/4.1

commit 1e290c398712683566a11c78ceea532719e1f72e
Author: Sergey Vasilenko <email address hidden>
Date: Wed Apr 23 19:02:41 2014 +0400

    kill all proceses inside all l3-agent's net.namespaces,

    that using ip protocol. When l3 agent start/stop.
    Also remove unneeded namespaces.

    Change-Id: I7e5203f6cec646a8b70609f2bf3b785066068aeb
    Closes-Bug: #1310926

Andrew Woodward (xarses) wrote :

This commit was applied to the wrong bug.

Changed in fuel:
status: Fix Committed → In Progress
status: In Progress → Confirmed
Vladimir Kuklin (vkuklin) wrote :

all the commits for this bug were successfully merged. moving this bug to 4.1.1

Changed in fuel:
milestone: 5.0 → 4.1.1
Andrew Woodward (xarses) wrote :
Changed in fuel:
milestone: 4.1.1 → 5.0
assignee: Fuel Library Team (fuel-library) → Fuel OSCI Team (fuel-osci)
Vladimir Kuklin (vkuklin) wrote :

this fix is already merged into FUEL packages

Changed in fuel:
status: Confirmed → Fix Committed
milestone: 5.0 → 4.1.1
Nastya Urlapova (aurlapova) wrote :

{
build_id: "2014-06-05_15-25-50",
mirantis: "yes",
build_number: "344",
nailgun_sha: "a828d6b7610f872980d5a2113774f1cda6f6810b",
ostf_sha: "2b7b39e4b6ea89751b65171f24a8e80b5cac56aa",
fuelmain_sha: "9964da7dec34d3100419c1c77c8f5235d8e30f14",
astute_sha: "55df06b2e84fa5d71a1cc0e78dbccab5db29d968",
release: "4.1B",
fuellib_sha: "3511461a2b529619a787a6306441d9039699e71d"
}

Changed in fuel:
status: Fix Committed → Fix Released
Changed in mos:
status: New → In Progress
Alexander Ignatov (aignatov) wrote :

Change is merged to 6.0 in the following patch: https://gerrit.mirantis.com/#/c/29630/

OSCI Robot (oscirobot) wrote :

Package neutron has been built from changeset: https://review.fuel-infra.org/92
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0-stable-92/centos

OSCI Robot (oscirobot) wrote :

Package neutron has been built from changeset: https://review.fuel-infra.org/92
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0-stable-92/ubuntu

OSCI Robot (oscirobot) wrote :

Package neutron has been built from changeset: https://review.fuel-infra.org/92
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0-stable/centos

OSCI Robot (oscirobot) wrote :

Package neutron has been built from changeset: https://review.fuel-infra.org/92
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0-stable/ubuntu

Verify 6.0
iso
 {"build_id": "2014-12-26_14-25-46", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "58", "auth_required": true, "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "81d38d6f2903b5a8b4bee79ca45a54b76c1361b8", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-12-26_14-25-46", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "58", "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "81d38d6f2903b5a8b4bee79ca45a54b76c1361b8", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "fde8ba5e11a1acaf819d402c645c731af450aff0"}}}, "fuellib_sha": "fde8ba5e11a1acaf819d402c645c731af450aff0"}

ubuntu neutron vlan, 3 controller + 2 compute / ceph

Steps to reproduce:
1) Create new net, subnet, router
2) lounch instance on this new net
2) associate floating IP to the VM
3) in CLi ping floating IP
4) go with ssh to the VM

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Sergey Vasilenko <email address hidden>
Review: https://review.fuel-infra.org/7996

Change abandoned by Oleg Bondarev <email address hidden> on branch: openstack-ci/fuel-7.0/2015.1.0
Review: https://review.fuel-infra.org/7996
Reason: Not needed now as from Kilo there is an option to use root rights when listing namespaces, True by default use_helper_for_ns_read

Farkhad (fkhatamov) on 2016-03-14
information type: Public → Public Security
information type: Public Security → Public
Curtis Hovey (sinzui) on 2016-07-29
Changed in fuel:
assignee: Registry Administrators (registry) → nobody
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers