neutron L3-agent fail to start interfaces

[Andrew Woodward]

iso 120

deployed ubuntu neutron vlan
3 controller + 2 compute / ceph
ceph for images, volumes, instance, and object

after deployment found that neutron-l3-agent was running but no addresses where plumbed. Found that moving vip around would not bring up neutron routers. Finally was able to start router by un-managing neutron-l3-agent and running it on one host, but still was un-able to ping floating ip.

when agent wasn't starting it was usually raising tracebacks while attempting to plumb the net namespace

Logs are useless as they are nearly empty, I will have to pull some files by hand.

[Vladimir Kuklin (vkuklin)]

Andrew, this snapshot contains no logs at all, unfortunately. We cannot debug anything without it.

[Vladimir Kuklin (vkuklin)]

If it is snapshot-generation tool that is broken - please, pull the log files from master and slave nodes and attach them to the bug.

[Vladimir Kuklin (vkuklin)]

Also, it may be a duplicate of https://bugs.launchpad.net/fuel/+bug/1309087 as it depends on newer upstart script for ovs-agent in Ubuntu.

[Andrew Woodward (xarses)]

There is no useful information in https://bugs.launchpad.net/fuel/+bug/1309087 to make it appear to be same.

[Andrew Woodward (xarses)]

From /var/log/neutron/l3-agent.log

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/eventlet/greenpool.py", line 80, in _spawn_n_impl
    func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/neutron/agent/l3_agent.py", line 438, in process_router
    p['ip_cidr'], p['mac_address'])
  File "/usr/lib/python2.7/dist-packages/neutron/agent/l3_agent.py", line 707, in internal_network_added
    prefix=INTERNAL_DEV_PREFIX)
  File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/interface.py", line 195, in plug
    namespace_obj = ip.ensure_namespace(namespace)
  File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 136, in ensure_namespace
    ip = self.netns.add(name)
  File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 446, in add
    self._as_root('add', name, use_root_namespace=True)
  File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 217, in _as_root
    kwargs.get('use_root_namespace', False))
  File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 70, in _as_root
    namespace)
  File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 81, in _execute
    root_helper=root_helper)
  File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 76, in execute
    raise RuntimeError(m)
RuntimeError:
Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'add', 'qrouter-27429c22-0ec8-46dc-83e2-13baf13e6241']
Exit code: 1
Stdout: ''
Stderr: 'Could not create /var/run/netns/qrouter-27429c22-0ec8-46dc-83e2-13baf13e6241: File exists\n'

[Vladimir Kuklin (vkuklin)]

Andrew, the issue I referenced was related to OVS agent not creating ports due to agent not functioning on Ubuntu nodes.

[Sergey Vasilenko (xenolog)]

Neutron call 'ip netns list' without using rootwrap:

2014-04-23 16:15:43.806 28240 DEBUG neutron.agent.linux.utils [req-d0f812f6-d987-45f5-9cff-11f1fa52fed6 None] Running command: ['ip', '-o', 'netns', 'list'] create_process /usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py:48
2014-04-23 16:15:43.828 28240 DEBUG neutron.agent.linux.utils [req-d0f812f6-d987-45f5-9cff-11f1fa52fed6 None]
Command: ['ip', '-o', 'netns', 'list']
Exit code: 0
Stdout: ''
Stderr: '' execute /usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py:74
2014-04-23 16:15:43.829 28240 DEBUG neutron.agent.linux.utils [req-d0f812f6-d987-45f5-9cff-11f1fa52fed6 None] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'add', 'qrouter-b582586e-70e3-4a38-8b19-039f30ce87a9'] create_process /usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py:48
2014-04-23 16:15:43.909 28240 DEBUG neutron.agent.linux.utils [req-d0f812f6-d987-45f5-9cff-11f1fa52fed6 None]
Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'add', 'qrouter-b582586e-70e3-4a38-8b19-039f30ce87a9']
Exit code: 0
Stdout: ''

[Sergey Vasilenko (xenolog)]

HOTFIX (CENTOS):
# vim /usr/lib/python2.6/site-packages/neutron/agent/linux/ip_lib.py
line #469:
    def exists(self, name):
        output = self._parent._execute('o', 'netns', ['list'])

Should be changed to:
    def exists(self, name):
        output = self._parent._execute('o', 'netns', ['list'], root_helper=self._parent.root_helper)

[Sergey Vasilenko (xenolog)]

Bug in Neutron: https://bugs.launchpad.net/neutron/+bug/1311804

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/89872
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=c0e3d76a8a96d17d2f20695a36d170824cd581ca
Submitter: Jenkins
Branch: master

commit c0e3d76a8a96d17d2f20695a36d170824cd581ca
Author: Sergey Vasilenko <email address hidden>
Date: Wed Apr 23 19:02:41 2014 +0400

    kill all proceses inside all l3-agent's net.namespaces,

    that using ip protocol. When l3 agent start/stop.
    Also remove unnided namespaces.

    Change-Id: I7e5203f6cec646a8b70609f2bf3b785066068aeb
    Closes-Bug: #1310926

[Openstack Gerrit (openstack-gerrit)]

Fix proposed to branch: stable/4.1
Review: https://review.openstack.org/90221

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/90221
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=1e290c398712683566a11c78ceea532719e1f72e
Submitter: Jenkins
Branch: stable/4.1

commit 1e290c398712683566a11c78ceea532719e1f72e
Author: Sergey Vasilenko <email address hidden>
Date: Wed Apr 23 19:02:41 2014 +0400

    kill all proceses inside all l3-agent's net.namespaces,

    that using ip protocol. When l3 agent start/stop.
    Also remove unneeded namespaces.

    Change-Id: I7e5203f6cec646a8b70609f2bf3b785066068aeb
    Closes-Bug: #1310926

[Andrew Woodward (xarses)]

This commit was applied to the wrong bug.

[Vladimir Kuklin (vkuklin)]

all the commits for this bug were successfully merged. moving this bug to 4.1.1

[Andrew Woodward (xarses)]

needs fix in https://bugs.launchpad.net/fuel/+bug/1310926/comments/10

[Vladimir Kuklin (vkuklin)]

this fix is already merged into FUEL packages

[Nastya Urlapova (aurlapova)]

{
build_id: "2014-06-05_15-25-50",
mirantis: "yes",
build_number: "344",
nailgun_sha: "a828d6b7610f872980d5a2113774f1cda6f6810b",
ostf_sha: "2b7b39e4b6ea89751b65171f24a8e80b5cac56aa",
fuelmain_sha: "9964da7dec34d3100419c1c77c8f5235d8e30f14",
astute_sha: "55df06b2e84fa5d71a1cc0e78dbccab5db29d968",
release: "4.1B",
fuellib_sha: "3511461a2b529619a787a6306441d9039699e71d"
}

[Alexander Ignatov (aignatov)]

Change is merged to 6.0 in the following patch: https://gerrit.mirantis.com/#/c/29630/

[OSCI Robot (oscirobot)]

Package neutron has been built from changeset: https://review.fuel-infra.org/92
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0-stable-92/centos

[OSCI Robot (oscirobot)]

Package neutron has been built from changeset: https://review.fuel-infra.org/92
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0-stable-92/ubuntu

[OSCI Robot (oscirobot)]

Package neutron has been built from changeset: https://review.fuel-infra.org/92
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0-stable/centos

[OSCI Robot (oscirobot)]

Package neutron has been built from changeset: https://review.fuel-infra.org/92
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0-stable/ubuntu

[Bogdan Dobrelya (bogdando)]

Related https://bugs.launchpad.net/fuel/+bug/1397284

[Kristina Berezovskaia (kkuznetsova)]

Verify 6.0
iso
 {"build_id": "2014-12-26_14-25-46", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "58", "auth_required": true, "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "81d38d6f2903b5a8b4bee79ca45a54b76c1361b8", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-12-26_14-25-46", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "58", "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "81d38d6f2903b5a8b4bee79ca45a54b76c1361b8", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "fde8ba5e11a1acaf819d402c645c731af450aff0"}}}, "fuellib_sha": "fde8ba5e11a1acaf819d402c645c731af450aff0"}

ubuntu neutron vlan, 3 controller + 2 compute / ceph

Steps to reproduce:
1) Create new net, subnet, router
2) lounch instance on this new net
2) associate floating IP to the VM
3) in CLi ping floating IP
4) go with ssh to the VM

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Sergey Vasilenko <email address hidden>
Review: https://review.fuel-infra.org/7996

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Oleg Bondarev <email address hidden> on branch: openstack-ci/fuel-7.0/2015.1.0
Review: https://review.fuel-infra.org/7996
Reason: Not needed now as from Kilo there is an option to use root rights when listing namespaces, True by default use_helper_for_ns_read