Grizzly : l3-agent is not able to attach the device representing "network:router_gateway"

Bug #1180734 reported by Ranjit Ranjan
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
neutron
Expired
Undecided
Unassigned

Bug Description

Hi,

I am trying to setup the multi-node environment for grizzly by using the instruction in the link:
https://github.com/mseknibilel/OpenStack-Grizzly-Install-Guide/blob/OVS_MultiNode/OpenStack_Grizzly_Install_Guide.rst.

we are getting errror in l3-agent.log file while setting the gateway for the router.
Below is the steps , I used to create the external network and setting the gateway.

root@ControllerOS:~# keystone tenant-list
+----------------------------------+-------------+---------+
| id | name | enabled |
+----------------------------------+-------------+---------+
| 942a2647fc264c1eba700fcdefba9762 | admin | True |
| 940bb16dbd2548ebbe71f141db2d9e23 | project_one | True |
| adf48648c5f9414dbbb768a172911363 | service | True |
+----------------------------------+-------------+---------+
root@ControllerOS:~#

root@ControllerOS:~# quantum net-create --tenant-id 942a2647fc264c1eba700fcdefba9762 ext-net --router:external true
Created a new network:
+---------------------------+--------------------------------------+
| Field | Value |
+---------------------------+--------------------------------------+
| admin_state_up | True |
| id | d57d10e5-ae54-4c5b-9793-de810c7c90c8 |
| name | ext-net |
| provider:network_type | gre |
| provider:physical_network | |
| provider:segmentation_id | 1 |
| router:external | True |
| shared | False |
| status | ACTIVE |
| subnets | |
| tenant_id | 942a2647fc264c1eba700fcdefba9762 |
+---------------------------+--------------------------------------+

root@ControllerOS:~# quantum subnet-create --tenant-id 942a2647fc264c1eba700fcdefba9762 --allocation-pool start=10.19.11.170,end=10.19.11.180 --gateway 10.19.11.1 ext-net 10.19.11.0/24
Created a new subnet:
+------------------+--------------------------------------------------+
| Field | Value |
+------------------+--------------------------------------------------+
| allocation_pools | {"start": "10.19.11.170", "end": "10.19.11.180"} |
| cidr | 10.19.11.0/24 |
| dns_nameservers | |
| enable_dhcp | True |
| gateway_ip | 10.19.11.1 |
| host_routes | |
| id | 14585957-9c0a-497d-acce-a50e7cb19bb4 |
| ip_version | 4 |
| name | |
| network_id | d57d10e5-ae54-4c5b-9793-de810c7c90c8 |
| tenant_id | 942a2647fc264c1eba700fcdefba9762 |
+------------------+--------------------------------------------------+

root@ControllerOS:~# quantum router-list
+--------------------------------------+-----------------+-----------------------+
| id | name | external_gateway_info |
+--------------------------------------+-----------------+-----------------------+
| dd081989-eeb5-421b-b793-ad04932225e4 | router_proj_one | null |
+--------------------------------------+-----------------+-----------------------+

root@ControllerOS:~# quantum router-gateway-set router_proj_one ext-net
Set gateway for router router_proj_one
root@ControllerOS:~#

ERROR FROM /var/log/quantum/l3-agent.log
--------------------------------------------------------------------------
2013-05-16 14:57:43 WARNING [quantum.openstack.common.loopingcall] task run outlasted interval by 12.887548 sec
2013-05-16 14:58:43 ERROR [quantum.agent.l3_agent] Failed synchronizing routers
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/quantum/agent/l3_agent.py", line 639, in _sync_routers_task
    self._process_routers(routers, all_routers=True)
  File "/usr/lib/python2.7/dist-packages/quantum/agent/l3_agent.py", line 622, in _process_routers
    self.process_router(ri)
  File "/usr/lib/python2.7/dist-packages/quantum/agent/l3_agent.py", line 320, in process_router
    self.external_gateway_added(ri, ex_gw_port, internal_cidrs)
  File "/usr/lib/python2.7/dist-packages/quantum/agent/l3_agent.py", line 411, in external_gateway_added
    prefix=EXTERNAL_DEV_PREFIX)
  File "/usr/lib/python2.7/dist-packages/quantum/agent/linux/interface.py", line 177, in plug
    ns_dev.link.set_address(mac_address)
  File "/usr/lib/python2.7/dist-packages/quantum/agent/linux/ip_lib.py", line 173, in set_address
    self._as_root('set', self.name, 'address', mac_address)
  File "/usr/lib/python2.7/dist-packages/quantum/agent/linux/ip_lib.py", line 160, in _as_root
    kwargs.get('use_root_namespace', False))
  File "/usr/lib/python2.7/dist-packages/quantum/agent/linux/ip_lib.py", line 47, in _as_root
    namespace)
  File "/usr/lib/python2.7/dist-packages/quantum/agent/linux/ip_lib.py", line 58, in _execute
    root_helper=root_helper)
  File "/usr/lib/python2.7/dist-packages/quantum/agent/linux/utils.py", line 61, in execute
    raise RuntimeError(m)
RuntimeError:
Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ip', 'link', 'set', 'qg-6d99c462-40', 'address', 'fa:16:3e:39:a4:d8']
Exit code: 1
Stdout: ''
Stderr: 'Cannot find device "qg-6d99c462-40"\n'
--------------------------------------------------------------------------------

From Network Node , the ovs-vsctl show command is showing the interface is added to the external bridge which is also
having the interface having external internet access.
root@networknode:/var/log/quantum# ovs-vsctl show
e47ecc7e-0bc6-44ab-8471-74cef3ca023d
    Bridge br-ex
        Port "eth1"
            Interface "eth1"
        Port "qg-6d99c462-40"
            Interface "qg-6d99c462-40"
                type: internal
        Port br-ex
            Interface br-ex
                type: internal

-------------------------------------------------------------------------------------------------------

Port information from HORIZON:

Port Overview
Port
Name
None
ID
6d99c462-40b3-4d5e-bb99-a5f995d1acd9
Network ID
d57d10e5-ae54-4c5b-9793-de810c7c90c8
Project ID
-
Fixed IP
IP address: 10.19.11.171, Subnet ID 14585957-9c0a-497d-acce-a50e7cb19bb4
Mac Address
fa:16:3e:39:a4:d8
Status
DOWN
Admin State
UP
Attached Device
Device Owner: network:router_gateway
Device ID: dd081989-eeb5-421b-b793-ad04932225e4

------------------------------------------------------------------------------------------------------------

What could be problem with l3-agent. I have tried this multiple time but it is giving the same error.
Due to this, i am not able to access the VM instance by using floating-IP .

Thanks for looking into this issue.

Tags: l3-ipam-dhcp
summary: - Grizzly : l3-agent is not able attach the device representing
+ Grizzly : l3-agent is not able to attach the device representing
"network:router_gateway"
Revision history for this message
yong sheng gong (gongysh) wrote :

what tenant env are u in when u are running these quantum command such as:
root@ControllerOS:~# quantum router-gateway-set router_proj_one ext-net
root@ControllerOS:~# quantum subnet-create --tenant-id 942a2647fc264c1eba700fcdefba9762

Revision history for this message
Ranjit Ranjan (ranjit-sastra) wrote :

Hi yong,

I used the 'admin' tenant environment for running the above commands.

Thanks
Ranjit

Revision history for this message
Coulin Jean-Luc (jean-luc-coulin) wrote :
Download full text (7.8 KiB)

Hello guys,

I am in the same situation.

But I created external network on command line as admin environment, and set the router and its gateway thru the horizon interface as a project member.

Note that I use namesapce.

I set the debug in l3_config.ini file so I got these errors when attaching the gateway to the router :

2013-05-24 16:26:15 DEBUG [quantum.agent.linux.utils] Running command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-vsctl', '--', '--may-exist', 'add-port', 'br-ex', 'qg-196a49aa-ec', '--', 'set', 'Interface', 'qg-196a49aa-ec', 'type=internal', '--', 'set', 'Interface', 'qg-196a49aa-ec', 'external-ids:iface-id=196a49aa-ec68-44f6-9249-aa9372441090', '--', 'set', 'Interface', 'qg-196a49aa-ec', 'external-ids:iface-status=active', '--', 'set', 'Interface', 'qg-196a49aa-ec', 'external-ids:attached-mac=fa:16:3e:bd:79:57']
2013-05-24 16:26:15 DEBUG [quantum.agent.linux.utils]
Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-vsctl', '--', '--may-exist', 'add-port', 'br-ex', 'qg-196a49aa-ec', '--', 'set', 'Interface', 'qg-196a49aa-ec', 'type=internal', '--', 'set', 'Interface', 'qg-196a49aa-ec', 'external-ids:iface-id=196a49aa-ec68-44f6-9249-aa9372441090', '--', 'set', 'Interface', 'qg-196a49aa-ec', 'external-ids:iface-status=active', '--', 'set', 'Interface', 'qg-196a49aa-ec', 'external-ids:attached-mac=fa:16:3e:bd:79:57']
Exit code: 0
Stdout: ''
Stderr: ''
2013-05-24 16:26:15 DEBUG [quantum.agent.linux.utils] Running command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ip', 'link', 'set', 'qg-196a49aa-ec', 'address', 'fa:16:3e:bd:79:57']
2013-05-24 16:26:15 DEBUG [quantum.agent.linux.utils]
Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ip', 'link', 'set', 'qg-196a49aa-ec', 'address', 'fa:16:3e:bd:79:57']
Exit code: 1
Stdout: ''
Stderr: 'Cannot find device "qg-196a49aa-ec"\n'
2013-05-24 16:26:15 DEBUG [quantum.agent.l3_agent] Failed dealing with routers update RPC message

And afterward every 30seconds I get these error messages :

2013-05-24 16:26:29 DEBUG [quantum.agent.linux.utils] Running command: ['ip', '-o', 'link', 'show', 'br-ex']
2013-05-24 16:26:29 DEBUG [quantum.agent.linux.utils]
Command: ['ip', '-o', 'link', 'show', 'br-ex']
Exit code: 0
Stdout: '4: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP \\ link/ether 00:50:56:bb:0c:b7 brd ff:ff:ff:ff:ff:ff\n'
Stderr: ''
2013-05-24 16:26:29 DEBUG [quantum.openstack.common.rpc.amqp] Making synchronous call on q-plugin ...
2013-05-24 16:26:29 DEBUG [quantum.openstack.common.rpc.amqp] MSG_ID is df7bb28ae1a14b85ad43244b83f19a1d
2013-05-24 16:26:29 DEBUG [quantum.openstack.common.rpc.amqp] UNIQUE_ID is 4857f234f473454d866c36ed59186c08.
2013-05-24 16:26:29 DEBUG [quantum.agent.linux.utils] Running command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-1e62de05-2702-45ca-8a27-daf60f0f8768', 'ip', '-o', 'link', 'show', 'qg-196a49aa-ec']
2013-05-24 16:26:29 DEBUG [quantum.agent.linux.utils]
Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-1e6...

Read more...

Revision history for this message
Coulin Jean-Luc (jean-luc-coulin) wrote :

Hello all,

I corrected this point for me.

What I did was : set an IP address to br-ex bridge at boot time via the /etc/network/interfaces file.

Today I tried to remove this configuration and add an ip address from command line after the boot.

Magicly :
     - every errors desapeared from the l3_agent.log file;
     - the qg .... device is set with its mac address;
     - and it is attached to the router in its ip namespace.

hope this can help someone else.

Best regards.

Revision history for this message
Ranjit Ranjan (ranjit-sastra) wrote :

HI Jean-Luc,

I was having the same configuration pointed by you.
Today I have removed the IP address for br-ex from /etc/network/interface file and i also deleted some configuration suggested in
https://github.com/mseknibilel/OpenStack-Grizzly-Install-Guide/blob/OVS_MultiNode/OpenStack_Grizzly_Install_Guide.rst

And it starts working. :).

one query: Do we need the below setting in interface file? Currently i am not using this.

auto eth2
iface eth2 inet manual
up ifconfig $IFACE 0.0.0.0 up
up ip link set $IFACE promisc on
down ip link set $IFACE promisc off
down ifconfig $IFACE down

Revision history for this message
Coulin Jean-Luc (jean-luc-coulin) wrote :

Hello Ranjit,

I just keep the following configuration lines in the /etc/network/interfaces file :
   auto eth2
   iface eth2 inet manual

Changed in quantum:
status: New → Incomplete
tags: added: l3-ipam-dhcp
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for neutron because there has been no activity for 60 days.]

Changed in neutron:
status: Incomplete → Expired
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.