Grizzly : l3-agent is not able to attach the device representing "network:router_gateway"

[Ranjit Ranjan]

Hi,

I am trying to setup the multi-node environment for grizzly by using the instruction in the link:
https://github.com/mseknibilel/OpenStack-Grizzly-Install-Guide/blob/OVS_MultiNode/OpenStack_Grizzly_Install_Guide.rst.

we are getting errror in l3-agent.log file while setting the gateway for the router.
Below is the steps , I used to create the external network and setting the gateway.

root@ControllerOS:~# keystone tenant-list
+----------------------------------+-------------+---------+
| id | name | enabled |
+----------------------------------+-------------+---------+
| 942a2647fc264c1eba700fcdefba9762 | admin | True |
| 940bb16dbd2548ebbe71f141db2d9e23 | project_one | True |
| adf48648c5f9414dbbb768a172911363 | service | True |
+----------------------------------+-------------+---------+
root@ControllerOS:~#

root@ControllerOS:~# quantum net-create --tenant-id 942a2647fc264c1eba700fcdefba9762 ext-net --router:external true
Created a new network:
+---------------------------+--------------------------------------+
| Field | Value |
+---------------------------+--------------------------------------+
| admin_state_up | True |
| id | d57d10e5-ae54-4c5b-9793-de810c7c90c8 |
| name | ext-net |
| provider:network_type | gre |
| provider:physical_network | |
| provider:segmentation_id | 1 |
| router:external | True |
| shared | False |
| status | ACTIVE |
| subnets | |
| tenant_id | 942a2647fc264c1eba700fcdefba9762 |
+---------------------------+--------------------------------------+

root@ControllerOS:~# quantum subnet-create --tenant-id 942a2647fc264c1eba700fcdefba9762 --allocation-pool start=10.19.11.170,end=10.19.11.180 --gateway 10.19.11.1 ext-net 10.19.11.0/24
Created a new subnet:
+------------------+--------------------------------------------------+
| Field | Value |
+------------------+--------------------------------------------------+
| allocation_pools | {"start": "10.19.11.170", "end": "10.19.11.180"} |
| cidr | 10.19.11.0/24 |
| dns_nameservers | |
| enable_dhcp | True |
| gateway_ip | 10.19.11.1 |
| host_routes | |
| id | 14585957-9c0a-497d-acce-a50e7cb19bb4 |
| ip_version | 4 |
| name | |
| network_id | d57d10e5-ae54-4c5b-9793-de810c7c90c8 |
| tenant_id | 942a2647fc264c1eba700fcdefba9762 |
+------------------+--------------------------------------------------+

root@ControllerOS:~# quantum router-list
+--------------------------------------+-----------------+-----------------------+
| id | name | external_gateway_info |
+--------------------------------------+-----------------+-----------------------+
| dd081989-eeb5-421b-b793-ad04932225e4 | router_proj_one | null |
+--------------------------------------+-----------------+-----------------------+

root@ControllerOS:~# quantum router-gateway-set router_proj_one ext-net
Set gateway for router router_proj_one
root@ControllerOS:~#

ERROR FROM /var/log/quantum/l3-agent.log
--------------------------------------------------------------------------
2013-05-16 14:57:43 WARNING [quantum.openstack.common.loopingcall] task run outlasted interval by 12.887548 sec
2013-05-16 14:58:43 ERROR [quantum.agent.l3_agent] Failed synchronizing routers
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/quantum/agent/l3_agent.py", line 639, in _sync_routers_task
    self._process_routers(routers, all_routers=True)
  File "/usr/lib/python2.7/dist-packages/quantum/agent/l3_agent.py", line 622, in _process_routers
    self.process_router(ri)
  File "/usr/lib/python2.7/dist-packages/quantum/agent/l3_agent.py", line 320, in process_router
    self.external_gateway_added(ri, ex_gw_port, internal_cidrs)
  File "/usr/lib/python2.7/dist-packages/quantum/agent/l3_agent.py", line 411, in external_gateway_added
    prefix=EXTERNAL_DEV_PREFIX)
  File "/usr/lib/python2.7/dist-packages/quantum/agent/linux/interface.py", line 177, in plug
    ns_dev.link.set_address(mac_address)
  File "/usr/lib/python2.7/dist-packages/quantum/agent/linux/ip_lib.py", line 173, in set_address
    self._as_root('set', self.name, 'address', mac_address)
  File "/usr/lib/python2.7/dist-packages/quantum/agent/linux/ip_lib.py", line 160, in _as_root
    kwargs.get('use_root_namespace', False))
  File "/usr/lib/python2.7/dist-packages/quantum/agent/linux/ip_lib.py", line 47, in _as_root
    namespace)
  File "/usr/lib/python2.7/dist-packages/quantum/agent/linux/ip_lib.py", line 58, in _execute
    root_helper=root_helper)
  File "/usr/lib/python2.7/dist-packages/quantum/agent/linux/utils.py", line 61, in execute
    raise RuntimeError(m)
RuntimeError:
Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ip', 'link', 'set', 'qg-6d99c462-40', 'address', 'fa:16:3e:39:a4:d8']
Exit code: 1
Stdout: ''
Stderr: 'Cannot find device "qg-6d99c462-40"\n'
--------------------------------------------------------------------------------

From Network Node , the ovs-vsctl show command is showing the interface is added to the external bridge which is also
having the interface having external internet access.
root@networknode:/var/log/quantum# ovs-vsctl show
e47ecc7e-0bc6-44ab-8471-74cef3ca023d
    Bridge br-ex
        Port "eth1"
            Interface "eth1"
        Port "qg-6d99c462-40"
            Interface "qg-6d99c462-40"
                type: internal
        Port br-ex
            Interface br-ex
                type: internal

-------------------------------------------------------------------------------------------------------

Port information from HORIZON:

Port Overview
Port
Name
None
ID
6d99c462-40b3-4d5e-bb99-a5f995d1acd9
Network ID
d57d10e5-ae54-4c5b-9793-de810c7c90c8
Project ID
-
Fixed IP
IP address: 10.19.11.171, Subnet ID 14585957-9c0a-497d-acce-a50e7cb19bb4
Mac Address
fa:16:3e:39:a4:d8
Status
DOWN
Admin State
UP
Attached Device
Device Owner: network:router_gateway
Device ID: dd081989-eeb5-421b-b793-ad04932225e4

------------------------------------------------------------------------------------------------------------

What could be problem with l3-agent. I have tried this multiple time but it is giving the same error.
Due to this, i am not able to access the VM instance by using floating-IP .

Thanks for looking into this issue.

[yong sheng gong (gongysh)]

what tenant env are u in when u are running these quantum command such as:
root@ControllerOS:~# quantum router-gateway-set router_proj_one ext-net
root@ControllerOS:~# quantum subnet-create --tenant-id 942a2647fc264c1eba700fcdefba9762

[Ranjit Ranjan (ranjit-sastra)]

Hi yong,

I used the 'admin' tenant environment for running the above commands.

Thanks
Ranjit

[Coulin Jean-Luc (jean-luc-coulin)]

Hello guys,

I am in the same situation.

But I created external network on command line as admin environment, and set the router and its gateway thru the horizon interface as a project member.

Note that I use namesapce.

I set the debug in l3_config.ini file so I got these errors when attaching the gateway to the router :

2013-05-24 16:26:15 DEBUG [quantum.agent.linux.utils] Running command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-vsctl', '--', '--may-exist', 'add-port', 'br-ex', 'qg-196a49aa-ec', '--', 'set', 'Interface', 'qg-196a49aa-ec', 'type=internal', '--', 'set', 'Interface', 'qg-196a49aa-ec', 'external-ids:iface-id=196a49aa-ec68-44f6-9249-aa9372441090', '--', 'set', 'Interface', 'qg-196a49aa-ec', 'external-ids:iface-status=active', '--', 'set', 'Interface', 'qg-196a49aa-ec', 'external-ids:attached-mac=fa:16:3e:bd:79:57']
2013-05-24 16:26:15 DEBUG [quantum.agent.linux.utils]
Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-vsctl', '--', '--may-exist', 'add-port', 'br-ex', 'qg-196a49aa-ec', '--', 'set', 'Interface', 'qg-196a49aa-ec', 'type=internal', '--', 'set', 'Interface', 'qg-196a49aa-ec', 'external-ids:iface-id=196a49aa-ec68-44f6-9249-aa9372441090', '--', 'set', 'Interface', 'qg-196a49aa-ec', 'external-ids:iface-status=active', '--', 'set', 'Interface', 'qg-196a49aa-ec', 'external-ids:attached-mac=fa:16:3e:bd:79:57']
Exit code: 0
Stdout: ''
Stderr: ''
2013-05-24 16:26:15 DEBUG [quantum.agent.linux.utils] Running command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ip', 'link', 'set', 'qg-196a49aa-ec', 'address', 'fa:16:3e:bd:79:57']
2013-05-24 16:26:15 DEBUG [quantum.agent.linux.utils]
Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ip', 'link', 'set', 'qg-196a49aa-ec', 'address', 'fa:16:3e:bd:79:57']
Exit code: 1
Stdout: ''
Stderr: 'Cannot find device "qg-196a49aa-ec"\n'
2013-05-24 16:26:15 DEBUG [quantum.agent.l3_agent] Failed dealing with routers update RPC message

And afterward every 30seconds I get these error messages :

2013-05-24 16:26:29 DEBUG [quantum.agent.linux.utils] Running command: ['ip', '-o', 'link', 'show', 'br-ex']
2013-05-24 16:26:29 DEBUG [quantum.agent.linux.utils]
Command: ['ip', '-o', 'link', 'show', 'br-ex']
Exit code: 0
Stdout: '4: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP \\ link/ether 00:50:56:bb:0c:b7 brd ff:ff:ff:ff:ff:ff\n'
Stderr: ''
2013-05-24 16:26:29 DEBUG [quantum.openstack.common.rpc.amqp] Making synchronous call on q-plugin ...
2013-05-24 16:26:29 DEBUG [quantum.openstack.common.rpc.amqp] MSG_ID is df7bb28ae1a14b85ad43244b83f19a1d
2013-05-24 16:26:29 DEBUG [quantum.openstack.common.rpc.amqp] UNIQUE_ID is 4857f234f473454d866c36ed59186c08.
2013-05-24 16:26:29 DEBUG [quantum.agent.linux.utils] Running command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-1e62de05-2702-45ca-8a27-daf60f0f8768', 'ip', '-o', 'link', 'show', 'qg-196a49aa-ec']
2013-05-24 16:26:29 DEBUG [quantum.agent.linux.utils]
Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-1e6...

[Coulin Jean-Luc (jean-luc-coulin)]

Hello all,

I corrected this point for me.

What I did was : set an IP address to br-ex bridge at boot time via the /etc/network/interfaces file.

Today I tried to remove this configuration and add an ip address from command line after the boot.

Magicly :
     - every errors desapeared from the l3_agent.log file;
     - the qg .... device is set with its mac address;
     - and it is attached to the router in its ip namespace.

hope this can help someone else.

Best regards.

[Ranjit Ranjan (ranjit-sastra)]

HI Jean-Luc,

I was having the same configuration pointed by you.
Today I have removed the IP address for br-ex from /etc/network/interface file and i also deleted some configuration suggested in
https://github.com/mseknibilel/OpenStack-Grizzly-Install-Guide/blob/OVS_MultiNode/OpenStack_Grizzly_Install_Guide.rst

And it starts working. :).

one query: Do we need the below setting in interface file? Currently i am not using this.

auto eth2
iface eth2 inet manual
up ifconfig $IFACE 0.0.0.0 up
up ip link set $IFACE promisc on
down ip link set $IFACE promisc off
down ifconfig $IFACE down

[Coulin Jean-Luc (jean-luc-coulin)]

Hello Ranjit,

I just keep the following configuration lines in the /etc/network/interfaces file :
   auto eth2
   iface eth2 inet manual

[Launchpad Janitor (janitor)]

[Expired for neutron because there has been no activity for 60 days.]