OpenStack Neutron (virtual network service)

rootwrap error with L3 agent

Reported by dan wendlandt on 2012-10-22
38
This bug affects 5 people
Affects Status Importance Assigned to Milestone
neutron
High
Akihiro Motoki
Folsom
High
Gary Kotton
quantum (Ubuntu)
Undecided
Unassigned
Precise
Undecided
Unassigned
Quantal
Undecided
Unassigned

Bug Description

I am seeing the following error in /var/log/quantum/l3_agent.log:

2012-10-22 09:00:48 DEBUG [quantum.agent.linux.utils] Running command: sudo /usr/bin/quantum-rootwrap /etc/quantum/rootwrap.conf /sbin/iptables-save -t filter
2012-10-22 09:00:48 DEBUG [quantum.agent.linux.utils]
Command: ['sudo', '/usr/bin/quantum-rootwrap', '/etc/quantum/rootwrap.conf', '/sbin/iptables-save', '-t', 'filter']
Exit code: 99
Stdout: 'Unauthorized command: /sbin/iptables-save -t filter\n'
Stderr: ''
2012-10-22 09:00:48 ERROR [quantum.agent.l3_agent] Error running l3_nat daemon_loop
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/quantum/agent/l3_agent.py", line 170, in daemon_loop
    self.do_single_loop()
  File "/usr/lib/python2.7/dist-packages/quantum/agent/l3_agent.py", line 227, in do_single_loop
    self.process_router(ri)
  File "/usr/lib/python2.7/dist-packages/quantum/agent/l3_agent.py", line 300, in process_router
    self.external_gateway_added(ri, ex_gw_port, internal_cidrs)
  File "/usr/lib/python2.7/dist-packages/quantum/agent/l3_agent.py", line 398, in external_gateway_added
    ri.iptables_manager.apply()
  File "/usr/lib/python2.7/dist-packages/quantum/agent/linux/iptables_manager.py", line 282, in apply
    root_helper=self.root_helper))
  File "/usr/lib/python2.7/dist-packages/quantum/agent/linux/utils.py", line 55, in execute
    raise RuntimeError(m)
RuntimeError:
Command: ['sudo', '/usr/bin/quantum-rootwrap', '/etc/quantum/rootwrap.conf', '/sbin/iptables-save', '-t', 'filter']
Exit code: 99
Stdout: 'Unauthorized command: /sbin/iptables-save -t filter\n'
Stderr: ''

If I run "sudo /usr/bin/quantum-rootwrap /etc/quantum/rootwrap.conf /sbin/iptables-save -t filter" it does indeed give me an Unauthorized command error.

If I run "sudo /usr/bin/quantum-rootwrap /etc/quantum/rootwrap.conf iptables-save -t filter" (without the /sbin/) it works OK.
Otherwise, I don't see errors in the log.

Is this a problem?

Thanks,
   Graham

dan wendlandt (danwent) wrote :

also from graham:

I have focused in on this a bit and wanted to add some more details.
First, this only happens once I set the router_id in l3_agent.ini. Only then does this error occur.

I noticed that line 272 of quantum/agent/linux/iptables_manager.py is:

        s = [('/sbin/iptables', self.ipv4)]

If I change this to:

        s = [('iptables', self.ipv4)]

It seems to work without error. Is this correct?

dan wendlandt (danwent) on 2012-10-22
tags: added: folsom-backport-potential

On 10/22/2012 09:33 PM, dan wendlandt wrote:
> ** Tags added: folsom-backport-potential
>
Hi,
This happens the first time that the program is run. I do not think that
this is a bug. The next time this is called the error does not happen.
It is easily reproducible.
Thanks
Gary

Akihiro Motoki (amotoki) wrote :

rootwrap expects the command name is not absolute, so rootwrap filter is not found when /sbin/iptables is used.
I think changing '/sbin/iptables-save' to 'iptables' is a reasonable solution for it.

@garyk
iptables-save always fails in l3-agent when I ran devstack with use_namespaces=False. Does it only happen only the first time that l3-agent runs?

Gary Kotton (garyk) on 2012-10-24
Changed in quantum:
status: New → Confirmed
Akihiro Motoki (amotoki) on 2012-10-24
Changed in quantum:
assignee: nobody → Akihiro Motoki (amotoki)

Fix proposed to branch: master
Review: https://review.openstack.org/14756

Changed in quantum:
status: Confirmed → In Progress

Reviewed: https://review.openstack.org/14756
Committed: http://github.com/openstack/quantum/commit/84d60f5fd477237bd856b97b9970dd796b10647e
Submitter: Jenkins
Branch: master

commit 84d60f5fd477237bd856b97b9970dd796b10647e
Author: Akihiro MOTOKI <email address hidden>
Date: Wed Oct 24 22:14:06 2012 +0900

    Call iptables without absolute path.

    Fixes bug 1069966

    rootwrap expects the command name is not absolute. We need to call
    the command without path to make rootwrap work well.

    Change-Id: I6120103908d10ca257d177a320294de06a89c646

Changed in quantum:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/14811
Committed: http://github.com/openstack/quantum/commit/425e942fa28667f57282044f653a73c7fa60cfff
Submitter: Jenkins
Branch: stable/folsom

commit 425e942fa28667f57282044f653a73c7fa60cfff
Author: Akihiro MOTOKI <email address hidden>
Date: Wed Oct 24 22:14:06 2012 +0900

    Call iptables without absolute path.

    Fixes bug 1069966

    rootwrap expects the command name is not absolute. We need to call
    the command without path to make rootwrap work well.

    Change-Id: I6120103908d10ca257d177a320294de06a89c646

tags: added: in-stable-folsom
Gary Kotton (garyk) on 2012-11-01
tags: removed: folsom-backport-potential
Chuck Short (zulcss) on 2012-11-06
Changed in quantum (Ubuntu):
status: New → Fix Released
Changed in quantum (Ubuntu Precise):
status: New → Confirmed
dan wendlandt (danwent) on 2012-11-16
Changed in quantum:
milestone: none → grizzly-1
importance: Undecided → High
Thierry Carrez (ttx) on 2012-11-21
Changed in quantum:
status: Fix Committed → Fix Released
Changed in quantum (Ubuntu Quantal):
status: New → Confirmed

Hello dan, or anyone else affected,

Accepted quantum into quantal-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/quantum/2012.2.1-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in quantum (Ubuntu Quantal):
status: Confirmed → Fix Committed
tags: added: verification-needed
Thiago Martins (martinx) wrote :

Hi!

 The quantum 2012.2.1-0ubuntu1 isn't available at -proposed repository yet.

 How much it will take to upload it?

 HAPPY NEW YEAR!

Tks!
Thiago

Thiago Martins (martinx) wrote :

Just for the record, I'm using:

quantum-l3-agent - 2012.2-0ubuntu1~cloud0

And this problem persist... Is this the fixed version?

Gary Kotton (garyk) wrote :

Hi,
The issue is fixed in the stable/folsom version. I am not sure if this is part of the official Ubuntu packages. I'll let someone else familair with that answer.
Thanks
Gary

Thiago Martins (martinx) wrote :

Hi!

 I'm following the document: http://docs.openstack.org/folsom/basic-install/content/basic-install_network.html

 Which sets use_namespaces to False... And the problem still exists...

Tks,
Thiago

Thiago Martins (martinx) wrote :

Okay, thank you Gary!

dan wendlandt (danwent) wrote :

adding adam to the bug, who can comment for sure, but it looks like the issues is just "fix committed" for quantal, not "fixed released", so I would guess its not in the latest released version for ubuntu.

Thiago Martins (martinx) wrote :

Dan,

 I'm using Ubuntu 12.04 with "Ubuntu Cloud Archives" repository enabled.

 Also, both -proposed are enabled (for Ubuntu and for UCA).

Tks!
Thiago

Thiago Martins (martinx) wrote :

Oh! Now I'm seeing that it is only valid for Quantal... Sorry... Tks!

Thiago Martins (martinx) wrote :

Well, I install Ubuntu 12.10 to try this new package and the problem disappear but, I'm getting this now:

2012-12-31 00:49:10 ERROR [quantum.plugins.openvswitch.agent.ovs_quantum_agent] Failed to create OVS patch port. Cannot have tunneling enabled on this agent, since this version of OVS does not support tunnels or patch ports. Agent terminated!

I think it is not related to this BUG but, well... Both Precise and Quantal have problems... Can't continue with my Openstack PoC... :-(

Thanks anyway,
Thiago

Download full text (3.7 KiB)

This is indeed a separate issue, but one that I believe the Ubuntu people
had fixed. Can you make sure you have the latest OVS? Ubuntu originally
shipped Quantal with only the built-in kernel support for OVS, which does
not support tunneling.

Dan

On Sun, Dec 30, 2012 at 6:57 PM, Thiago Martins
<email address hidden>wrote:

> Well, I install Ubuntu 12.10 to try this new package and the problem
> disappear but, I'm getting this now:
>
> 2012-12-31 00:49:10 ERROR
> [quantum.plugins.openvswitch.agent.ovs_quantum_agent] Failed to create
> OVS patch port. Cannot have tunneling enabled on this agent, since this
> version of OVS does not support tunnels or patch ports. Agent
> terminated!
>
> I think it is not related to this BUG but, well... Both Precise and
> Quantal have problems... Can't continue with my Openstack PoC... :-(
>
> Thanks anyway,
> Thiago
>
> --
> You received this bug notification because you are a member of Netstack
> Core Developers, which is subscribed to quantum.
> https://bugs.launchpad.net/bugs/1069966
>
> Title:
> rootwrap error with L3 agent
>
> Status in OpenStack Quantum (virtual network service):
> Fix Released
> Status in quantum folsom series:
> Fix Released
> Status in “quantum” package in Ubuntu:
> Fix Released
> Status in “quantum” source package in Precise:
> Confirmed
> Status in “quantum” source package in Quantal:
> Fix Committed
>
> Bug description:
> I am seeing the following error in /var/log/quantum/l3_agent.log:
>
> 2012-10-22 09:00:48 DEBUG [quantum.agent.linux.utils] Running
> command: sudo /usr/bin/quantum-rootwrap /etc/quantum/rootwrap.conf
> /sbin/iptables-save -t filter
> 2012-10-22 09:00:48 DEBUG [quantum.agent.linux.utils]
> Command: ['sudo', '/usr/bin/quantum-rootwrap',
> '/etc/quantum/rootwrap.conf', '/sbin/iptables-save', '-t', 'filter']
> Exit code: 99
> Stdout: 'Unauthorized command: /sbin/iptables-save -t filter\n'
> Stderr: ''
> 2012-10-22 09:00:48 ERROR [quantum.agent.l3_agent] Error running
> l3_nat daemon_loop
> Traceback (most recent call last):
> File "/usr/lib/python2.7/dist-packages/quantum/agent/l3_agent.py",
> line 170, in daemon_loop
> self.do_single_loop()
> File "/usr/lib/python2.7/dist-packages/quantum/agent/l3_agent.py",
> line 227, in do_single_loop
> self.process_router(ri)
> File "/usr/lib/python2.7/dist-packages/quantum/agent/l3_agent.py",
> line 300, in process_router
> self.external_gateway_added(ri, ex_gw_port, internal_cidrs)
> File "/usr/lib/python2.7/dist-packages/quantum/agent/l3_agent.py",
> line 398, in external_gateway_added
> ri.iptables_manager.apply()
> File
> "/usr/lib/python2.7/dist-packages/quantum/agent/linux/iptables_manager.py",
> line 282, in apply
> root_helper=self.root_helper))
> File "/usr/lib/python2.7/dist-packages/quantum/agent/linux/utils.py",
> line 55, in execute
> raise RuntimeError(m)
> RuntimeError:
> Command: ['sudo', '/usr/bin/quantum-rootwrap',
> '/etc/quantum/rootwrap.conf', '/sbin/iptables-save', '-t', 'filter']
> Exit code: 99
> Stdout: 'Unauthorized command: /sbin/iptables-save -t filter\n'
> Stderr: ''
>
> If I ...

Read more...

Thiago Martins (martinx) wrote :

Dan,

 I don't want to hijack this thread because of another problem but, well, I have installed openvswitch (1.4.3-0ubuntu2) and I thought that is was enough but it doesn't...

 I think that the discussion of this new problem is here: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1068365

 Right?

 Thank you!

Best,
Thiago

dan wendlandt (danwent) wrote :
Download full text (3.4 KiB)

yup. let's put any further questions/comments there.

On Sun, Dec 30, 2012 at 9:09 PM, Thiago Martins
<email address hidden>wrote:

> Dan,
>
> I don't want to hijack this thread because of another problem but,
> well, I have installed openvswitch (1.4.3-0ubuntu2) and I thought that
> is was enough but it doesn't...
>
> I think that the discussion of this new problem is here:
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1068365
>
> Right?
>
> Thank you!
>
> Best,
> Thiago
>
> --
> You received this bug notification because you are a member of Netstack
> Core Developers, which is subscribed to quantum.
> https://bugs.launchpad.net/bugs/1069966
>
> Title:
> rootwrap error with L3 agent
>
> Status in OpenStack Quantum (virtual network service):
> Fix Released
> Status in quantum folsom series:
> Fix Released
> Status in “quantum” package in Ubuntu:
> Fix Released
> Status in “quantum” source package in Precise:
> Confirmed
> Status in “quantum” source package in Quantal:
> Fix Committed
>
> Bug description:
> I am seeing the following error in /var/log/quantum/l3_agent.log:
>
> 2012-10-22 09:00:48 DEBUG [quantum.agent.linux.utils] Running
> command: sudo /usr/bin/quantum-rootwrap /etc/quantum/rootwrap.conf
> /sbin/iptables-save -t filter
> 2012-10-22 09:00:48 DEBUG [quantum.agent.linux.utils]
> Command: ['sudo', '/usr/bin/quantum-rootwrap',
> '/etc/quantum/rootwrap.conf', '/sbin/iptables-save', '-t', 'filter']
> Exit code: 99
> Stdout: 'Unauthorized command: /sbin/iptables-save -t filter\n'
> Stderr: ''
> 2012-10-22 09:00:48 ERROR [quantum.agent.l3_agent] Error running
> l3_nat daemon_loop
> Traceback (most recent call last):
> File "/usr/lib/python2.7/dist-packages/quantum/agent/l3_agent.py",
> line 170, in daemon_loop
> self.do_single_loop()
> File "/usr/lib/python2.7/dist-packages/quantum/agent/l3_agent.py",
> line 227, in do_single_loop
> self.process_router(ri)
> File "/usr/lib/python2.7/dist-packages/quantum/agent/l3_agent.py",
> line 300, in process_router
> self.external_gateway_added(ri, ex_gw_port, internal_cidrs)
> File "/usr/lib/python2.7/dist-packages/quantum/agent/l3_agent.py",
> line 398, in external_gateway_added
> ri.iptables_manager.apply()
> File
> "/usr/lib/python2.7/dist-packages/quantum/agent/linux/iptables_manager.py",
> line 282, in apply
> root_helper=self.root_helper))
> File "/usr/lib/python2.7/dist-packages/quantum/agent/linux/utils.py",
> line 55, in execute
> raise RuntimeError(m)
> RuntimeError:
> Command: ['sudo', '/usr/bin/quantum-rootwrap',
> '/etc/quantum/rootwrap.conf', '/sbin/iptables-save', '-t', 'filter']
> Exit code: 99
> Stdout: 'Unauthorized command: /sbin/iptables-save -t filter\n'
> Stderr: ''
>
> If I run "sudo /usr/bin/quantum-rootwrap /etc/quantum/rootwrap.conf
> /sbin/iptables-save -t filter" it does indeed give me an Unauthorized
> command error.
>
> If I run "sudo /usr/bin/quantum-rootwrap /etc/quantum/rootwrap.conf
> iptables-save -t filter" (without the /sbin/) it works OK.
> Otherwise, I don't see errors in the log.
>
> Is this a problem?
>
> Thanks,
> ...

Read more...

Mark McLoughlin (markmc) on 2013-01-22
tags: removed: in-stable-folsom
Launchpad Janitor (janitor) wrote :
Download full text (3.8 KiB)

This bug was fixed in the package quantum - 2012.2.1-0ubuntu1

---------------
quantum (2012.2.1-0ubuntu1) quantal-proposed; urgency=low

  * Resynchronize with stable/folsom (1e774867) (LP: #1085255):
    - [aeabb42] There are routing problems when the dnsmasq port does not come
      first in the routing table (LP: #1083238)
    - [04aab72] Quantum linux bridge not optimized with libvirt (LP: #1078210)
    - [ca7fc10] getting quotas from database has severe performance implications
      (LP: #1075369)
    - [66605e8] failed to update an external network into non external network
      (LP: #1083387)
    - [c60051a] Quantum test suite leaks memory like a sieve (LP: #1065276)
    - [3179dfc] clear_db() does incomplete db teardown (LP: #1080988)
    - [c1e19d7] Unauthorized command: cat /proc/None/cmdline (LP: #1077651)
    - [af9e076] At times a instance will not receive an IP address from the DHCP
      agent (LP: #1081664)
    - [e0d1a7d] allow multiple floating-ip on single port if they use different
      fixed ips and/or external nets (LP: #1057844)
    - [8471d79] Delete port fails to gateway ip (LP: #1079980)
    - [aca8b4a] fixed_ip allocation which is not included within
      allocation_pools makes error when delete port or re-create port
      (LP: #1077292)
    - [eacc9d3] Mapping same bridge to different phyiscal networks succeed
      (LP: #1067669)
    - [51b4c82] python-quantum: not region aware (LP: #1080793)
    - [6f0a486] delete floatingip should be in one transaction to delete port
      (LP: #1080516)
    - [db6cda7] Remove qpid configuration variables no longer supported
    - [a112840] Allow NVP plugin to use per-tenant quota extension
    - [82b1a55] Quantum service does not restart after reboot (LP: #1073999)
    - [c01a839] There are some cases that L3 API with an invalid parameter
      returns 500. (LP: #1064765)
    - [26b383f] external network can be plugged also as internal network for one
      router (LP: #1053633)
    - [49f649c] There is a lot of cases that API with an invalid parameter
      returns 500. (LP: #1062046)
    - [4546a18] When create subnet, you con set up the value as cidr (the value
      isn't cidr form). (LP: #1067959)
    - [9ba453a] killfilter should handle updated/deleted executables
      (LP: #1073768)
    - [7c8a55c] a port which is not able to delete is made when floatingip
      create fails. (LP: #1064748)
    - [c9b84cf] Linux bridge port update causes exception (LP: #1072713)
    - [cb57932] I can't add interface to router, if there is another port in
      non-shared network of other tenant (LP: #1057558)
    - [574e278] Ryu plugin does not support Security Groups (LP: #1059393)
    - [607f486] tap device added to integration bridge without tag
      (LP: #1064070)
    - [21a0fdf] L3 agent external network flag (LP: #1056720)
    - [5cbaff4] router create with external_gateway_info fails with 500 always.
      (LP: #1064235)
    - [63b81f6] l3 db operations failed in multiple transactions (LP: #1070335)
    - [bff17fb] Ensure that the SqlSoup import is still supported.
    - [e091a29] l3_nat_agent was renamed to l3_agent
    - [9030969] remove default value of 'local_ip' of 10...

Read more...

Changed in quantum (Ubuntu Quantal):
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2013-04-04
Changed in quantum:
milestone: grizzly-1 → 2013.1
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Related questions