rootwrap error with L3 agent

Bug #1069966 reported by dan wendlandt
38
This bug affects 5 people
Affects Status Importance Assigned to Milestone
neutron
Fix Released
High
Akihiro Motoki
Folsom
Fix Released
High
Gary Kotton
quantum (Ubuntu)
Fix Released
Undecided
Unassigned
Precise
Won't Fix
Undecided
Unassigned
Quantal
Fix Released
Undecided
Unassigned

Bug Description

I am seeing the following error in /var/log/quantum/l3_agent.log:

2012-10-22 09:00:48 DEBUG [quantum.agent.linux.utils] Running command: sudo /usr/bin/quantum-rootwrap /etc/quantum/rootwrap.conf /sbin/iptables-save -t filter
2012-10-22 09:00:48 DEBUG [quantum.agent.linux.utils]
Command: ['sudo', '/usr/bin/quantum-rootwrap', '/etc/quantum/rootwrap.conf', '/sbin/iptables-save', '-t', 'filter']
Exit code: 99
Stdout: 'Unauthorized command: /sbin/iptables-save -t filter\n'
Stderr: ''
2012-10-22 09:00:48 ERROR [quantum.agent.l3_agent] Error running l3_nat daemon_loop
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/quantum/agent/l3_agent.py", line 170, in daemon_loop
    self.do_single_loop()
  File "/usr/lib/python2.7/dist-packages/quantum/agent/l3_agent.py", line 227, in do_single_loop
    self.process_router(ri)
  File "/usr/lib/python2.7/dist-packages/quantum/agent/l3_agent.py", line 300, in process_router
    self.external_gateway_added(ri, ex_gw_port, internal_cidrs)
  File "/usr/lib/python2.7/dist-packages/quantum/agent/l3_agent.py", line 398, in external_gateway_added
    ri.iptables_manager.apply()
  File "/usr/lib/python2.7/dist-packages/quantum/agent/linux/iptables_manager.py", line 282, in apply
    root_helper=self.root_helper))
  File "/usr/lib/python2.7/dist-packages/quantum/agent/linux/utils.py", line 55, in execute
    raise RuntimeError(m)
RuntimeError:
Command: ['sudo', '/usr/bin/quantum-rootwrap', '/etc/quantum/rootwrap.conf', '/sbin/iptables-save', '-t', 'filter']
Exit code: 99
Stdout: 'Unauthorized command: /sbin/iptables-save -t filter\n'
Stderr: ''

If I run "sudo /usr/bin/quantum-rootwrap /etc/quantum/rootwrap.conf /sbin/iptables-save -t filter" it does indeed give me an Unauthorized command error.

If I run "sudo /usr/bin/quantum-rootwrap /etc/quantum/rootwrap.conf iptables-save -t filter" (without the /sbin/) it works OK.
Otherwise, I don't see errors in the log.

Is this a problem?

Thanks,
   Graham

Revision history for this message
dan wendlandt (danwent) wrote :

also from graham:

I have focused in on this a bit and wanted to add some more details.
First, this only happens once I set the router_id in l3_agent.ini. Only then does this error occur.

I noticed that line 272 of quantum/agent/linux/iptables_manager.py is:

        s = [('/sbin/iptables', self.ipv4)]

If I change this to:

        s = [('iptables', self.ipv4)]

It seems to work without error. Is this correct?

dan wendlandt (danwent)
tags: added: folsom-backport-potential
Revision history for this message
Gary Kotton (garyk) wrote : Re: [Bug 1069966] Re: rootwrap error with L3 agent

On 10/22/2012 09:33 PM, dan wendlandt wrote:
> ** Tags added: folsom-backport-potential
>
Hi,
This happens the first time that the program is run. I do not think that
this is a bug. The next time this is called the error does not happen.
It is easily reproducible.
Thanks
Gary

Revision history for this message
Akihiro Motoki (amotoki) wrote :

rootwrap expects the command name is not absolute, so rootwrap filter is not found when /sbin/iptables is used.
I think changing '/sbin/iptables-save' to 'iptables' is a reasonable solution for it.

@garyk
iptables-save always fails in l3-agent when I ran devstack with use_namespaces=False. Does it only happen only the first time that l3-agent runs?

Gary Kotton (garyk)
Changed in quantum:
status: New → Confirmed
Akihiro Motoki (amotoki)
Changed in quantum:
assignee: nobody → Akihiro Motoki (amotoki)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to quantum (master)

Fix proposed to branch: master
Review: https://review.openstack.org/14756

Changed in quantum:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to quantum (master)

Reviewed: https://review.openstack.org/14756
Committed: http://github.com/openstack/quantum/commit/84d60f5fd477237bd856b97b9970dd796b10647e
Submitter: Jenkins
Branch: master

commit 84d60f5fd477237bd856b97b9970dd796b10647e
Author: Akihiro MOTOKI <email address hidden>
Date: Wed Oct 24 22:14:06 2012 +0900

    Call iptables without absolute path.

    Fixes bug 1069966

    rootwrap expects the command name is not absolute. We need to call
    the command without path to make rootwrap work well.

    Change-Id: I6120103908d10ca257d177a320294de06a89c646

Changed in quantum:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to quantum (stable/folsom)

Fix proposed to branch: stable/folsom
Review: https://review.openstack.org/14811

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to quantum (stable/folsom)

Reviewed: https://review.openstack.org/14811
Committed: http://github.com/openstack/quantum/commit/425e942fa28667f57282044f653a73c7fa60cfff
Submitter: Jenkins
Branch: stable/folsom

commit 425e942fa28667f57282044f653a73c7fa60cfff
Author: Akihiro MOTOKI <email address hidden>
Date: Wed Oct 24 22:14:06 2012 +0900

    Call iptables without absolute path.

    Fixes bug 1069966

    rootwrap expects the command name is not absolute. We need to call
    the command without path to make rootwrap work well.

    Change-Id: I6120103908d10ca257d177a320294de06a89c646

tags: added: in-stable-folsom
Gary Kotton (garyk)
tags: removed: folsom-backport-potential
Chuck Short (zulcss)
Changed in quantum (Ubuntu):
status: New → Fix Released
Changed in quantum (Ubuntu Precise):
status: New → Confirmed
dan wendlandt (danwent)
Changed in quantum:
milestone: none → grizzly-1
importance: Undecided → High
Thierry Carrez (ttx)
Changed in quantum:
status: Fix Committed → Fix Released
Changed in quantum (Ubuntu Quantal):
status: New → Confirmed
Revision history for this message
Clint Byrum (clint-fewbar) wrote : Please test proposed package

Hello dan, or anyone else affected,

Accepted quantum into quantal-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/quantum/2012.2.1-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in quantum (Ubuntu Quantal):
status: Confirmed → Fix Committed
tags: added: verification-needed
Revision history for this message
Thiago Martins (martinx) wrote :

Hi!

 The quantum 2012.2.1-0ubuntu1 isn't available at -proposed repository yet.

 How much it will take to upload it?

 HAPPY NEW YEAR!

Tks!
Thiago

Revision history for this message
Thiago Martins (martinx) wrote :

Just for the record, I'm using:

quantum-l3-agent - 2012.2-0ubuntu1~cloud0

And this problem persist... Is this the fixed version?

Revision history for this message
Gary Kotton (garyk) wrote :

Hi,
The issue is fixed in the stable/folsom version. I am not sure if this is part of the official Ubuntu packages. I'll let someone else familair with that answer.
Thanks
Gary

Revision history for this message
Thiago Martins (martinx) wrote :

Hi!

 I'm following the document: http://docs.openstack.org/folsom/basic-install/content/basic-install_network.html

 Which sets use_namespaces to False... And the problem still exists...

Tks,
Thiago

Revision history for this message
Thiago Martins (martinx) wrote :

Okay, thank you Gary!

Revision history for this message
dan wendlandt (danwent) wrote :

adding adam to the bug, who can comment for sure, but it looks like the issues is just "fix committed" for quantal, not "fixed released", so I would guess its not in the latest released version for ubuntu.

Revision history for this message
Thiago Martins (martinx) wrote :

Dan,

 I'm using Ubuntu 12.04 with "Ubuntu Cloud Archives" repository enabled.

 Also, both -proposed are enabled (for Ubuntu and for UCA).

Tks!
Thiago

Revision history for this message
Thiago Martins (martinx) wrote :

Oh! Now I'm seeing that it is only valid for Quantal... Sorry... Tks!

Revision history for this message
Thiago Martins (martinx) wrote :

Well, I install Ubuntu 12.10 to try this new package and the problem disappear but, I'm getting this now:

2012-12-31 00:49:10 ERROR [quantum.plugins.openvswitch.agent.ovs_quantum_agent] Failed to create OVS patch port. Cannot have tunneling enabled on this agent, since this version of OVS does not support tunnels or patch ports. Agent terminated!

I think it is not related to this BUG but, well... Both Precise and Quantal have problems... Can't continue with my Openstack PoC... :-(

Thanks anyway,
Thiago

Revision history for this message
dan wendlandt (danwent) wrote : Re: [Bug 1069966] Re: rootwrap error with L3 agent
Download full text (3.7 KiB)

This is indeed a separate issue, but one that I believe the Ubuntu people
had fixed. Can you make sure you have the latest OVS? Ubuntu originally
shipped Quantal with only the built-in kernel support for OVS, which does
not support tunneling.

Dan

On Sun, Dec 30, 2012 at 6:57 PM, Thiago Martins
<email address hidden>wrote:

> Well, I install Ubuntu 12.10 to try this new package and the problem
> disappear but, I'm getting this now:
>
> 2012-12-31 00:49:10 ERROR
> [quantum.plugins.openvswitch.agent.ovs_quantum_agent] Failed to create
> OVS patch port. Cannot have tunneling enabled on this agent, since this
> version of OVS does not support tunnels or patch ports. Agent
> terminated!
>
> I think it is not related to this BUG but, well... Both Precise and
> Quantal have problems... Can't continue with my Openstack PoC... :-(
>
> Thanks anyway,
> Thiago
>
> --
> You received this bug notification because you are a member of Netstack
> Core Developers, which is subscribed to quantum.
> https://bugs.launchpad.net/bugs/1069966
>
> Title:
> rootwrap error with L3 agent
>
> Status in OpenStack Quantum (virtual network service):
> Fix Released
> Status in quantum folsom series:
> Fix Released
> Status in “quantum” package in Ubuntu:
> Fix Released
> Status in “quantum” source package in Precise:
> Confirmed
> Status in “quantum” source package in Quantal:
> Fix Committed
>
> Bug description:
> I am seeing the following error in /var/log/quantum/l3_agent.log:
>
> 2012-10-22 09:00:48 DEBUG [quantum.agent.linux.utils] Running
> command: sudo /usr/bin/quantum-rootwrap /etc/quantum/rootwrap.conf
> /sbin/iptables-save -t filter
> 2012-10-22 09:00:48 DEBUG [quantum.agent.linux.utils]
> Command: ['sudo', '/usr/bin/quantum-rootwrap',
> '/etc/quantum/rootwrap.conf', '/sbin/iptables-save', '-t', 'filter']
> Exit code: 99
> Stdout: 'Unauthorized command: /sbin/iptables-save -t filter\n'
> Stderr: ''
> 2012-10-22 09:00:48 ERROR [quantum.agent.l3_agent] Error running
> l3_nat daemon_loop
> Traceback (most recent call last):
> File "/usr/lib/python2.7/dist-packages/quantum/agent/l3_agent.py",
> line 170, in daemon_loop
> self.do_single_loop()
> File "/usr/lib/python2.7/dist-packages/quantum/agent/l3_agent.py",
> line 227, in do_single_loop
> self.process_router(ri)
> File "/usr/lib/python2.7/dist-packages/quantum/agent/l3_agent.py",
> line 300, in process_router
> self.external_gateway_added(ri, ex_gw_port, internal_cidrs)
> File "/usr/lib/python2.7/dist-packages/quantum/agent/l3_agent.py",
> line 398, in external_gateway_added
> ri.iptables_manager.apply()
> File
> "/usr/lib/python2.7/dist-packages/quantum/agent/linux/iptables_manager.py",
> line 282, in apply
> root_helper=self.root_helper))
> File "/usr/lib/python2.7/dist-packages/quantum/agent/linux/utils.py",
> line 55, in execute
> raise RuntimeError(m)
> RuntimeError:
> Command: ['sudo', '/usr/bin/quantum-rootwrap',
> '/etc/quantum/rootwrap.conf', '/sbin/iptables-save', '-t', 'filter']
> Exit code: 99
> Stdout: 'Unauthorized command: /sbin/iptables-save -t filter\n'
> Stderr: ''
>
> If I ...

Read more...

Revision history for this message
Thiago Martins (martinx) wrote :

Dan,

 I don't want to hijack this thread because of another problem but, well, I have installed openvswitch (1.4.3-0ubuntu2) and I thought that is was enough but it doesn't...

 I think that the discussion of this new problem is here: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1068365

 Right?

 Thank you!

Best,
Thiago

Revision history for this message
dan wendlandt (danwent) wrote :
Download full text (3.4 KiB)

yup. let's put any further questions/comments there.

On Sun, Dec 30, 2012 at 9:09 PM, Thiago Martins
<email address hidden>wrote:

> Dan,
>
> I don't want to hijack this thread because of another problem but,
> well, I have installed openvswitch (1.4.3-0ubuntu2) and I thought that
> is was enough but it doesn't...
>
> I think that the discussion of this new problem is here:
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1068365
>
> Right?
>
> Thank you!
>
> Best,
> Thiago
>
> --
> You received this bug notification because you are a member of Netstack
> Core Developers, which is subscribed to quantum.
> https://bugs.launchpad.net/bugs/1069966
>
> Title:
> rootwrap error with L3 agent
>
> Status in OpenStack Quantum (virtual network service):
> Fix Released
> Status in quantum folsom series:
> Fix Released
> Status in “quantum” package in Ubuntu:
> Fix Released
> Status in “quantum” source package in Precise:
> Confirmed
> Status in “quantum” source package in Quantal:
> Fix Committed
>
> Bug description:
> I am seeing the following error in /var/log/quantum/l3_agent.log:
>
> 2012-10-22 09:00:48 DEBUG [quantum.agent.linux.utils] Running
> command: sudo /usr/bin/quantum-rootwrap /etc/quantum/rootwrap.conf
> /sbin/iptables-save -t filter
> 2012-10-22 09:00:48 DEBUG [quantum.agent.linux.utils]
> Command: ['sudo', '/usr/bin/quantum-rootwrap',
> '/etc/quantum/rootwrap.conf', '/sbin/iptables-save', '-t', 'filter']
> Exit code: 99
> Stdout: 'Unauthorized command: /sbin/iptables-save -t filter\n'
> Stderr: ''
> 2012-10-22 09:00:48 ERROR [quantum.agent.l3_agent] Error running
> l3_nat daemon_loop
> Traceback (most recent call last):
> File "/usr/lib/python2.7/dist-packages/quantum/agent/l3_agent.py",
> line 170, in daemon_loop
> self.do_single_loop()
> File "/usr/lib/python2.7/dist-packages/quantum/agent/l3_agent.py",
> line 227, in do_single_loop
> self.process_router(ri)
> File "/usr/lib/python2.7/dist-packages/quantum/agent/l3_agent.py",
> line 300, in process_router
> self.external_gateway_added(ri, ex_gw_port, internal_cidrs)
> File "/usr/lib/python2.7/dist-packages/quantum/agent/l3_agent.py",
> line 398, in external_gateway_added
> ri.iptables_manager.apply()
> File
> "/usr/lib/python2.7/dist-packages/quantum/agent/linux/iptables_manager.py",
> line 282, in apply
> root_helper=self.root_helper))
> File "/usr/lib/python2.7/dist-packages/quantum/agent/linux/utils.py",
> line 55, in execute
> raise RuntimeError(m)
> RuntimeError:
> Command: ['sudo', '/usr/bin/quantum-rootwrap',
> '/etc/quantum/rootwrap.conf', '/sbin/iptables-save', '-t', 'filter']
> Exit code: 99
> Stdout: 'Unauthorized command: /sbin/iptables-save -t filter\n'
> Stderr: ''
>
> If I run "sudo /usr/bin/quantum-rootwrap /etc/quantum/rootwrap.conf
> /sbin/iptables-save -t filter" it does indeed give me an Unauthorized
> command error.
>
> If I run "sudo /usr/bin/quantum-rootwrap /etc/quantum/rootwrap.conf
> iptables-save -t filter" (without the /sbin/) it works OK.
> Otherwise, I don't see errors in the log.
>
> Is this a problem?
>
> Thanks,
> ...

Read more...

Mark McLoughlin (markmc)
tags: removed: in-stable-folsom
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (3.8 KiB)

This bug was fixed in the package quantum - 2012.2.1-0ubuntu1

---------------
quantum (2012.2.1-0ubuntu1) quantal-proposed; urgency=low

  * Resynchronize with stable/folsom (1e774867) (LP: #1085255):
    - [aeabb42] There are routing problems when the dnsmasq port does not come
      first in the routing table (LP: #1083238)
    - [04aab72] Quantum linux bridge not optimized with libvirt (LP: #1078210)
    - [ca7fc10] getting quotas from database has severe performance implications
      (LP: #1075369)
    - [66605e8] failed to update an external network into non external network
      (LP: #1083387)
    - [c60051a] Quantum test suite leaks memory like a sieve (LP: #1065276)
    - [3179dfc] clear_db() does incomplete db teardown (LP: #1080988)
    - [c1e19d7] Unauthorized command: cat /proc/None/cmdline (LP: #1077651)
    - [af9e076] At times a instance will not receive an IP address from the DHCP
      agent (LP: #1081664)
    - [e0d1a7d] allow multiple floating-ip on single port if they use different
      fixed ips and/or external nets (LP: #1057844)
    - [8471d79] Delete port fails to gateway ip (LP: #1079980)
    - [aca8b4a] fixed_ip allocation which is not included within
      allocation_pools makes error when delete port or re-create port
      (LP: #1077292)
    - [eacc9d3] Mapping same bridge to different phyiscal networks succeed
      (LP: #1067669)
    - [51b4c82] python-quantum: not region aware (LP: #1080793)
    - [6f0a486] delete floatingip should be in one transaction to delete port
      (LP: #1080516)
    - [db6cda7] Remove qpid configuration variables no longer supported
    - [a112840] Allow NVP plugin to use per-tenant quota extension
    - [82b1a55] Quantum service does not restart after reboot (LP: #1073999)
    - [c01a839] There are some cases that L3 API with an invalid parameter
      returns 500. (LP: #1064765)
    - [26b383f] external network can be plugged also as internal network for one
      router (LP: #1053633)
    - [49f649c] There is a lot of cases that API with an invalid parameter
      returns 500. (LP: #1062046)
    - [4546a18] When create subnet, you con set up the value as cidr (the value
      isn't cidr form). (LP: #1067959)
    - [9ba453a] killfilter should handle updated/deleted executables
      (LP: #1073768)
    - [7c8a55c] a port which is not able to delete is made when floatingip
      create fails. (LP: #1064748)
    - [c9b84cf] Linux bridge port update causes exception (LP: #1072713)
    - [cb57932] I can't add interface to router, if there is another port in
      non-shared network of other tenant (LP: #1057558)
    - [574e278] Ryu plugin does not support Security Groups (LP: #1059393)
    - [607f486] tap device added to integration bridge without tag
      (LP: #1064070)
    - [21a0fdf] L3 agent external network flag (LP: #1056720)
    - [5cbaff4] router create with external_gateway_info fails with 500 always.
      (LP: #1064235)
    - [63b81f6] l3 db operations failed in multiple transactions (LP: #1070335)
    - [bff17fb] Ensure that the SqlSoup import is still supported.
    - [e091a29] l3_nat_agent was renamed to l3_agent
    - [9030969] remove default value of 'local_ip' of 10...

Read more...

Changed in quantum (Ubuntu Quantal):
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in quantum:
milestone: grizzly-1 → 2013.1
Revision history for this message
Steve Langasek (vorlon) wrote :

The Precise Pangolin has reached end of life, so this bug will not be fixed for that release

Changed in quantum (Ubuntu Precise):
status: Confirmed → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Related questions

Remote bug watches

Bug watches keep track of this bug in other bug trackers.