OpenStack Compute (Nova)

linux_net.ensure_metadata_ip() fails on precise

Reported by Adam Gandelman on 2012-02-08
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Undecided
Adam Gandelman
nova (Ubuntu)
High
Unassigned

Bug Description

Now that nova-rootwrap is returning real return codes, ensure_metadata_ip() is failing on return code 2 when creating the metadata ip, because it already exists:

2012-02-08 15:30:13,309 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap ip addr add 169.254.169.254/32 scope link dev lo from (pid=8942) execute /usr/lib/python2.7/dist-packages/nova/utils.py:208
2012-02-08 15:30:13,339 DEBUG nova.utils [-] Result was 2 from (pid=8942) execute /usr/lib/python2.7/dist-packages/nova/utils.py:224
2012-02-08 15:30:13,340 CRITICAL nova [-] Unexpected error while running command.
Command: sudo nova-rootwrap ip addr add 169.254.169.254/32 scope link dev lo
Exit code: 2
Stdout: ''
Stderr: 'RTNETLINK answers: File exists\n'
(nova): TRACE: Traceback (most recent call last):
(nova): TRACE: File "/usr/bin/nova-network", line 49, in <module>
(nova): TRACE: service.wait()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/service.py", line 414, in wait
(nova): TRACE: _launcher.wait()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/service.py", line 134, in wait
(nova): TRACE: service.wait()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 166, in wait
(nova): TRACE: return self._exit_event.wait()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
(nova): TRACE: return hubs.get_hub().switch()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 177, in switch
(nova): TRACE: return self.greenlet.switch()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in main
(nova): TRACE: result = function(*args, **kwargs)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/service.py", line 104, in run_server
(nova): TRACE: server.start()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/service.py", line 164, in start
(nova): TRACE: self.manager.init_host()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 1558, in init_host
(nova): TRACE: self.l3driver.initialize()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/l3.py", line 81, in initialize
(nova): TRACE: linux_net.ensure_metadata_ip()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 479, in ensure_metadata_ip
(nova): TRACE: run_as_root=True, check_exit_code=[0, 254])
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 839, in _execute
(nova): TRACE: return utils.execute(*cmd, **kwargs)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 232, in execute
(nova): TRACE: cmd=' '.join(cmd))
(nova): TRACE: ProcessExecutionError: Unexpected error while running command.
(nova): TRACE: Command: sudo nova-rootwrap ip addr add 169.254.169.254/32 scope link dev lo
(nova): TRACE: Exit code: 2
(nova): TRACE: Stdout: ''
(nova): TRACE: Stderr: 'RTNETLINK answers: File exists\n'
(nova): TRACE:

There seems to have been a change upstream wrt return codes from iproute:

root@ip-10-252-15-139:~# dpkg -l | grep iproute
ii iproute 20110629-1 networking and traffic control tools
root@ip-10-252-15-139:~# sudo ip addr add 169.254.169.254/32 scope link dev lo ; echo $?
RTNETLINK answers: File exists
254

Processing triggers for man-db ...
Setting up iproute (20111117-1ubuntu1) ...
root@ip-10-252-15-139:~# sudo ip addr add 169.254.169.254/32 scope link dev lo ; echo $?
RTNETLINK answers: File exists
2

Update, the correct upstream commit:
http://git.kernel.org/?p=linux/kernel/git/shemminger/iproute2.git;a=commit;h=7397944de6c11519a5951fc1bcff20225e71c4bd

Seems like a crazy change, but in any case, precise is now on this version and nova-network should check for both 2 and 254.

description: updated

Fix proposed to branch: master
Review: https://review.openstack.org/3934

Changed in nova:
assignee: nobody → Adam Gandelman (gandelman-a)
status: New → In Progress

Reviewed: https://review.openstack.org/3934
Committed: http://github.com/openstack/nova/commit/4804690ba401a23d62df489875da6c546d3224bb
Submitter: Jenkins
Branch: master

commit 4804690ba401a23d62df489875da6c546d3224bb
Author: Adam Gandelman <email address hidden>
Date: Wed Feb 8 14:16:08 2012 -0800

    linux_net: Also ignore shell error 2 from ip addr

    Ignores error code 2 from 'ip addr {add, del}' which, in recent
    versions of iproute2, means the address has already been removed
    or added.

    Fixes bug 929127

    Change-Id: I09274454847ff834ac3da0e38022b30a7a7f1676

Changed in nova:
status: In Progress → Fix Committed
Dave Walker (davewalker) on 2012-02-11
Changed in nova (Ubuntu):
status: New → Fix Committed
importance: Undecided → High
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package nova - 2012.1~e4~20120217.12709-0ubuntu1

---------------
nova (2012.1~e4~20120217.12709-0ubuntu1) precise; urgency=low

  [ Dave Walker (Daviey) ]
  * New upstream snapshot
  * debian/patches/temp_fix_linux_net.patch:
    - Dropped, applied upstream. LP: #929127
  * debian/patches/libvirt-use-console-pipe.patch:
    - Rebased against latest trunk

  [ Chuck Short ]
  * debian/nova.conf: Re-enable default iscsi_helper.
  * debian/nova.conf: More fixups.
  * debian/control: Dont depend and conflicts on nova-compute-
    hypervisor. (LP: #923681)
  * debian/patches/libvirt-us-console-pipe.patch: Refreshed.
  * Temporarily disable console patch. (LP: #932787)
  * New usptream version.
 -- Chuck Short <email address hidden> Fri, 17 Feb 2012 10:59:59 -0500

Changed in nova (Ubuntu):
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2012-02-29
Changed in nova:
milestone: none → essex-4
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2012-04-05
Changed in nova:
milestone: essex-4 → 2012.1
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers