Looks likeyou are running out of ips. Maybe you are running /27 which only has 32 addresses?
On Sep 21, 2011, at 6:22 AM, David Kranz wrote:
> ii python-kombu 1.0.4-2 AMQP Messaging Framework for Python
> ii rabbitmq-server 2.5.0-1ubuntu2 An AMQP server written in Erlang
>
> I added the carrot suggestion to nova.conf and it still fails about the
> same 1 in 20 or so vms not coming up but with a different error. I would
> be happy to help diagnose this if you have any other suggestions.
>
>
>
> 2011-09-21 09:17:24,546 DEBUG nova.network.manager [db549763-93f2-444f-bdd8-d4931b2fa6c7 6deb27e2-4b73-4d73-8196-6ccdbcee6859 kranzproje
> ct] network allocations for instance 108 from (pid=1096) allocate_for_instance /usr/lib/pymodules/python2.7/nova/network/manager.py:431
> 2011-09-21 09:17:24,705 ERROR nova.rpc [-] Exception during message handling
> (nova.rpc): TRACE: Traceback (most recent call last):
> (nova.rpc): TRACE: File "/usr/lib/pymodules/python2.7/nova/rpc/impl_carrot.py", line 285, in _process_data
> (nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args)
> (nova.rpc): TRACE: File "/usr/lib/pymodules/python2.7/nova/network/manager.py", line 216, in allocate_for_instance
> (nova.rpc): TRACE: ips = super(FloatingIP, self).allocate_for_instance(context, **kwargs)
> (nova.rpc): TRACE: File "/usr/lib/pymodules/python2.7/nova/network/manager.py", line 438, in allocate_for_instance
> (nova.rpc): TRACE: requested_networks=requested_networks)
> (nova.rpc): TRACE: File "/usr/lib/pymodules/python2.7/nova/network/manager.py", line 166, in _allocate_fixed_ips
> (nova.rpc): TRACE: vpn=vpn, address=address)
> (nova.rpc): TRACE: File "/usr/lib/pymodules/python2.7/nova/network/manager.py", line 609, in allocate_fixed_ip
> (nova.rpc): TRACE: instance_id)
> (nova.rpc): TRACE: File "/usr/lib/pymodules/python2.7/nova/db/api.py", line 343, in fixed_ip_associate_pool
> (nova.rpc): TRACE: instance_id, host)
> (nova.rpc): TRACE: File "/usr/lib/pymodules/python2.7/nova/db/sqlalchemy/api.py", line 101, in wrapper
> (nova.rpc): TRACE: return f(*args, **kwargs)
> (nova.rpc): TRACE: File "/usr/lib/pymodules/python2.7/nova/db/sqlalchemy/api.py", line 720, in fixed_ip_associate_pool
> (nova.rpc): TRACE: raise exception.NoMoreFixedIps()
> (nova.rpc): TRACE: NoMoreFixedIps: None
> (nova.rpc): TRACE:
> 2011-09-21 09:17:24,739 ERROR nova.rpc [-] Returning exception None to caller
> 2011-09-21 09:17:24,739 ERROR nova.rpc [-] ['Traceback (most recent call last):\n', ' File "/usr/lib/pymodules/python2.7/nova/rpc/impl_
>
> --
> You received this bug notification because you are a member of Nova Bug
> Team, which is subscribed to OpenStack Compute (nova).
> https://bugs.launchpad.net/bugs/855030
>
> Title:
> Encountering sporadic AMQPChannelException
>
> Status in OpenStack Compute (Nova):
> New
>
> Bug description:
> Running one controller and one compute, using nova-network version
> 2011.3~rc~20110909.r1155-0ub on Oneiric . Repeating the following:
>
> Create four instances, terminate them.
>
> This mostly works but one in every three or four times, one of the vms
> does not come up and I see this error in the nova-network log. This
> can occur both in the controller node (also running compute) or the
> compute node. (contents of nova.conf follows)
>
> 2011-09-20 13:22:59,295 DEBUG nova.utils [-] Attempting to grab semaphore "iptables" for method "apply"... from (pid=1082) inner /usr/lib/pymodules/python2.7/nova/utils.py:672
> 2011-09-20 13:22:59,295 DEBUG nova.utils [-] Attempting to grab file lock "iptables" for method "apply"... from (pid=1082) inner /usr/lib/pymodules/python2.7/nova/utils.py:677
> 2011-09-20 13:22:59,296 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-save -t filter from (pid=1082) execute /usr/lib/pymodules/python2.7/nova/utils.py:165
> 2011-09-20 13:22:59,311 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-restore from (pid=1082) execute /usr/lib/pymodules/python2.7/nova/utils.py:165
> 2011-09-20 13:22:59,350 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-save -t nat from (pid=1082) execute /usr/lib/pymodules/python2.7/nova/utils.py:165
> 2011-09-20 13:22:59,366 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-restore from (pid=1082) execute /usr/lib/pymodules/python2.7/nova/utils.py:165
> 2011-09-20 13:22:59,424 ERROR nova.rpc [-] Exception during message handling
> (nova.rpc): TRACE: Traceback (most recent call last):
> (nova.rpc): TRACE: File "/usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py", line 628, in _process_data
> (nova.rpc): TRACE: ctxt.reply(None, None)
> (nova.rpc): TRACE: File "/usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py", line 673, in reply
> (nova.rpc): TRACE: msg_reply(self.msg_id, *args, **kwargs)
> (nova.rpc): TRACE: File "/usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py", line 781, in msg_reply
> (nova.rpc): TRACE: conn.direct_send(msg_id, msg)
> (nova.rpc): TRACE: File "/usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py", line 562, in __exit__
> (nova.rpc): TRACE: self._done()
> (nova.rpc): TRACE: File "/usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py", line 547, in _done
> (nova.rpc): TRACE: self.connection.reset()
> (nova.rpc): TRACE: File "/usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py", line 382, in reset
> (nova.rpc): TRACE: self.channel.close()
> (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py", line 194, in close
> (nova.rpc): TRACE: (20, 41), # Channel.close_ok
> (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 97, in wait
> (nova.rpc): TRACE: return self.dispatch_method(method_sig, args, content)
> (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 115, in dispatch_method
> (nova.rpc): TRACE: return amqp_method(self, args)
> (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py", line 273, in _close
> (nova.rpc): TRACE: (class_id, method_id))
> (nova.rpc): TRACE: AMQPChannelException: (404, u"NOT_FOUND - no exchange '3ff1ba7e274a4ec2a2b0a217d7532c70' in vhost '/'", (60, 40), 'Channel.basic_publish')
> (nova.rpc): TRACE:
> 2011-09-20 13:22:59,451 ERROR nova.rpc [-] Returning exception (404, u"NOT_FOUND - no exchange '3ff1ba7e274a4ec2a2b0a217d7532c70' in vhost '/'", (60, 40), 'Channel.basic_publish') to caller
> 2011-09-20 13:22:59,452 ERROR nova.rpc [-] ['Traceback (most recent call last):\n', ' File "/usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py", line 628, in _process_data\n ctxt.reply(None, None)\n', ' File "/usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py", line 673, in reply\n msg_reply(self.msg_id, *args, **kwargs)\n', ' File "/usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py", line 781, in msg_reply\n conn.direct_send(msg_id, msg)\n', ' File "/usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py", line 562, in __exit__\n self._done()\n', ' File "/usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py", line 547, in _done\n self.connection.reset()\n', ' File "/usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py", line 382, in reset\n self.channel.close()\n', ' File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py", line 194, in close\n (20, 41), # Channel.close_ok\n', ' File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 97, in wait\n return self.dispatch_method(method_sig, args, content)\n', ' File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 115, in dispatch_method\n return amqp_method(self, args)\n', ' File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py", line 273, in _close\n (class_id, method_id))\n', 'AMQPChannelException: (404, u"NOT_FOUND - no exchange \'3ff1ba7e274a4ec2a2b0a217d7532c70\' in vhost \'/\'", (60, 40), \'Channel.basic_publish\')\n']
>
>
> I also see this at the same time in the rabbitmq log:
>
> =ERROR REPORT==== 20-Sep-2011::13:22:59 ===
> connection <0.1061.0>, channel 1 - error:
> {amqp_error,not_found,
> "no exchange '3ff1ba7e274a4ec2a2b0a217d7532c70' in vhost '/'",
> 'basic.publish'}
>
>
>
> --flagfile=/etc/nova/nova-compute.conf
> --use_deprecated_auth
> --dhcpbridge_flagfile=/etc/nova/nova.conf
> --dhcpbridge=/usr/bin/nova-dhcpbridge
> --sql_connection=mysql://nova:notnova@172.18.0.141/nova
> --s3_host=172.18.0.141
> --rabbit_host=172.18.0.141
> --glance_api_servers=172.18.0.141:9292
> --logdir=/var/log/nova
> --state_path=/var/lib/nova
> --lock_path=/var/lock/nova
> --verbose
> --ec2_url=http://172.18.0.141:8773/services/Cloud
> --dmz_cidr=172.18.0.141/32
> --fixed_range=172.18.77.0/27
> --network_size=8
> --flat_network_dhcp_start=172.18.7.6
> --image_service=nova.image.glance.GlanceImageService
> --bridge_interface=eth1
> --flat_network_bridge=br100
> --network_manager=nova.network.manager.FlatDHCPManager
> --public_interface=eth0
> --multi_host
> --osapi_host=172.18.0.141
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/nova/+bug/855030/+subscriptions
Looks likeyou are running out of ips. Maybe you are running /27 which only has 32 addresses?
On Sep 21, 2011, at 6:22 AM, David Kranz wrote:
> ii python-kombu 1.0.4-2 AMQP Messaging Framework for Python manager [db549763- 93f2-444f- bdd8-d4931b2fa6 c7 6deb27e2- 4b73-4d73- 8196-6ccdbcee68 59 kranzproje for_instance /usr/lib/ pymodules/ python2. 7/nova/ network/ manager. py:431 pymodules/ python2. 7/nova/ rpc/impl_ carrot. py", line 285, in _process_data context= ctxt, **node_args) pymodules/ python2. 7/nova/ network/ manager. py", line 216, in allocate_ for_instance for_instance( context, **kwargs) pymodules/ python2. 7/nova/ network/ manager. py", line 438, in allocate_ for_instance networks= requested_ networks) pymodules/ python2. 7/nova/ network/ manager. py", line 166, in _allocate_fixed_ips pymodules/ python2. 7/nova/ network/ manager. py", line 609, in allocate_fixed_ip pymodules/ python2. 7/nova/ db/api. py", line 343, in fixed_ip_ associate_ pool pymodules/ python2. 7/nova/ db/sqlalchemy/ api.py" , line 101, in wrapper pymodules/ python2. 7/nova/ db/sqlalchemy/ api.py" , line 720, in fixed_ip_ associate_ pool NoMoreFixedIps( ) pymodules/ python2. 7/nova/ rpc/impl_ /bugs.launchpad .net/bugs/ 855030 ption rc~20110909. r1155-0ub on Oneiric . Repeating the following: pymodules/ python2. 7/nova/ utils.py: 672 pymodules/ python2. 7/nova/ utils.py: 677 pymodules/ python2. 7/nova/ utils.py: 165 pymodules/ python2. 7/nova/ utils.py: 165 pymodules/ python2. 7/nova/ utils.py: 165 pymodules/ python2. 7/nova/ utils.py: 165 pymodules/ python2. 7/nova/ rpc/impl_ kombu.py" , line 628, in _process_data pymodules/ python2. 7/nova/ rpc/impl_ kombu.py" , line 673, in reply self.msg_ id, *args, **kwargs) pymodules/ python2. 7/nova/ rpc/impl_ kombu.py" , line 781, in msg_reply send(msg_ id, msg) pymodules/ python2. 7/nova/ rpc/impl_ kombu.py" , line 562, in __exit__ pymodules/ python2. 7/nova/ rpc/impl_ kombu.py" , line 547, in _done .reset( ) pymodules/ python2. 7/nova/ rpc/impl_ kombu.py" , line 382, in reset close() python2. 7/dist- packages/ amqplib/ client_ 0_8/channel. py", line 194, in close python2. 7/dist- packages/ amqplib/ client_ 0_8/abstract_ channel. py", line 97, in wait method( method_ sig, args, content) python2. 7/dist- packages/ amqplib/ client_ 0_8/abstract_ channel. py", line 115, in dispatch_method python2. 7/dist- packages/ amqplib/ client_ 0_8/channel. py", line 273, in _close ption: (404, u"NOT_FOUND - no exchange '3ff1ba7e274a4e c2a2b0a217d7532 c70' in vhost '/'", (60, 40), 'Channel. basic_publish' ) c2a2b0a217d7532 c70' in vhost '/'", (60, 40), 'Channel. basic_publish' ) to caller pymodules/ python2. 7/nova/ rpc/impl_ kombu.py" , line 628, in _process_data\n ctxt.reply(None, None)\n', ' File "/usr/lib/ pymodules/ python2. 7/nova/ rpc/impl_ kombu.py" , line 673, in reply\n msg_reply( self.msg_ id, *args, **kwargs)\n', ' File "/usr/lib/ pymodules/ python2. 7/nova/ rpc/impl_ kombu.py" , line 781, in msg_reply\n conn.direct_ send(msg_ id, msg)\n', ' File "/usr/lib/ pymodules/ python2. 7/nova/ rpc/impl_ kombu.py" , line 562, in __exit__\n self._done()\n', ' File "/usr/lib/ pymodules/ python2. 7/nova/ rpc/impl_ kombu.py" , line 547, in _done\n self.connection .reset( )\n', ' File "/usr/lib/ pymodules/ python2. 7/nova/ rpc/impl_ kombu.py" , line 382, in reset\n self.channel. close() \n', ' File "/usr/lib/ python2. 7/dist- packages/ amqplib/ client_ 0_8/channel. py", line 194, in close\n (20, 41), # Channel. close_ok\ n', ' File "/usr/lib/ python2. 7/dist- packages/ amqplib/ client_ 0_8/abstract_ channel. py", line 97, in wait\n return self.dispatch_ method( method_ sig, args, content)\n', ' File "/usr/lib/ python2. 7/dist- packages/ amqplib/ client_ 0_8/abstract_ channel. py", line 115, in dispatch_method\n return amqp_method(self, args)\n', ' File "/usr/lib/ python2. 7/dist- packages/ amqplib/ client_ 0_8/channel. py", line 273, in _close\n (class_id, method_id))\n', 'AMQPChannelExc eption: (404, u"NOT_FOUND - no exchange \'3ff1ba7e274a4 ec2a2b0a217d753 2c70\' in vhost \'/\'", (60, 40), \'Channel. basic_publish\ ')\n'] 2011::13: 22:59 === not_found, c2a2b0a217d7532 c70' in vhost '/'", /etc/nova/ nova-compute. conf d_auth flagfile= /etc/nova/ nova.conf /usr/bin/ nova-dhcpbridge n=mysql: //nova: notnova@ 172.18. 0.141/nova 172.18. 0.141 host=172. 18.0.141 api_servers= 172.18. 0.141:9292 /var/log/ nova path=/var/ lib/nova path=/var/ lock/nova 172.18. 0.141:8773/ services/ Cloud 172.18. 0.141/32 range=172. 18.77.0/ 27 network_ dhcp_start= 172.18. 7.6 service= nova.image. glance. GlanceImageServ ice interface= eth1 network_ bridge= br100 manager= nova.network. manager. FlatDHCPManager interface= eth0 host=172. 18.0.141 /bugs.launchpad .net/nova/ +bug/855030/ +subscriptions
> ii rabbitmq-server 2.5.0-1ubuntu2 An AMQP server written in Erlang
>
> I added the carrot suggestion to nova.conf and it still fails about the
> same 1 in 20 or so vms not coming up but with a different error. I would
> be happy to help diagnose this if you have any other suggestions.
>
>
>
> 2011-09-21 09:17:24,546 DEBUG nova.network.
> ct] network allocations for instance 108 from (pid=1096) allocate_
> 2011-09-21 09:17:24,705 ERROR nova.rpc [-] Exception during message handling
> (nova.rpc): TRACE: Traceback (most recent call last):
> (nova.rpc): TRACE: File "/usr/lib/
> (nova.rpc): TRACE: rval = node_func(
> (nova.rpc): TRACE: File "/usr/lib/
> (nova.rpc): TRACE: ips = super(FloatingIP, self).allocate_
> (nova.rpc): TRACE: File "/usr/lib/
> (nova.rpc): TRACE: requested_
> (nova.rpc): TRACE: File "/usr/lib/
> (nova.rpc): TRACE: vpn=vpn, address=address)
> (nova.rpc): TRACE: File "/usr/lib/
> (nova.rpc): TRACE: instance_id)
> (nova.rpc): TRACE: File "/usr/lib/
> (nova.rpc): TRACE: instance_id, host)
> (nova.rpc): TRACE: File "/usr/lib/
> (nova.rpc): TRACE: return f(*args, **kwargs)
> (nova.rpc): TRACE: File "/usr/lib/
> (nova.rpc): TRACE: raise exception.
> (nova.rpc): TRACE: NoMoreFixedIps: None
> (nova.rpc): TRACE:
> 2011-09-21 09:17:24,739 ERROR nova.rpc [-] Returning exception None to caller
> 2011-09-21 09:17:24,739 ERROR nova.rpc [-] ['Traceback (most recent call last):\n', ' File "/usr/lib/
>
> --
> You received this bug notification because you are a member of Nova Bug
> Team, which is subscribed to OpenStack Compute (nova).
> https:/
>
> Title:
> Encountering sporadic AMQPChannelExce
>
> Status in OpenStack Compute (Nova):
> New
>
> Bug description:
> Running one controller and one compute, using nova-network version
> 2011.3~
>
> Create four instances, terminate them.
>
> This mostly works but one in every three or four times, one of the vms
> does not come up and I see this error in the nova-network log. This
> can occur both in the controller node (also running compute) or the
> compute node. (contents of nova.conf follows)
>
> 2011-09-20 13:22:59,295 DEBUG nova.utils [-] Attempting to grab semaphore "iptables" for method "apply"... from (pid=1082) inner /usr/lib/
> 2011-09-20 13:22:59,295 DEBUG nova.utils [-] Attempting to grab file lock "iptables" for method "apply"... from (pid=1082) inner /usr/lib/
> 2011-09-20 13:22:59,296 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-save -t filter from (pid=1082) execute /usr/lib/
> 2011-09-20 13:22:59,311 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-restore from (pid=1082) execute /usr/lib/
> 2011-09-20 13:22:59,350 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-save -t nat from (pid=1082) execute /usr/lib/
> 2011-09-20 13:22:59,366 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-restore from (pid=1082) execute /usr/lib/
> 2011-09-20 13:22:59,424 ERROR nova.rpc [-] Exception during message handling
> (nova.rpc): TRACE: Traceback (most recent call last):
> (nova.rpc): TRACE: File "/usr/lib/
> (nova.rpc): TRACE: ctxt.reply(None, None)
> (nova.rpc): TRACE: File "/usr/lib/
> (nova.rpc): TRACE: msg_reply(
> (nova.rpc): TRACE: File "/usr/lib/
> (nova.rpc): TRACE: conn.direct_
> (nova.rpc): TRACE: File "/usr/lib/
> (nova.rpc): TRACE: self._done()
> (nova.rpc): TRACE: File "/usr/lib/
> (nova.rpc): TRACE: self.connection
> (nova.rpc): TRACE: File "/usr/lib/
> (nova.rpc): TRACE: self.channel.
> (nova.rpc): TRACE: File "/usr/lib/
> (nova.rpc): TRACE: (20, 41), # Channel.close_ok
> (nova.rpc): TRACE: File "/usr/lib/
> (nova.rpc): TRACE: return self.dispatch_
> (nova.rpc): TRACE: File "/usr/lib/
> (nova.rpc): TRACE: return amqp_method(self, args)
> (nova.rpc): TRACE: File "/usr/lib/
> (nova.rpc): TRACE: (class_id, method_id))
> (nova.rpc): TRACE: AMQPChannelExce
> (nova.rpc): TRACE:
> 2011-09-20 13:22:59,451 ERROR nova.rpc [-] Returning exception (404, u"NOT_FOUND - no exchange '3ff1ba7e274a4e
> 2011-09-20 13:22:59,452 ERROR nova.rpc [-] ['Traceback (most recent call last):\n', ' File "/usr/lib/
>
>
> I also see this at the same time in the rabbitmq log:
>
> =ERROR REPORT==== 20-Sep-
> connection <0.1061.0>, channel 1 - error:
> {amqp_error,
> "no exchange '3ff1ba7e274a4e
> 'basic.publish'}
>
>
>
> --flagfile=
> --use_deprecate
> --dhcpbridge_
> --dhcpbridge=
> --sql_connectio
> --s3_host=
> --rabbit_
> --glance_
> --logdir=
> --state_
> --lock_
> --verbose
> --ec2_url=http://
> --dmz_cidr=
> --fixed_
> --network_size=8
> --flat_
> --image_
> --bridge_
> --flat_
> --network_
> --public_
> --multi_host
> --osapi_
>
> To manage notifications about this bug go to:
> https:/