Comment 22 for bug 855030

Revision history for this message
David Kranz (david-kranz) wrote :

OK. I just did it again. I am not sure what "git commit tip" means. The running out of addresses was a one-time issue I had before understanding that the leased IP addresses for the vms are not returned immediately when the vms are terminated.
I am assuming that this kind of error and stacktrace in the log would be considered a bug even if it does not cause any obvious misbehavior immediately.

 I used the current version of devstack but with a modified stack.sh from last week. I modified it to log to files in /var/log/nova instead of stdout so you have to create that directory before running. I used one compute node running api,network,compute and one controller node running everything. After running the modified stack.sh I exported the shell variables in exercise.sh, modified the min count in nova client to create 10 vms, and did

nova boot --flavor 1 --image 3 testvms

In this case the vms seen to have started but there is the AMQP error stacktrace in the log. When this happens, sometimes all the vms come up and sometimes not. I have attached:

1. nova-network log with stacktrace
2. modified stack.sh
3. localrc for both controller and compute
4. nova.conf (even though it is generated)

(nova.rpc): TRACE: Traceback (most recent call last):
(nova.rpc): TRACE: File "/opt/stack/nova/nova/rpc/impl_kombu.py", line 628, in _process_data
(nova.rpc): TRACE: ctxt.reply(None, None)
(nova.rpc): TRACE: File "/opt/stack/nova/nova/rpc/impl_kombu.py", line 673, in reply
(nova.rpc): TRACE: msg_reply(self.msg_id, *args, **kwargs)
(nova.rpc): TRACE: File "/opt/stack/nova/nova/rpc/impl_kombu.py", line 781, in msg_reply
(nova.rpc): TRACE: conn.direct_send(msg_id, msg)
(nova.rpc): TRACE: File "/opt/stack/nova/nova/rpc/impl_kombu.py", line 562, in __exit__
(nova.rpc): TRACE: self._done()
(nova.rpc): TRACE: File "/opt/stack/nova/nova/rpc/impl_kombu.py", line 547, in _done
(nova.rpc): TRACE: self.connection.reset()
(nova.rpc): TRACE: File "/opt/stack/nova/nova/rpc/impl_kombu.py", line 382, in reset
(nova.rpc): TRACE: self.channel.close()
(nova.rpc): TRACE: File "/usr/local/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 201, in close
(nova.rpc): TRACE: super(Channel, self).close()
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py", line 194, in close
(nova.rpc): TRACE: (20, 41), # Channel.close_ok
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 97, in wait
(nova.rpc): TRACE: return self.dispatch_method(method_sig, args, content)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 115, in dispatch_method
(nova.rpc): TRACE: return amqp_method(self, args)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py", line 273, in _close
(nova.rpc): TRACE: (class_id, method_id))
(nova.rpc): TRACE: AMQPChannelException: (404, u"NOT_FOUND - no exchange '5b9a19d1b5e849d7ba749fada648a375' in vhost '/'", (60, 40), 'Channel.basic_publish')
(nova.rpc): TRACE:
2011-10-20 10:45:24,724 ERROR nova.rpc [-] Returning exception (404, u"NOT_FOUND - no exchange '5b9a19d1b5e849d7ba749fada648a375' in vhost '/'", (60, 40), 'Channel.basic_publish') to caller
2011-10-20 10:45:24,724 ERROR nova.rpc [-] ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/rpc/impl_kombu.py", line 628, in _process_data\n ctxt.reply(None, None)\n', ' File "/opt/stack/nova/nova/rpc/impl_kombu.py", line 673, in reply\n msg_reply(self.msg_id, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/rpc/impl_kombu.py", line 781, in msg_reply\n conn.direct_send(msg_id, msg)\n', ' File "/opt/stack/nova/nova/rpc/impl_kombu.py", line 562, in __exit__\n self._done()\n', ' File "/opt/stack/nova/nova/rpc/impl_kombu.py", line 547, in _done\n self.connection.reset()\n', ' File "/opt/stack/nova/nova/rpc/impl_kombu.py", line 382, in reset\n self.channel.close()\n', ' File "/usr/local/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 201, in close\n super(Channel, self).close()\n', ' File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py", line 194, in close\n (20, 41), # Channel.close_ok\n', ' File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 97, in wait\n return self.dispatch_method(method_sig, args, content)\n', ' File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 115, in dispatch_method\n return amqp_method(self, args)\n', ' File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py", line 273, in _close\n (class_id, method_id))\n', 'AMQPChannelException: (404, u"NOT_FOUND - no exchange \'5b9a19d1b5e849d7ba749fada648a375\' in vhost \'/\'", (60, 40), \'Channel.basic_publish\')\n']