Boto fails to boot instances with a security group

[Koji]

Some of our users are having an issue when booting instances on OpenStack with using Boto. It seems Boto is having an issue with Nova when it assigns a security group at booting instances.

Here's a simple example to make the issue happens.

image = conn.get_image("ami-XXXXXX")
for a in range (20):
    image.run(instance_type="m1.small", key_name="mykey1", security_groups=["secgroup1"])

Several instances fail to boot with error status.

But the follows is fine.

image = conn.get_image("ami-XXXXXX")
for a in range (20):
    image.run(instance_type="m1.small", key_name="mykey1")

Here's the version info.
Nova: 2012.1.3+stable-20130423-e52e6912-0ubuntu1.1
Boto: 2.9.7

Thanks and best regards,
Koji

[Koji (kj-tanaka)]

I see this error log on the nova-compute.

2013-07-12 14:50:13 ERROR nova.compute.manager [req-33af7629-c8ba-45b4-83a1-45b9c70afa77 2655a21658cd4b99a14bae40698db00b 461884eef90047fbb4eb9ec92f22a1e3] [instance: e01cea61-1cdf-4350-8e8f-3fd1129a578f] Instance failed network setup
2013-07-12 14:50:13 TRACE nova.compute.manager [instance: e01cea61-1cdf-4350-8e8f-3fd1129a578f] Traceback (most recent call last):
2013-07-12 14:50:13 TRACE nova.compute.manager [instance: e01cea61-1cdf-4350-8e8f-3fd1129a578f] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 584, in _allocate_network
2013-07-12 14:50:13 TRACE nova.compute.manager [instance: e01cea61-1cdf-4350-8e8f-3fd1129a578f] requested_networks=requested_networks)
2013-07-12 14:50:13 TRACE nova.compute.manager [instance: e01cea61-1cdf-4350-8e8f-3fd1129a578f] File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 178, in allocate_for_instance
2013-07-12 14:50:13 TRACE nova.compute.manager [instance: e01cea61-1cdf-4350-8e8f-3fd1129a578f] 'args': args})
2013-07-12 14:50:13 TRACE nova.compute.manager [instance: e01cea61-1cdf-4350-8e8f-3fd1129a578f] File "/usr/lib/python2.7/dist-packages/nova/rpc/__init__.py", line 68, in call
2013-07-12 14:50:13 TRACE nova.compute.manager [instance: e01cea61-1cdf-4350-8e8f-3fd1129a578f] return _get_impl().call(context, topic, msg, timeout)
2013-07-12 14:50:13 TRACE nova.compute.manager [instance: e01cea61-1cdf-4350-8e8f-3fd1129a578f] File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 674, in call
2013-07-12 14:50:13 TRACE nova.compute.manager [instance: e01cea61-1cdf-4350-8e8f-3fd1129a578f] return rpc_amqp.call(context, topic, msg, timeout, Connection.pool)
2013-07-12 14:50:13 TRACE nova.compute.manager [instance: e01cea61-1cdf-4350-8e8f-3fd1129a578f] File "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 343, in call
2013-07-12 14:50:13 TRACE nova.compute.manager [instance: e01cea61-1cdf-4350-8e8f-3fd1129a578f] rv = list(rv)
2013-07-12 14:50:13 TRACE nova.compute.manager [instance: e01cea61-1cdf-4350-8e8f-3fd1129a578f] File "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 304, in __iter__
2013-07-12 14:50:13 TRACE nova.compute.manager [instance: e01cea61-1cdf-4350-8e8f-3fd1129a578f] self.done()
2013-07-12 14:50:13 TRACE nova.compute.manager [instance: e01cea61-1cdf-4350-8e8f-3fd1129a578f] File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-07-12 14:50:13 TRACE nova.compute.manager [instance: e01cea61-1cdf-4350-8e8f-3fd1129a578f] self.gen.next()
2013-07-12 14:50:13 TRACE nova.compute.manager [instance: e01cea61-1cdf-4350-8e8f-3fd1129a578f] File "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 301, in __iter__
2013-07-12 14:50:13 TRACE nova.compute.manager [instance: e01cea61-1cdf-4350-8e8f-3fd1129a578f] self._iterator.next()
2013-07-12 14:50:13 TRACE nova.compute.manager [instance: e01cea61-1cdf-4350-8e8f-3fd1129a578f] File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 572, in iterconsume
2013-07-12 14:50:13 TRACE nova.compute.manager [instance: e01cea61-1cdf-4350-8e8f-3fd1129a578f] yield self.ensure(_error_call...

[Koji (kj-tanaka)]

Also, we use auto_assign_floating_ip which is possibly related with this issue, not sure though...

And also, there are two bad things after the issues happen.

1. Deleting these instances takes very long, and need to execute nova delete command several times.
2. The floating IPs are not released properly.

"nova-manage floating list" shows the problematic ip likethis.
 :
461884eef90047fbb4eb9ec92f22a1e3 <ip address> None nova ethX
 :
"nova floating-ip-list" don't show them up.

I think the ip addresses should go back to this after deletion.
 :
None <ip address> None nova ethX
 :

[Sean Dague (sdague)]

This is definitely not clear what the fix it, put it back to Confirmed