euca-run-instances should fail if you try to run more instances than you can (in a single security group)

[C de-Avillez]

Using the tamarind-based test rig, and ISO images from cdimage (20100323.2).

 I tried to run the maximum number of instances on the setup (using lucid-amd64-topol, 66 sessions) from a idling cloud. All failed.

Looking at the configuration, I found VNET_ADDRSPERNET="32". I then tried to start 31 instances from a clean system again: all instances are shown in Pending, then go to Terminated (may take about 10 minutes to happen).

Waited for the cache to clear, and tried starting 29 instances -- success. Tried adding more, got a message stating "FinishedVerify: Not enough addresses left in the network subnet assigned to requested group: default". Interesting that I saw no such message when I tried to start from idle either 66 or 31 sessions.

Anyway. Closed them via via 'euca-terminate-instances $(euca-describe-instances | egrep ^INSTANCE| awk '{print $2}')'.

Started 30 sessions from idle: 29 started, one was left in Pending for a while (then Terminated), no message like above displayed. When I tried to start one more session, I got the "not enough addresses..." message.

Seems related to bug 462140

[Etienne Goyer (etienne-goyer-outlands)]

Hi Carlos,

I think you are having the problem discussed at:

    https://answers.launchpad.net/ubuntu/+source/eucalyptus/+question/87145

[C de-Avillez (hggdh2)]

Hi Etyenne,

Yes, the basic symptom is the same. The difference is that *no* instances succeed when I run with a -n > 30. This is rather unexpected.

BTW, thank you, adjusted the bug title.

[Etienne Goyer (etienne-goyer-outlands)]

It is probably the same root cause, though. Does "euca-run-instances -n 30 ..." works if you increase the value of VNET_ADDRSPERNET and do a clean restart? If yes, then it's not really a bug, although I guess euca2ools should say something meaningful about why no instance are being started.

i am not sure what the correct behavior should be in this case. Should Eucalyptus actually run instances until it runs out of private address in the security group, or should it reject the entire request (like it seems to be doing here). My guess is that it should do the later, and return that it does not have resources to complete the request (or something along that line). The Eucalyptus people would know for sure what the intended behavior is supposed to be.

[C de-Avillez (hggdh2)]

I am not sure either. But we do have some options:

1. if rejecting all requests, it should print out a reason, and end immediately;
2. if accepting up to (VNET_ADDRSPERNET - 3), then it should print out messages for the refusals (and not even try to start them) (but I guess there is a timing issue here). This is what happens when you are already at the limit, and you try another euca-run-instance on the same group.

But to accept all requests and keep them all Pending for many minutes before marking all Terminated is plainly wrong. Note that this happens from an idle system, so (on the default) we *do* have 29 instances to run.

[Mathias Gug (mathiaz)]

Dan mentioned this was fixed in an upstream branch. But won't make it in the 1.6.2 branch (lucid).

[Andy Grimm (agrimm)]

This issue is now being tracked upstream at http://eucalyptus.atlassian.net/browse/EUCA-2687

Please watch that issue for further updates.