In SYSTEM mode + multicluster, instances do not run on multiple clusters

[Neil Soman]

The first request works just fine. Subsequent requests produce the following exception (only SYSTEM mode is affected).

                                         | ********************************************************************************
                                         | Message : Component that caused exception is: FinishedVerify. Message payload is of type: VmAllocationInfo
                                         | Type : org.mule.api.service.ServiceException
                                         | Code : MULE_ERROR--2
                                         | Payload : edu.ucsb.eucalyptus.cloud.VmAllocationInfo@3330290b
                                         | JavaDoc : http://mule.mulesource.org/docs/apidocs/org/mule/api/service/ServiceException.html
                                         | ********************************************************************************
                                         | Exception stack is:
                                         | 1. null (java.lang.NullPointerException)
                                         | java.util.concurrent.ConcurrentSkipListMap:645 (null)
                                         | 2. Internal Error. (com.eucalyptus.util.EucalyptusCloudException)
                                         | edu.ucsb.eucalyptus.cloud.ws.VmAdmissionControl:150 (null)
                                         | 3. Component that caused exception is: FinishedVerify. Message payload is of type: VmAllocationInfo (org.mule.api.service.ServiceException)
                                         | org.mule.component.DefaultLifecycleAdapter:214 (http://mule.mulesource.org/docs/apidocs/org/mule/api/service/ServiceException.html)
                                         | ********************************************************************************
                                         | Root Exception stack trace:
                                         | java.lang.NullPointerException
                                         | at java.util.concurrent.ConcurrentSkipListMap.comparable(ConcurrentSkipListMap.java:645)
                                         | at java.util.concurrent.ConcurrentSkipListMap.doPut(ConcurrentSkipListMap.java:890)
                                         | at java.util.concurrent.ConcurrentSkipListMap.putIfAbsent(ConcurrentSkipListMap.java:1921)
                                         | at java.util.concurrent.ConcurrentSkipListSet.add(ConcurrentSkipListSet.java:230)
                                         | at sun.reflect.GeneratedMethodAccessor47.invoke(Unknown Source)
                                         | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                                         | at java.lang.reflect.Method.invoke(Method.java:616)
                                         | at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:229)
                                         | at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:52)
                                         | at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
                                         | at edu.ucsb.eucalyptus.cloud.Network.allocateNetworkIndex(Extra.groovy:370)
                                         | at edu.ucsb.eucalyptus.cloud.SLAs.doNetworkAllocation(SLAs.java:175)
                                         | at edu.ucsb.eucalyptus.cloud.ws.VmAdmissionControl.evaluate(VmAdmissionControl.java:135)
                                         | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                                         | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
                                         | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                                         | at java.lang.reflect.Method.invoke(Method.java:616)
                                         | at org.mule.model.resolvers.AbstractEntryPointResolver.invokeMethod(AbstractEntryPointResolver.java:147)
                                         | at org.mule.model.resolvers.ReflectionEntryPointResolver.invoke(ReflectionEntryPointResolver.java:127)
                                         | at org.mule.model.resolvers.DefaultEntryPointResolverSet.invoke(DefaultEntryPointResolverSet.java:50)
                                         | at org.mule.component.DefaultLifecycleAdapter.intercept(DefaultLifecycleAdapter.java:202)
                                         | at org.mule.component.AbstractJavaComponent.invokeComponentInstance(AbstractJavaComponent.java:82)
                                         | at org.mule.component.AbstractJavaComponent.doOnCall(AbstractJavaComponent.java:73)
                                         | at org.mule.component.AbstractComponent.onCall(AbstractComponent.java:87)
                                         | at org.mule.model.seda.SedaService$ComponentStageWorker.run(SedaService.java:533)
                                         | at org.mule.work.WorkerContext.run(WorkerContext.java:310)
                                         | at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:665)
                                         | at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:690)
                                         | at java.lang.Thread.run(Thread.java:636)
                                         |
                                         | ********************************************************************************
14:05:44 ERROR tServiceExceptionStrategy | Message being processed is: org.mule.transport.DefaultMessageAdapter/org.mule.transport.DefaultMessageAdapter@1f90d293{id=02dfa36a-b2bc-11de-a216-7d03e54c2ef1, payload=edu.ucsb.eucalyptus.cloud.VmAllocationInfo, correlationId=02dfa36a-b2bc-11de-a216-7d03e54c2ef1, correlationGroup=-1, correlationSeq=-1, encoding=UTF-8, exceptionPayload=org.mule.message.DefaultExceptionPayload@5cf0c115}

[chris grzegorczyk (chris-grze)]

From the code:
./clc/modules/cluster-manager/src/main/java/edu/ucsb/eucalyptus/cloud/SLAs.java:169://TODO: verify this doesn't break static/system mode.

From the stack trace:
at edu.ucsb.eucalyptus.cloud.SLAs.doNetworkAllocation(SLAs.java:175)

I guess it does.

[Neil Soman (neilsoman)]

Yep, as you suspected, this affects SYSTEM mode + single cluster as well.

Only 1 run instance requests succeeds and further attempts result in the above exception.

[chris grzegorczyk (chris-grze)]

fix pending.

[chris grzegorczyk (chris-grze)]

------------------------------------------------------------
revno: 916
committer: decker <decker@personal-army>
branch nick: 1.6
timestamp: Wed 2009-10-07 01:36:56 -0700
message:
  fix handling of network indexes for SYSTEM mode lp:#444908
------------------------------------------------------------

[Neil Soman (neilsoman)]

Still doesn't quite work as expected. Reopening.

[Launchpad Janitor (janitor)]

This bug was fixed in the package eucalyptus - 1.6~bzr916-0ubuntu1

---------------
eucalyptus (1.6~bzr916-0ubuntu1) karmic; urgency=low

  * Merge upstream revision 916
  * This snapshot is expected to fix the following bugs:
    - LP: #443314 - fix system address selection
    - LP: #443325 - fix regression in heartbeat, fix NPE in system mode
    - LP: #444838 - fix meta-data service in multi-cluster
    - LP: #444908 - fix running instances on multi-cluster
    - LP: #445206 - fix remote bootstrap on walrus+sc
  * debian/eucalyptus-cc.eucalyptus-cc-publication.upstart,
    debian/local/euca_find_cluster.c: when doing the avahi-publish, use
    $CC_IP_ADDR rather than $(hostname) for the "name" of the service;
    use this ip address in the case where euca_find_cluster gets the
    link-local 169.254.169.254 address, LP: #445738

 -- Dustin Kirkland <email address hidden> Wed, 07 Oct 2009 20:01:56 -0500

[chris grzegorczyk (chris-grze)]

fix pending.

[chris grzegorczyk (chris-grze)]

------------------------------------------------------------
revno: 918
committer: decker <decker@personal-army>
branch nick: 1.6
timestamp: Thu 2009-10-08 01:08:31 -0700
message:
  fixes addressing conflicts between MANAGED and SYSTEM mode which cause run-instances to fail lp:#444908
------------------------------------------------------------

[Neil Soman (neilsoman)]

Confirmed! thanks.