Registering big images fails (21G)

[Olivier Gagnon]

Hi,
I'm trying to "euca-register" an image file (.img) of 20G.

I can make "euca-bundle-image" and "euca-upload-image" without problem.

When I attempt to "euca-register", it takes so time and I get the following error:

-------------------------
         4 0 8
path=/services/Eucalyptus/?AWSAccessKeyId=nqyz2f90O0glmpdBHLx3M9bBBhlBonefkMQ&Action=RegisterImage&ImageLocation=winxp20g%2Fwinxp64-20G.img.manifest.xml&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2010-01-27T05%3A14%3A20&Version=2009-04-04&Signature=6QjrnWVTMxX7VOlzp2eGTeD5H/YWrC0Paa7Wf39rNzE%3D
Failure: 408 Request Timeout

-------------------------
-------------------------
         4 0 8
path=/services/Eucalyptus/?AWSAccessKeyId=nqyz2f90O0glmpdBHLx3M9bBBhlBonefkMQ&Action=RegisterImage&ImageLocation=winxp20g%2Fwinxp64-20G.img.manifest.xml&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2010-01-27T05%3A14%3A20&Version=2009-04-04&Signature=6QjrnWVTMxX7VOlzp2eGTeD5H/YWrC0Paa7Wf39rNzE%3D
Failure: 408 Request Timeout

-------------------

I look at the log and I get:
--------
Root Exception stack trace:
java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOfRange(Arrays.java:3221)
        at java.lang.String.<init>(String.java:233)

and

00:11:51 [LocalDispatcher:Image.2] ERROR com.eucalyptus.util.EucalyptusCloudException: Java heap space
com.eucalyptus.util.EucalyptusCloudException: Java heap space
        at com.eucalyptus.ws.client.LocalDispatcher.send(LocalDispatcher.java:43)
        at com.eucalyptus.images.util.WalrusUtil.verifyManifestIntegrity(WalrusUtil.java:162)
        at edu.ucsb.eucalyptus.cloud.ws.ImageManager.register(ImageManager.java:298)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)

---------------
I use Ubuntu 9.10-server
xxx@cloud-cc:~$ dpkg -la | grep euca
ii euca2ools 1.0+bzr20091007-0ubuntu1.1 managing cloud instances for Eucalyptus
ii eucalyptus-cc 1.6~bzr931-0ubuntu7.4 Elastic Utility Computing Architecture - Clu
ii eucalyptus-cloud 1.6~bzr931-0ubuntu7.4 Elastic Utility Computing Architecture - Clo
ii eucalyptus-common 1.6~bzr931-0ubuntu7.4 Elastic Utility Computing Architecture - Com
ii eucalyptus-gl 1.6~bzr931-0ubuntu7.4 Elastic Utility Computing Architecture - Log
ii eucalyptus-java-common 1.6~bzr931-0ubuntu7.4 Elastic Utility Computing Architecture - Com
ii eucalyptus-sc 1.6~bzr931-0ubuntu7.4 Elastic Utility Computing Architecture - Sto
ii eucalyptus-walrus 1.6~bzr931-0ubuntu7.4 Elastic Utility Computing Architecture - Wal
ii libeucalyptus-commons-ext-java 0.4.2-0ubuntu1 Eucalyptus commons external Java library
xxx@cloud-cc:~$

-------------

From what I understand, it can't parse the manifest.xml (which is 145K). I tried to add heap memory to the JVM in /etc/init/eucalyptus.conf by adding the option "-Xmx2048m" to $opts. The only thing it did is to postpone the error.

Thank you.

[Olivier Gagnon (procule)]

03:05:09 [ImageManager:Image.9] ERROR com.eucalyptus.util.EucalyptusCloudException: Internal Error.
03:05:09 [DefaultServiceExceptionStrategy:Image.9] ERROR
********************************************************************************
Message : Component that caused exception is: Image. Message payload is of type: RegisterImageType
Type : org.mule.api.service.ServiceException
Code : MULE_ERROR--2
Payload : <?xml version="1.0" encoding="UTF-8"?>
<euca:RegisterImage xmlns:euca="http://msgs.eucalyptus.ucsb.edu">
  <euca:VmImageMessage>
    <euca:correlationId>a71f4698-e681-4776-8727-5c8f2eb6131f</euca:correlationId>
    <euca:userId>ogagnon</euca:userId>
    <euca:effectiveUserId>eucalyptus</euca:effectiveUserId>
  </euca:VmImageMessage>
  <euca:imageLocation>win20/winxp64-20G.img.manifest.xml</euca:imageLocation>
</euca:RegisterImage>
JavaDoc : http://mule.mulesource.org/docs/apidocs/org/mule/api/service/ServiceException.html
********************************************************************************
Exception stack is:
1. Image registration failed because the manifest referenced is invalid or unavailable. (com.eucalyptus.util.EucalyptusCloudException)
  edu.ucsb.eucalyptus.cloud.ws.ImageManager:300 (null)
2. Component that caused exception is: Image. Message payload is of type: RegisterImageType (org.mule.api.service.ServiceException)
  org.mule.component.DefaultLifecycleAdapter:214 (http://mule.mulesource.org/docs/apidocs/org/mule/api/service/ServiceException.html)
********************************************************************************
Root Exception stack trace:
com.eucalyptus.util.EucalyptusCloudException: Image registration failed because the manifest referenced is invalid or unavailable.
        at edu.ucsb.eucalyptus.cloud.ws.ImageManager.register(ImageManager.java:300)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.mule.model.resolvers.AbstractEntryPointResolver.invokeMethod(AbstractEntryPointResolver.java:147)
        at org.mule.model.resolvers.ReflectionEntryPointResolver.invoke(ReflectionEntryPointResolver.java:127)
        at org.mule.model.resolvers.DefaultEntryPointResolverSet.invoke(DefaultEntryPointResolverSet.java:50)
        at org.mule.component.DefaultLifecycleAdapter.intercept(DefaultLifecycleAdapter.java:202)
        at org.mule.component.AbstractJavaComponent.invokeComponentInstance(AbstractJavaComponent.java:82)
        at org.mule.component.AbstractJavaComponent.doOnCall(AbstractJavaComponent.java:73)
        at org.mule.component.AbstractComponent.onCall(AbstractComponent.java:87)
        at org.mule.model.seda.SedaService$ComponentStageWorker.run(SedaService.java:533)
        at org.mule.work.WorkerContext.run(WorkerContext.java:310)
        at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:...

[Olivier Gagnon (procule)]

I've added additional logs.

Also, i've updated to eucalyptus 1.6.1, it didn't help.

xxx@cloud-cc:~$ dpkg -la | grep euca
ii euca2ools 1.1-1 managing cloud instances for Eucalyptus
ii eucalyptus-cc 1.6.1~bzr1066-0euca1 Elastic Utility Computing Architecture - Clu
ii eucalyptus-cloud 1.6.1~bzr1066-0euca1 Elastic Utility Computing Architecture - Clo
ii eucalyptus-common 1.6.1~bzr1066-0euca1 Elastic Utility Computing Architecture - Com
ii eucalyptus-gl 1.6.1~bzr1066-0euca1 Elastic Utility Computing Architecture - Log
ii eucalyptus-java-common 1.6.1~bzr1066-0euca1 Elastic Utility Computing Architecture - Com
ii eucalyptus-sc 1.6.1~bzr1066-0euca1 Elastic Utility Computing Architecture - Sto
ii eucalyptus-walrus 1.6.1~bzr1066-0euca1 Elastic Utility Computing Architecture - Wal
ii libapache2-mod-axis2c 1.6.0-1euca8 Apache web services engine - apache module
ii libaxis2c0 1.6.0-1euca8 Apache web services engine - Runtime
ii libeucalyptus-commons-ext-java 0.4.2-0ubuntu1 Eucalyptus commons external Java library

[Olivier Gagnon (procule)]

I've dumped the heap memory and ran Eclipse Memory Analyzer on it.

Here is a snapshot of Eclipse.

[chris grzegorczyk (chris-grze)]

What is the size of the manifest.xml file? Is it, by chance, 100k+?

If so, it looks like you might be hitting this bug: https://bugs.launchpad.net/ubuntu/+source/eucalyptus/+bug/470355

[Olivier Gagnon (procule)]

As stated in the OM, yes: it is 145K.

I've looked into your link and the rev. 1068.1.28 of the 1.6.2 branch. That's exactly where I thought the bug was.

That's quite a High importance bug since you can't register big images. I'm trying to figure out to fix that since 1 month.

I'll try to update to a revision > 1068.1.28 and tell if it fixes the thing.

[Olivier Gagnon (procule)]

Oh, by High importance, I mean that it should be put into Ubuntu 9.10 updates.

[chris grzegorczyk (chris-grze)]

Good to hear that this was the issue. I am marking this as a duplicate of 470355.

As for updating the 9.10 package, it would be best to comment on the other bug report as it is linked/tracked against the package.

https://bugs.launchpad.net/ubuntu/+source/eucalyptus/+bug/470355

thanks.