Eucalyptus

problem with snapshot on walrus used for inter-cluster volume transfer

Bug #758712 reported by Cladmin on 2011-04-12

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Eucalyptus	New	Undecided	Neil Soman

Bug Description

Issue originally raised here:
http://ubuntuforums.org/showthread.php?p=10623367&posted=1#post10623367

I'm using Ubuntu 10.4.2 updated to the latest binaries, with Eucalyptus v1.6.2-0ubuntu30.4.

I have a cloud consisting of 5 servers in all: 1 clc/walrus, 1 cc/sc + 1 nc (cluster1-01), 1
cc/sc + 1 nc (cluster1-02).

ivacau@host0002:~$ sudo euca_conf --list-scs
registered storage controllers:
cluster1-02 192.168.121.5
cluster1-01 192.168.121.1
ivacau@host0002:~$ sudo euca_conf --list-clusters
registered clusters:
cluster1-02 192.168.121.5
cluster1-01 192.168.121.1
ivacau@host0002:~$ sudo euca_conf --list-nodes
registered nodes:
192.168.127.1 cluster1-01
192.168.128.1 cluster1-02

I am able to start instances in both clusters (availability zones) no problem. I'm also able to create volumes, associate volume to instance, create snapshot from a volume and subsequently create a volume from the snapshot, all within one of the two clusters.

ivacau@host0002:~$ euca-describe-volumes
VOLUME vol-5F880657 1 snap-5F170646 cluster1-02 available 2011-03-25T08:10:26.986Z
VOLUME vol-5F970659 1 cluster1-02 available 2011-03-25T07:45:44.709Z
VOLUME vol-5F40064C 1 snap-58DE0616 cluster1-01 available 2011-03-25T04:49:04.251Z
VOLUME vol-5F0B064A 1 cluster1-01 available 2011-03-25T03:00:16.941Z
VOLUME vol-6000065F 1 cluster1-01 available 2011-03-25T04:11:33.592Z

However when I attempt to create a volume (say in cluster1-02) from a snapshot that originally resided in the other cluster (say cluster1-01), I get the following error:

ivacau@host0002:~$ euca-create-volume --snapshot snap-58DE0616 -z cluster1-02
Volume: Error while communicating with Storage Controller: CreateStorageVolume:Internal Error.

The same error is encountered if I invert the source and destination availability zone.

On the clc/walrus server, I can see the following exceptions in /var/log/eucalyptus/cloud-error.log:

23:56:06 [NioResponseHandler:Hashed wheel timer #1] ERROR Caught exception in asynchronous response handler.
org.jboss.netty.handler.timeout.ReadTimeoutExcepti on
at org.jboss.netty.handler.timeout.ReadTimeoutHandler .<clinit>(ReadTimeoutHandler.java:59)
at com.eucalyptus.ws.util.ChannelUtil.addPipelineMoni tors(ChannelUtil.java:108)
at com.eucalyptus.cluster.handlers.AbstractClusterMes sageDispatcher.getPipeline(AbstractClusterMessageD ispatcher.java:109)
at com.eucalyptus.ws.client.NioBootstrap.connect(NioB ootstrap.java:177)
at com.eucalyptus.ws.client.NioBootstrap.connect(NioB ootstrap.java:163)
at com.eucalyptus.cluster.handlers.AbstractClusterMes sageDispatcher.write(AbstractClusterMessageDispatc her.java:129)
at com.eucalyptus.cluster.handlers.ClusterCertificate Handler.trigger(ClusterCertificateHandler.java:32)
at com.eucalyptus.cluster.handlers.ClusterCertificate Handler.fireEvent(ClusterCertificateHandler.java:5 7)
at com.eucalyptus.event.ReentrantListenerRegistry.fir eEvent(ReentrantListenerRegistry.java:80)
at com.eucalyptus.event.ReentrantListenerRegistry.fir eEvent(ReentrantListenerRegistry.java:65)
at com.eucalyptus.event.ListenerRegistry.fireEvent(Li stenerRegistry.java:69)
at com.eucalyptus.cluster.ClusterBootstrapper.load(Cl usterBootstrapper.java:140)
at com.eucalyptus.bootstrap.SystemBootstrapper.load(S ystemBootstrapper.java:175)
23:56:06 [RemoteDispatcher:Volume.10] ERROR com.eucalyptus.util.EucalyptusClusterException: Exception in NIO request.
com.eucalyptus.util.EucalyptusClusterException: Exception in NIO request.
at com.eucalyptus.ws.handlers.NioResponseHandler.getR esponse(NioResponseHandler.java:115)
at com.eucalyptus.ws.client.NioClient.send(NioClient. java:113)
at com.eucalyptus.ws.client.RemoteDispatcher.send(Rem oteDispatcher.java:110)
at com.eucalyptus.ws.client.ServiceDispatcher.send(Se rviceDispatcher.java:80)
at edu.ucsb.eucalyptus.cloud.ws.VolumeManager.CreateV olume(VolumeManager.java:170)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.mule.model.resolvers.AbstractEntryPointResolve r.invokeMethod(AbstractEntryPointResolver.java:147 )
at org.mule.model.resolvers.ReflectionEntryPointResol ver.invoke(ReflectionEntryPointResolver.java:178)
at org.mule.model.resolvers.DefaultEntryPointResolver Set.invoke(DefaultEntryPointResolverSet.java:50)
at org.mule.component.DefaultLifecycleAdapter.interce pt(DefaultLifecycleAdapter.java:202)
at org.mule.component.AbstractJavaComponent.invokeCom ponentInstance(AbstractJavaComponent.java:82)
at org.mule.component.AbstractJavaComponent.doOnCall( AbstractJavaComponent.java:73)
at org.mule.component.AbstractComponent.onCall(Abstra ctComponent.java:87)
at org.mule.model.seda.SedaService$ComponentStageWork er.run(SedaService.java:533)
at org.mule.work.WorkerContext.run(WorkerContext.java :310)
at edu.emory.mathcs.backport.java.util.concurrent.Thr eadPoolExecutor.runWorker(ThreadPoolExecutor.java: 1061)
at edu.emory.mathcs.backport.java.util.concurrent.Thr eadPoolExecutor$Worker.run(ThreadPoolExecutor.java :575)
at java.lang.Thread.run(Thread.java:636)
Caused by: org.jboss.netty.handler.timeout.ReadTimeoutExcepti on
at org.jboss.netty.handler.timeout.ReadTimeoutHandler .<clinit>(ReadTimeoutHandler.java:59)
at com.eucalyptus.ws.util.ChannelUtil.addPipelineMoni tors(ChannelUtil.java:108)
at com.eucalyptus.cluster.handlers.AbstractClusterMes sageDispatcher.getPipeline(AbstractClusterMessageD ispatcher.java:109)
at com.eucalyptus.ws.client.NioBootstrap.connect(NioB ootstrap.java:177)
at com.eucalyptus.ws.client.NioBootstrap.connect(NioB ootstrap.java:163)
at com.eucalyptus.cluster.handlers.AbstractClusterMes sageDispatcher.write(AbstractClusterMessageDispatc her.java:129)
at com.eucalyptus.cluster.handlers.ClusterCertificate Handler.trigger(ClusterCertificateHandler.java:32)
at com.eucalyptus.cluster.handlers.ClusterCertificate Handler.fireEvent(ClusterCertificateHandler.java:5 7)
at com.eucalyptus.event.ReentrantListenerRegistry.fir eEvent(ReentrantListenerRegistry.java:80)
at com.eucalyptus.event.ReentrantListenerRegistry.fir eEvent(ReentrantListenerRegistry.java:65)
at com.eucalyptus.event.ListenerRegistry.fireEvent(Li stenerRegistry.java:69)
at com.eucalyptus.cluster.ClusterBootstrapper.load(Cl usterBootstrapper.java:140)
at com.eucalyptus.bootstrap.SystemBootstrapper.load(S ystemBootstrapper.java:175)
23:56:06 [DefaultServiceExceptionStrategy:Volume.10] ERROR
************************************************** ******************************
Message : Component that caused exception is: Volume. Message payload is of type: CreateVolumeType
Type : org.mule.api.service.ServiceException
Code : MULE_ERROR--2
Payload : <?xml version="1.0" encoding="UTF-8"?>
<euca:CreateVolume xmlns:euca="http://msgs.eucalyptus.ucsb.edu">
<euca:BlockVolumeMessage>
<euca:correlationId>c5b404ad-f63c-4038-8c21-5fe0f282d81f</euca:correlationId>
<euca:userId>admin</euca:userId>
<euca:effectiveUserId>eucalyptus</euca:effectiveUserId>
</euca:BlockVolumeMessage>
<euca:snapshotId>snap-58DE0616</euca:snapshotId>
<euca:availabilityZone>cluster1-02</euca:availabilityZone>
</euca:CreateVolume>
JavaDoc : http://mule.mulesource.org/docs/apid...Exception.html
************************************************** ******************************
Exception stack is:
1. Error while communicating with Storage Controller: CreateStorageVolume:Internal Error. (com.eucalyptus.util.EucalyptusCloudException)
edu.ucsb.eucalyptus.cloud.ws.VolumeManager:182 (null)
2. Component that caused exception is: Volume. Message payload is of type: CreateVolumeType (org.mule.api.service.ServiceException)
org.mule.component.DefaultLifecycleAdapter:214 (http://mule.mulesource.org/docs/apid...Exception.html)
************************************************** ******************************
Root Exception stack trace:
com.eucalyptus.util.EucalyptusCloudException: Error while communicating with Storage Controller: CreateStorageVolume:Internal Error.
at edu.ucsb.eucalyptus.cloud.ws.VolumeManager.CreateV olume(VolumeManager.java:182)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.mule.model.resolvers.AbstractEntryPointResolve r.invokeMethod(AbstractEntryPointResolver.java:147 )
at org.mule.model.resolvers.ReflectionEntryPointResol ver.invoke(ReflectionEntryPointResolver.java:178)
at org.mule.model.resolvers.DefaultEntryPointResolver Set.invoke(DefaultEntryPointResolverSet.java:50)
at org.mule.component.DefaultLifecycleAdapter.interce pt(DefaultLifecycleAdapter.java:202)
at org.mule.component.AbstractJavaComponent.invokeCom ponentInstance(AbstractJavaComponent.java:82)
at org.mule.component.AbstractJavaComponent.doOnCall( AbstractJavaComponent.java:73)
at org.mule.component.AbstractComponent.onCall(Abstra ctComponent.java:87)
at org.mule.model.seda.SedaService$ComponentStageWork er.run(SedaService.java:533)
at org.mule.work.WorkerContext.run(WorkerContext.java :310)
at edu.emory.mathcs.backport.java.util.concurrent.Thr eadPoolExecutor.runWorker(ThreadPoolExecutor.java: 1061)
at edu.emory.mathcs.backport.java.util.concurrent.Thr eadPoolExecutor$Worker.run(ThreadPoolExecutor.java :575)
at java.lang.Thread.run(Thread.java:636)

************************************************** ******************************

23:56:06 [DefaultServiceExceptionStrategy:Volume.10] ERROR Message being processed is: org.mule.transport.DefaultMessageAdapter/org.mule.transport.DefaultMessageAdapter@105a9{id= acd6fb00-593a-11e0-80e4-e3ba4516fe20, payload=edu.ucsb.eucalyptus.msgs.CreateVolumeType, correlationId=acd6fb00-593a-11e0-80e4-e3ba4516fe20, correlationGroup=-1, correlationSeq=-1, encoding=UTF-8, exceptionPayload=org.mule.message.DefaultException Payload@11aaec3}

on the example given, the 'source' storage controller for cluster1-01 (i.e. containing the snapshot) has the no relevant exceptions in /var/log/eucalyptus/cloud-error.log at the time the euca-create-volume command is given.

The 'destination' storage controller for cluster1-02 (i.e. where the volume is being attempted to be stored) has the following exceptions in /var/log/eucalyptus/cloud-error.log:

23:55:38 [DefaultServiceExceptionStrategy:New I/O server worker #1-29] ERROR
************************************************** ******************************
Message : Component that caused exception is: StorageInternal. Message payload is of type: CreateStorageVolumeType
Type : org.mule.api.service.ServiceException
Code : MULE_ERROR--2
Payload : <?xml version="1.0" encoding="UTF-8"?>
<euca:CreateStorageVolume xmlns:euca="http://msgs.eucalyptus.ucsb.edu">
<euca:StorageRequest>
<euca:correlationId>7f31b339-10eb-4e1e-8d93-286ade1bdb0f</euca:correlationId>
<euca:userId>admin</euca:userId>
<euca:effectiveUserId>eucalyptus</euca:effectiveUserId>
</euca:StorageRequest>
<euca:volumeId>vol-5F8E065B</euca:volumeId>
<euca:snapshotId>snap-58DE0616</euca:snapshotId>
</euca:CreateStorageVolume>
JavaDoc : http://mule.mulesource.org/docs/apid...Exception.html
************************************************** ******************************
Exception stack is:
1. The specified entity was not found (edu.ucsb.eucalyptus.cloud.NoSuchEntityException)
edu.ucsb.eucalyptus.cloud.ws.BlockStorage:544 (null)
2. Component that caused exception is: StorageInternal. Message payload is of type: CreateStorageVolumeType (org.mule.api.service.ServiceException)
org.mule.component.DefaultLifecycleAdapter:214 (http://mule.mulesource.org/docs/apid...Exception.html)
************************************************** ******************************
Root Exception stack trace:
edu.ucsb.eucalyptus.cloud.NoSuchEntityException: The specified entity was not found
at edu.ucsb.eucalyptus.cloud.ws.BlockStorage.CreateSt orageVolume(BlockStorage.java:544)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.mule.model.resolvers.AbstractEntryPointResolve r.invokeMethod(AbstractEntryPointResolver.java:147 )
at org.mule.model.resolvers.ReflectionEntryPointResol ver.invoke(ReflectionEntryPointResolver.java:178)
at org.mule.model.resolvers.DefaultEntryPointResolver Set.invoke(DefaultEntryPointResolverSet.java:50)
at org.mule.component.DefaultLifecycleAdapter.interce pt(DefaultLifecycleAdapter.java:202)
at org.mule.component.AbstractJavaComponent.invokeCom ponentInstance(AbstractJavaComponent.java:82)
at org.mule.component.AbstractJavaComponent.doOnCall( AbstractJavaComponent.java:73)
at org.mule.component.AbstractComponent.onCall(Abstra ctComponent.java:87)
at org.mule.model.seda.SedaService.doSend(SedaService .java:234)
at org.mule.service.AbstractService.sendEvent(Abstrac tService.java:510)
at org.mule.DefaultMuleSession.sendEvent(DefaultMuleS ession.java:351)
at org.mule.routing.inbound.DefaultInboundRouterColle ction.send(DefaultInboundRouterCollection.java:196 )
at org.mule.routing.inbound.DefaultInboundRouterColle ction.route(DefaultInboundRouterCollection.java:16 4)
at org.mule.transport.AbstractMessageReceiver$Default InternalMessageListener.onMessage(AbstractMessageR eceiver.java:604)
at org.mule.transport.AbstractMessageReceiver.routeMe ssage(AbstractMessageReceiver.java:346)
at org.mule.transport.AbstractMessageReceiver.routeMe ssage(AbstractMessageReceiver.java:269)
at org.mule.transport.AbstractMessageReceiver.routeMe ssage(AbstractMessageReceiver.java:262)
at com.eucalyptus.ws.handlers.ServiceSinkHandler.hand leUpstream(ServiceSinkHandler.java:203)
at org.jboss.netty.channel.DefaultChannelPipeline.sen dUpstream(DefaultChannelPipeline.java:567)
at org.jboss.netty.channel.DefaultChannelPipeline$Def aultChannelHandlerContext.sendUpstream(DefaultChan nelPipeline.java:803)
at com.eucalyptus.ws.handlers.MessageStackHandler.han dleUpstream(MessageStackHandler.java:133)
at org.jboss.netty.channel.DefaultChannelPipeline.sen dUpstream(DefaultChannelPipeline.java:567)
at org.jboss.netty.channel.DefaultChannelPipeline$Def aultChannelHandlerContext.sendUpstream(DefaultChan nelPipeline.java:803)
at com.eucalyptus.ws.handlers.MessageStackHandler.han dleUpstream(MessageStackHandler.java:133)
at org.jboss.netty.channel.DefaultChannelPipeline.sen dUpstream(DefaultChannelPipeline.java:567)
at org.jboss.netty.channel.DefaultChannelPipeline$Def aultChannelHandlerContext.sendUpstream(DefaultChan nelPipeline.java:803)
at com.eucalyptus.ws.handlers.MessageStackHandler.han dleUpstream(MessageStackHandler.java:133)
at org.jboss.netty.channel.DefaultChannelPipeline.sen dUpstream(DefaultChannelPipeline.java:567)
at org.jboss.netty.channel.DefaultChannelPipeline$Def aultChannelHandlerContext.sendUpstream(DefaultChan nelPipeline.java:803)
at com.eucalyptus.ws.handlers.MessageStackHandler.han dleUpstream(MessageStackHandler.java:133)
at org.jboss.netty.channel.DefaultChannelPipeline.sen dUpstream(DefaultChannelPipeline.java:567)
at org.jboss.netty.channel.DefaultChannelPipeline$Def aultChannelHandlerContext.sendUpstream(DefaultChan nelPipeline.java:803)
at com.eucalyptus.ws.handlers.MessageStackHandler.han dleUpstream(MessageStackHandler.java:133)
at org.jboss.netty.channel.DefaultChannelPipeline.sen dUpstream(DefaultChannelPipeline.java:567)
at org.jboss.netty.channel.DefaultChannelPipeline$Def aultChannelHandlerContext.sendUpstream(DefaultChan nelPipeline.java:803)
at com.eucalyptus.ws.server.NioServerHandler.messageR eceived(NioServerHandler.java:119)
at org.jboss.netty.channel.SimpleChannelUpstreamHandl er.handleUpstream(SimpleChannelUpstreamHandler.jav a:87)
at org.jboss.netty.channel.DefaultChannelPipeline.sen dUpstream(DefaultChannelPipeline.java:567)
at org.jboss.netty.channel.DefaultChannelPipeline$Def aultChannelHandlerContext.sendUpstream(DefaultChan nelPipeline.java:803)
at org.jboss.netty.handler.stream.ChunkedWriteHandler .handleUpstream(ChunkedWriteHandler.java:114)
at org.jboss.netty.channel.DefaultChannelPipeline.sen dUpstream(DefaultChannelPipeline.java:567)
at org.jboss.netty.channel.DefaultChannelPipeline$Def aultChannelHandlerContext.sendUpstream(DefaultChan nelPipeline.java:803)
at org.jboss.netty.channel.Channels.fireMessageReceiv ed(Channels.java:385)
at org.jboss.netty.handler.codec.replay.ReplayingDeco der.unfoldAndfireMessageReceived(ReplayingDecoder. java:459)
at org.jboss.netty.handler.codec.replay.ReplayingDeco der.callDecode(ReplayingDecoder.java:443)
at org.jboss.netty.handler.codec.replay.ReplayingDeco der.messageReceived(ReplayingDecoder.java:381)
at org.jboss.netty.channel.SimpleChannelUpstreamHandl er.handleUpstream(SimpleChannelUpstreamHandler.jav a:87)
at org.jboss.netty.channel.DefaultChannelPipeline.sen dUpstream(DefaultChannelPipeline.java:567)
at org.jboss.netty.channel.DefaultChannelPipeline$Def aultChannelHandlerContext.sendUpstream(DefaultChan nelPipeline.java:803)
at org.jboss.netty.handler.timeout.ReadTimeoutHandler .messageReceived(ReadTimeoutHandler.java:146)
at org.jboss.netty.channel.SimpleChannelUpstreamHandl er.handleUpstream(SimpleChannelUpstreamHandler.jav a:87)
at org.jboss.netty.channel.DefaultChannelPipeline.sen dUpstream(DefaultChannelPipeline.java:567)
at org.jboss.netty.channel.DefaultChannelPipeline$Def aultChannelHandlerContext.sendUpstream(DefaultChan nelPipeline.java:803)
at org.jboss.netty.handler.timeout.IdleStateHandler.m essageReceived(IdleStateHandler.java:214)
at org.jboss.netty.channel.SimpleChannelUpstreamHandl er.handleUpstream(SimpleChannelUpstreamHandler.jav a:87)
at org.jboss.netty.channel.DefaultChannelPipeline.sen dUpstream(DefaultChannelPipeline.java:567)
at org.jboss.netty.channel.DefaultChannelPipeline$Def aultChannelHandlerContext.sendUpstream(DefaultChan nelPipeline.java:803)
at org.jboss.netty.channel.SimpleChannelHandler.messa geReceived(SimpleChannelHandler.java:159)
at com.eucalyptus.ws.handlers.ChannelStateMonitor.mes sageReceived(ChannelStateMonitor.java:61)
at org.jboss.netty.channel.SimpleChannelHandler.handl eUpstream(SimpleChannelHandler.java:105)
at org.jboss.netty.channel.DefaultChannelPipeline.sen dUpstream(DefaultChannelPipeline.java:567)
at org.jboss.netty.channel.DefaultChannelPipeline.sen dUpstream(DefaultChannelPipeline.java:562)
at org.jboss.netty.channel.Channels.fireMessageReceiv ed(Channels.java:342)
at org.jboss.netty.channel.Channels.fireMessageReceiv ed(Channels.java:329)
at org.jboss.netty.channel.socket.nio.NioWorker.read( NioWorker.java:330)
at org.jboss.netty.channel.socket.nio.NioWorker.proce ssSelectedKeys(NioWorker.java:282)
at org.jboss.netty.channel.socket.nio.NioWorker.run(N ioWorker.java:203)
at org.jboss.netty.util.ThreadRenamingRunnable.run(Th readRenamingRunnable.java:110)
at org.jboss.netty.util.internal.IoWorkerRunnable.run (IoWorkerRunnable.java:53)
at org.jboss.netty.handler.execution.MemoryAwareThrea dPoolExecutor$MemoryAwareRunnable.run(MemoryAwareT hreadPoolExecutor.java:502)
at java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)

************************************************** ******************************

One strange thing I've noticed: in the cloud-output.log of the host on which the source storage controller (i.e. containing the original volume) is located, a huge quantity of lines similar to the following appear, at the time that the snapshots are generated (euca-create-snapshot). Does this indicate a crash of some kind?

10:01:33 DEBUG content | Wire.java:84 >> "[0x1f][0x8b][0x8][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0xed][0xc1][0x1][0x1][0x0][0x0][0x0][0x82] [0xff][0xaf]nH@[0x1][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0]|[0x18]a[0x85][0xf8]J[0x0][0x90][0x1][0x0]"

Also, following Graziano's description of the process, I've located the two copies of the snapshot, which are created sequentially first on the sc (/var/lib/eucalyptus/volumes) and then on walrus (/var/lib/eucalyptus/bukkits). I was surprised to note that the size of the snapshot file is different on the sc and walrus, using both 'ls -l' and 'du'. I would have expected them to be identical (unless only a compressed version is stored on walrus). Is it possible that the snapshot copy to walrus is incomplete for some reason, even if euca-describe-snapshots reports 100% completion?

Can anyone help to resolve the problem?