ec2-bundle-vol and ec2-upload-bundle result in non accepted manifest

[Torsten Spindler]

I try to re-bundle a Jaunty instance with help of the ec2 tools. This fails when trying to register the image:

$ euca-register jaunty-scripts/jaunty-scripts.manifest
Image: Image registration failed because the manifest referenced is invalid or unavailable.

[Torsten Spindler (tspindler)]

This is the cloud-debug output written for the operation above.

[Torsten Spindler (tspindler)]

This is the jaunty-scripts.manifest as found in the /var/lib/eucalyptus/bukkits/jaunty-scripts directory.

[Thierry Carrez (ttx)]

Do you have a history of the commands run before euca-register ?

[Torsten Spindler (tspindler)]

I haven't saved the history, sorry. I followed these steps:

sudo apt-get install default-jre ruby libopenssl-ruby curl rsync unzip
installed ec2-api and ec2-ami tools
rm /root/firstboot_done
mount ephemeral storage to /mnt
copy X.509 zip to /mnt, create .euca directory there
set JAVA_HOME

put into .bashrc or environment:
export EC2_HOME=/opt/ec2-api-tools-1.3-30349
export EC2_AMITOOL_HOME=/opt/ec2-ami-tools-1.3-26357
PATH=~/bin:"${PATH}":${EC2_HOME}/bin:${EC2_AMITOOL_HOME}/bin

ec2-bundle-vol -d /mnt -p jaunty-scripts -u 1111-2222-3333
-k /mnt/euca2-admin-7d6bf6ee-pk.pem
-c /mnt/euca2-admin-7d6bf6ee-cert.pem -s 2048 -e /mnt/,/tmp --no-inherit

ec2-upload-bundle --bucket jaunty-scripts
--manifest ./jaunty-scripts.manifest.xml -a $EC2_ACCESS_KEY -s
$EC2_SECRET_KEY --url $S3_URL

ec2-register <bucket>/<manifest> -K pk -C cert

[Torsten Spindler (tspindler)]

I re bundled a karmic image today and the euca-register command went fine. So this bug might only be found in Jaunty images.

[Eric Hammond (esh)]

Torsten: What AMI id are you starting with?

[Torsten Spindler (tspindler)]

On Tue, 2009-11-10 at 08:51 +0000, Eric Hammond wrote:
> Torsten: What AMI id are you starting with?

This one was a Jaunty image created with vmbuilder.
part file looks like:
root 2000
swap 1

and first boot looks like:
while (! ping -c 1 archive.ubuntu.com); do sleep 1; done
apt-get -y install openssh-server
ifconfig -a

[Scott Moser (smoser)]

Torsten,
  I'm somewhat confused... The description of this bug mentions euca-register, but comment 4 fairly clearly says 'ec2-register'.

Which are you using?

[Torsten Spindler (tspindler)]

I tried both. ec2-register from the instance initially and then euca-register from the front end.

[trung_bmt (trung-bmt)]

I encountered the same issue and here is my proposed fix:

--- WalrusManager.java 2009-11-05 08:15:54.000000000 +0800
+++ WalrusManager.java.new 2009-11-13 20:57:48.936302000 +0800
@@ -1521,8 +1521,10 @@ public class WalrusManager {
         byte[] bytes = new byte[102400/*TODO: NEIL WalrusQueryDispatcher.DATA_MESSAGE_SIZE*/];
         int bytesRead = 0;
         String base64Data = "";
- while((bytesRead = storageManager.readObject(bucketName, objectName, bytes, bytesRead)) > 0) {
+ int offset = 0;
+ while((bytesRead = storageManager.readObject(bucketName, objectName, bytes, offset)) > 0) {
          base64Data += new String(bytes, 0, bytesRead);
+ offset += bytesRead;
         }
         reply.setBase64Data(base64Data);
        } catch (IOException ex) {

The issue results in "out of memory" crash when the manifest file is bigger than 100kb. The paramter "bytesread" in storageManager.readObject should be the offset, not the number of read bytes.

[Dustin Kirkland  (kirkland)]

Assigning to Dan, as he'll take a look at the issue and the proposed patch.

[Dustin Kirkland  (kirkland)]

Lowering the priority, as this isn't happening against Karmic images.

[Dustin Kirkland  (kirkland)]

But leaving the bug open as this may resurface.

[Neil Soman (neilsoman)]

This was fixed a while ago. The fix should be in the current 1.6.2 branch.

    revno: 1068.1.28
    committer: Neil
    timestamp: Wed 2009-12-23 12:04:40 -0800
    message:
      fixes to get inline data.

[Olivier Gagnon (procule)]

Other infos here: lp:513100

[Dustin Kirkland  (kirkland)]

Marking fix-released per Neil.

[Tom Murphy (tmurphy-mlp)]

I'm having the same problem:

Uploaded image as MLPCENTOS2-IMG/ami-centos5.3-64bit-base.manifest.xml
root@mlp-cloud1-master:/tmp# euca-register MLPCENTOS2-IMG/ami-centos5.3-64bit-base.manifest.xml
Image: Image registration failed because the manifest referenced is invalid or unavailable.

The Image Info:
root@mlp-cloud1-master:/tmp# ls -l /tmp/ami-centos5.3-64bit-base.img
-rw-r--r-- 1 root root 10737418240 2010-03-10 11:53 /tmp/ami-centos5.3-64bit-base.img
root@mlp-cloud1-master:/tmp# ls -l ami-centos5.3-64bit-base.img.manifest.xml
-rw-r--r-- 1 root root 10867 2010-03-10 12:13 ami-centos5.3-64bit-base.img.manifest.xml
root@mlp-cloud1-master:/tmp#

root@mlp-cloud1-master:/tmp# dpkg -l | grep euc
ii euca2ools 1.2-0ubuntu6 managing cloud instances for Eucalyptus
ii eucalyptus-cc 1.6.2-0ubuntu12 Elastic Utility Computing Architecture - Clu
ii eucalyptus-cloud 1.6.2-0ubuntu12 Elastic Utility Computing Architecture - Clo
ii eucalyptus-common 1.6.2-0ubuntu12 Elastic Utility Computing Architecture - Com
ii eucalyptus-gl 1.6.2-0ubuntu12 Elastic Utility Computing Architecture - Log
ii eucalyptus-java-common 1.6.2-0ubuntu12 Elastic Utility Computing Architecture - Com
ii eucalyptus-nc 1.6.2-0ubuntu12 Elastic Utility Computing Architecture - Nod
ii eucalyptus-sc 1.6.2-0ubuntu12 Elastic Utility Computing Architecture - Sto
ii eucalyptus-walrus 1.6.2-0ubuntu12 Elastic Utility Computing Architecture - Wal
ii libeucalyptus-commons-ext-java 0.5.0-0ubuntu2 Eucalyptus commons external Java library
root@mlp-cloud1-master:/tmp#

root@mlp-cloud1-master:/tmp# cat /etc/*release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=9.10
DISTRIB_CODENAME=karmic
DISTRIB_DESCRIPTION="Ubuntu 9.10"
root@mlp-cloud1-master:/tmp#

The steps followed prior to this error:

#kernel
euca-bundle-image -i /tmp/vmlinuz-2.6.18-128.el5 --kernel true
euca-upload-bundle -b MLPCENTOS-KERN -m /tmp/vmlinuz-2.6.18-128.el5.manifest.xml
euca-register MLPCENTOS-KERN/vmlinuz-2.6.18-128.el5.manifest.xml

#initrd
euca-bundle-image -i /tmp/initrd-2.6.18-128.el5.img --ramdisk true
euca-upload-bundle -b MLPCENTOS-INIT -m /tmp/initrd-2.6.18-128.el5.img.manifest.xml
euca-register MLPCENTOS-INIT/initrd-2.6.18-128.el5.img.manifest.xml

#image
euca-bundle-image -i /tmp/ami-centos5.3-64bit-base.img
euca-upload-bundle -b MLPCENTOS2-IMG -m /export/home/EC2AMIFILES/ami-centos5.3-64bit-base.manifest.xml
euca-register MLPCENTOS2-IMG/ami-centos5.3-64bit-base.manifest.xml

[Jean-Pierre Dion (jean-pierre-dion)]

Hi,
I am experiencing this issue too.

1)
euca-bundle-image -i vmlinuz-2.6.31-14-server --kernel true -c /var/lib/eucalyptus/keys/cloud_jpd/node-cert.pem -k /var/lib/eucalyptus/keys/cloud_jpd/cluster-pk.pem --ec2cert /var/lib/eucalyptus/keys/cloud_jpd/cluster-cert.pem

Checking image
Tarring image
Encrypting image
Splitting image...
Part: vmlinuz-2.6.31-14-server.part.0
Generating manifest

2)
euca-upload-bundle -b kernel_bucket -m /tmp/vmlinuz-2.6.31-14-server.manifest.xml

Checking bucket: kernel_bucket
Uploading manifest file
Uploading part: vmlinuz-2.6.31-14-server.part.0
Uploaded image as kernel_bucket/vmlinuz-2.6.31-14-server.manifest.xml

3)
euca-register kernel_buckets/vmlinuz-2.6.31-14-server.manifest.xml

Image: Image registration failed because the manifest referenced is invalid or unavailable.

4)
Packages installed :
dpkg -l |grep euca
ii euca2ools 1.0+bzr20091007-0ubuntu1.1 managing cloud instances for Eucalyptus
ii eucalyptus-cc 1.6~bzr931-0ubuntu7.5 Elastic Utility Computing Architecture - Clu
ii eucalyptus-cloud 1.6~bzr931-0ubuntu7.5 Elastic Utility Computing Architecture - Clo
ii eucalyptus-common 1.6~bzr931-0ubuntu7.5 Elastic Utility Computing Architecture - Com
ii eucalyptus-gl 1.6~bzr931-0ubuntu7.5 Elastic Utility Computing Architecture - Log
ii eucalyptus-java-common 1.6~bzr931-0ubuntu7.5 Elastic Utility Computing Architecture - Com
ii eucalyptus-sc 1.6~bzr931-0ubuntu7.5 Elastic Utility Computing Architecture - Sto
ii eucalyptus-walrus 1.6~bzr931-0ubuntu7.5 Elastic Utility Computing Architecture - Wal
ii libeucalyptus-commons-ext-java 0.4.2-0ubuntu1 Eucalyptus commons external Java library

5)
Also tried with both versions below, same issue :
euca2ools 1.6.2~bzr1199~1.6.2-0euca1
euca2ools-1.2-bzr265-src

Any idea ?

Thank you.

Jean-Pierre Dion

[Torsten Spindler (tspindler)]

On Wed, 2010-03-17 at 13:32 +0000, Jean-Pierre Dion wrote:
...
> 2)
> euca-upload-bundle -b kernel_bucket -m /tmp/vmlinuz-2.6.31-14-server.manifest.xml
...
> 3)
> euca-register kernel_buckets/vmlinuz-2.6.31-14-server.manifest.xml

Just to make sure that there is no typing mistake: the euca-register
should say
euca-register kernel_bucket/vmlinuz-2.6.31-14-server.manifest.xml

[Jean-Pierre Dion (jean-pierre-dion)]

Sorry,

I forgot to tell you that I am working on squeeze/sid, from Ubuntu 9.10 Server.

Jean-Pierre Dion

[Tom Murphy (tmurphy-mlp)]

This is not fixed. I'm using the latest version of Umbuntu and the latest release of Eucalyptus and still experiencing the same problem. Please revisit this bug

[Jean-Pierre Dion (jean-pierre-dion)]

Hi Torsten,

>> 3)
>> euca-register kernel_buckets/vmlinuz-2.6.31-14-server.manifest.xml

> Just to make sure that there is no typing mistake: the euca-register
> should say
> euca-register kernel_bucket/vmlinuz-2.6.31-14-server.manifest.xml

you are right, but I just mistyped when copying here.
I just did the test again with correct params, still got the error.

Thanks.

jean-pierre dion

[Olivier Gagnon (ogagnon-fake-studio)]

As said in message #14, the fix was in rev 1068.1.28.
So if your bzr version is older than that, it is not fixed.

I synced the files and recompiled eucalyptus with bzr rev 1068.1.28 some months ago and it was indeed fixed.
My 70gigs image registered in seconds.

[Jean-Pierre Dion (jean-pierre-dion)]

OK
I am trying to update but I got an issue with the bzr config (trying to add repo ppa:bzr/ppa does not work = time out).
My http_proxy works fine otherwise...
I "webbed" a bit but I could not find any solution.
Is there some special config to do ?

Thank you.
jpd

[Tom Murphy (tmurphy-mlp)]

 apt-get install eucalyptus-cc eucalyptus-cloud eucalyptus-common euca2ools eucalyptus-gl eucalyptus-java-common eucalyptus-nc eucalyptus-sc eucalyptus-walrus
Reading package lists... Done
Building dependency tree
Reading state information... Done
eucalyptus-cc is already the newest version.
eucalyptus-cloud is already the newest version.
eucalyptus-common is already the newest version.
euca2ools is already the newest version.
eucalyptus-gl is already the newest version.
eucalyptus-java-common is already the newest version.
eucalyptus-nc is already the newest version.
eucalyptus-sc is already the newest version.
eucalyptus-walrus is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 13 not upgraded.

ok, where can I get rev. 1068.1.28 ?

[Olivier Gagnon (ogagnon-fake-studio)]

You would want to use "bzr" (Bazaar), which is a program like svn and git used by Ubuntu and Eucalyptus, to retrieve the sources you want.
You should be able to find some infos how to use it on lauchpad.

O L I V I E R G A G N O N
F A K E
ADMINISTRATEUR RÉSEAU |
NETWORK ADMIN
+ 1 514 . 846 . 8439 | #226
Cell: 514 . 965 . 9511
WWW.FAKE-STUDIO.COM

----- Mail Original -----
De: "Tom Murphy" <email address hidden>
À: <email address hidden>
Envoyé: Vendredi 26 Mars 2010 13:38:24
Objet: [Bug 470355] Re: ec2-bundle-vol and ec2-upload-bundle result in non accepted manifest

apt-get install eucalyptus-cc eucalyptus-cloud eucalyptus-common euca2ools eucalyptus-gl eucalyptus-java-common eucalyptus-nc eucalyptus-sc eucalyptus-walrus
Reading package lists... Done
Building dependency tree
Reading state information... Done
eucalyptus-cc is already the newest version.
eucalyptus-cloud is already the newest version.
eucalyptus-common is already the newest version.
euca2ools is already the newest version.
eucalyptus-gl is already the newest version.
eucalyptus-java-common is already the newest version.
eucalyptus-nc is already the newest version.
eucalyptus-sc is already the newest version.
eucalyptus-walrus is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 13 not upgraded.

ok, where can I get rev. 1068.1.28 ?

--
ec2-bundle-vol and ec2-upload-bundle result in non accepted manifest
https://bugs.launchpad.net/bugs/470355
You received this bug notification because you are a direct subscriber
of a duplicate bug.

Status in Eucalyptus: Incomplete
Status in “eucalyptus” package in Ubuntu: Fix Released
Status in “eucalyptus” source package in Lucid: Fix Released

Bug description:
I try to re-bundle a Jaunty instance with help of the ec2 tools. This fails when trying to register the image:

$ euca-register jaunty-scripts/jaunty-scripts.manifest
Image: Image registration failed because the manifest referenced is invalid or unavailable.

To unsubscribe from this bug, go to:
https://bugs.launchpad.net/eucalyptus/+bug/470355/+subscribe

[dirakx (dirakx)]

I'm having this same problem with latest packages in Maverick.

Any hint on how to fix it ?

[Andy Grimm (agrimm)]

This issue is now being tracked upstream at http://eucalyptus.atlassian.net/browse/EUCA-2671

Please watch that issue for further updates.