eucalyptus-cloud: cloud fails to start on separate install

Bug #613033 reported by C de-Avillez on 2010-08-03
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Eucalyptus
Fix Released
Undecided
chris grzegorczyk
eucalyptus (Ubuntu)
Critical
Dave Walker
Maverick
Critical
Dave Walker

Bug Description

Binary package hint: eucalyptus-cloud

Installing a fully-separate topology (CLC, Walrus, CC, SC, and NCs each on their own machine). After installing the CLC Eucalyptus fails to initialise with:

"Do not ever do whatever it is you did: Failed to lookup required component: walrus".

ii eucalyptus-cloud 2.0~bzr1211-0ubuntu1
ii eucalyptus-common 2.0~bzr1211-0ubuntu1
ii eucalyptus-java-common 2.0~bzr1211-0ubuntu1

ProblemType: BugDistroRelease: Ubuntu 10.10
Package: debian-installer 20100211ubuntu17
ProcVersionSignature: Ubuntu 2.6.35-14.19-generic 2.6.35
Uname: Linux 2.6.35-14-generic x86_64
Architecture: amd64
CheckboxSubmission: 48216eabc5cef2006b8478621d301a28
CheckboxSystem: d00f84de8a555815fa1c4660280da308
Date: Thu Jul 1 15:15:09 2010
Dependencies:

EcryptfsInUse: YesInstallationMedia: Ubuntu-Server 10.04 "Lucid Lynx" - Beta amd64 (20100406.1)
MachineType: Dell Inc. Inspiron 1721
MemoryUsage:
 total used free shared buffers cached
 Mem: 3796184 3741964 54220 0 66092 2333688
 -/+ buffers/cache: 1342184 2454000
 Swap: 3903484 30172 3873312
ProcCmdLine: BOOT_IMAGE=/vmlinuz-2.6.35-14-generic root=/dev/mapper/hostname-root ro crashkernel=384M-2G:64M,2G-:128M
ProcEnviron:
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bashSourcePackage: debian-installer

C de-Avillez (hggdh2) wrote :

Sigh. Somehow I opened the bug with the incorrect apport file. Here is the /var/log/eucalyptus/*.

description: updated
Thierry Carrez (ttx) wrote :

Reproduced.
Cloud-output shows:

17:27:39 ERROR 33 Exceptions | Fatal error occured: com.eucalyptus.util.Exceptions.uncatchable:29
17:27:39 ERROR 36 Exceptions | java.lang.Error: Failed to lookup required component: walrus

So it looks like in eucalyptus 2.0 the Walrus must be started before (or together with) the CLC. The Lucid autoregistration discussions made it clear the CLC must be up before the Walrus can be registered, so everything is designed to have the CLC installed before (the Walrus downloads its seed from the CLC). That's borken now, so I expect topology2 to always fail.

C de-Avillez (hggdh2) wrote :

Marking Triaged/High. Thierry confirmed it. We now have to find out if this is a new (undocumented) feature of 2.0, or simply a regression.

Changed in eucalyptus (Ubuntu):
importance: Undecided → High
status: New → Triaged
Dustin Kirkland  (kirkland) wrote :

Sending this upstream.

Upstream Eucalyptus: Can you confirm this new behavior? If so, we (Ubuntu) need to rework our install/startup procedure...

Changed in eucalyptus (Ubuntu):
assignee: nobody → Dave Walker (davewalker)
Dustin Kirkland  (kirkland) wrote :

Assigning to Dave Walker. This bug is going to need serious attention.

Changed in eucalyptus (Ubuntu):
milestone: none → ubuntu-10.10-beta
tags: added: iso-testing
C de-Avillez (hggdh2) wrote :

Heh. I just installed CLC+Walrus, and got "Do not ever do whatever it is you did: Failed to lookup required component: storage". This is quite broken right now.

C de-Avillez (hggdh2) wrote :

Resetting Importance to Critical -- the only topology that works, right now, is the all-in-one (CLC+Walrus+CC+SC).

Changed in eucalyptus (Ubuntu Maverick):
importance: High → Critical
Dave Walker (davewalker) wrote :

Mail sent to upstream Eucalyptus for comment if this is a regression or intended design change.

@Dustin, Please don't assign me bugs without discussing them with me first. Thanks.

C de-Avillez (hggdh2) wrote :

Still present on 2.0~bzr1218-0ubuntu1.

Changed in eucalyptus:
assignee: nobody → chris grzegorczyk (chris-grze)
status: New → In Progress
Dave Walker (davewalker) wrote :

@chris grzegorczyk, Thanks for looking into this - do you have any progress to report?

Thanks.

chris grzegorczyk (chris-grze) wrote :

Working on setting up regression test and merging the fix now.

C de-Avillez (hggdh2) wrote :
Download full text (6.2 KiB)

Just installed eucalyptus-cloud on a fully-separated topology:

15:36:40 ERROR [ServiceDispatchBootstrapper:main] Ignoring the error that occured: com.eucalyptus.ws.ServiceDispatchBootstrapper.load:105
15:36:40 ERROR [Exceptions:main] java.util.NoSuchElementException: Missing entry for component 'walrus' info type: Component (com.eucalyptus.component.Component
java.util.NoSuchElementException: Missing entry for component 'walrus' info type: Component (com.eucalyptus.component.Component
        at com.eucalyptus.component.Components.lookup(Components.java:98)
        at com.eucalyptus.component.Components.lookup(Components.java:110)
        at com.eucalyptus.ws.ServiceDispatchBootstrapper.load(ServiceDispatchBootstrapper.java:103)
        at com.eucalyptus.bootstrap.Bootstrap$Stage.load(Bootstrap.java:93)
        at com.eucalyptus.bootstrap.SystemBootstrapper.load(SystemBootstrapper.java:129)
15:36:40 ERROR [ServiceDispatchBootstrapper:main] Ignoring the error that occured: com.eucalyptus.ws.ServiceDispatchBootstrapper.load:105
15:36:40 ERROR [Exceptions:main] java.util.NoSuchElementException: Missing entry for component 'storage' info type: Component (com.eucalyptus.component.Component
java.util.NoSuchElementException: Missing entry for component 'storage' info type: Component (com.eucalyptus.component.Component
        at com.eucalyptus.component.Components.lookup(Components.java:98)
        at com.eucalyptus.component.Components.lookup(Components.java:110)
        at com.eucalyptus.ws.ServiceDispatchBootstrapper.load(ServiceDispatchBootstrapper.java:103)
        at com.eucalyptus.bootstrap.Bootstrap$Stage.load(Bootstrap.java:93)
        at com.eucalyptus.bootstrap.SystemBootstrapper.load(SystemBootstrapper.java:129)
15:36:46 ERROR [SystemConfiguration:1727693667@qtp-941762215-8] com.eucalyptus.util.EucalyptusCloudException: Error locating information for [SystemConfiguration id=-1 cloudHost=null defaultKernel=null defaultRamdisk=null registrationId=null maxUserPublicAddresses=null doDynamicPublicAddresses=null systemReservedPublicAddresses=null dnsDomain=null nameserver=null nameserverAddress=null]
com.eucalyptus.util.EucalyptusCloudException: Error locating information for [SystemConfiguration id=-1 cloudHost=null defaultKernel=null defaultRamdisk=null registrationId=null maxUserPublicAddresses=null doDynamicPublicAddresses=null systemReservedPublicAddresses=null dnsDomain=null nameserver=null nameserverAddress=null]
        at com.eucalyptus.entities.EntityWrapper.getUnique(EntityWrapper.java:138)
        at edu.ucsb.eucalyptus.cloud.entities.SystemConfiguration.getSystemConfiguration(SystemConfiguration.java:257)
        at edu.ucsb.eucalyptus.admin.server.Registration.getRegistrationId(Registration.java:142)
        at edu.ucsb.eucalyptus.admin.server.Registration.doGet(Registration.java:148)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:390)
        at org.mortbay.jetty.security.Se...

Read more...

C de-Avillez (hggdh2) wrote :

I installed a walrus (10.55.55.5) and a Cluster (10.55.55.8). The CLC stays running, the CC stays running (and seem to be recognised by the CLC), but the Walrus dies on startup:

17:06:29 ERROR [Transition$anonymously:main] :1281474389221:anonymously:TRANSITION_FAILED:Transition:from:PRIMORDIAL:to:INITIALIZED:com.eucalyptus.component.Component:Bootstrap.initialize.252
17:06:29 ERROR [Transition$anonymously:main] :1281474389221:anonymously:TRANSITION_ROLLBACK:Transition:from:PRIMORDIAL:to:INITIALIZED:com.eucalyptus.component.Component:Bootstrap.initialize.252
17:06:29 FATAL [SystemBootstrapper:main] java.util.NoSuchElementException: Missing entry for component 'eucalyptus' info type: Component (com.eucalyptus.component.Component
java.util.NoSuchElementException: Missing entry for component 'eucalyptus' info type: Component (com.eucalyptus.component.Component
        at com.eucalyptus.component.Components.lookup(Components.java:98)
        at com.eucalyptus.component.Components.lookup(Components.java:110)
        at com.eucalyptus.bootstrap.Component.isLocal(Component.java:114)
        at com.eucalyptus.bootstrap.Bootstrap$1.commit(Bootstrap.java:255)
        at com.eucalyptus.bootstrap.Bootstrap$1.commit(Bootstrap.java:252)
        at com.eucalyptus.component.Lifecycles$State$1.commit(Lifecycles.java:35)
        at com.eucalyptus.util.Transition$anonymously.commit(Transition.java:74)
        at com.eucalyptus.util.Transition.doTransition(Transition.java:133)
        at com.eucalyptus.util.Transition.transition(Transition.java:145)
        at com.eucalyptus.bootstrap.Bootstrap.initialize(Bootstrap.java:252)
        at com.eucalyptus.bootstrap.SystemBootstrapper.init(SystemBootstrapper.java:109)

So, better, but still no cigar.

chris grzegorczyk (chris-grze) wrote :

This appears to be resolved on our end in the following rev. Please see the merge log for details.
------------------------------------------------------------
revno: 1220 [merge]
committer: decker <decker@personal-army>
branch nick: devel
timestamp: Tue 2010-08-10 04:22:52 -0700
message:
  merge: fixes to registration and remote bootstrapping of components & forward port of addressing fixes
------------------------------------------------------------

Changed in eucalyptus:
status: In Progress → Fix Committed
chris grzegorczyk (chris-grze) wrote :

@carlos: sorry, i missed your message.

Can you try installing all of the components on each machine but only running Walrus, SC? This looks like it might be a matter of how things are packaged on our end.

I will try to reproduce on our side and get back ASAP.

C de-Avillez (hggdh2) wrote :

Right now I am running an install for (CLC+Walrus), (CC+SC), (NC)*. So far, the (CLC+Walrus) installed OK, and kept running. When I finish this I will try installing all on each.

C de-Avillez (hggdh2) wrote :

The (CC+SC) also fail to initialise:

18:15:02 FATAL [SystemBootstrapper:main] java.util.NoSuchElementException: Missing entry for component 'eucalyptus' info type: Component (com.eucalyptus.component.Component
java.util.NoSuchElementException: Missing entry for component 'eucalyptus' info type: Component (com.eucalyptus.component.Component
        at com.eucalyptus.component.Components.lookup(Components.java:98)
        at com.eucalyptus.component.Components.lookup(Components.java:110)
        at com.eucalyptus.bootstrap.Component.isLocal(Component.java:114)
        at com.eucalyptus.bootstrap.Bootstrap$1.commit(Bootstrap.java:255)
        at com.eucalyptus.bootstrap.Bootstrap$1.commit(Bootstrap.java:252)
        at com.eucalyptus.component.Lifecycles$State$1.commit(Lifecycles.java:35)
        at com.eucalyptus.util.Transition$anonymously.commit(Transition.java:74)
        at com.eucalyptus.util.Transition.doTransition(Transition.java:133)
        at com.eucalyptus.util.Transition.transition(Transition.java:145)
        at com.eucalyptus.bootstrap.Bootstrap.initialize(Bootstrap.java:252)
        at com.eucalyptus.bootstrap.SystemBootstrapper.init(SystemBootstrapper.java:109)

C de-Avillez (hggdh2) wrote :

OK. I installed eucalyptus-cloud on the (CC+SC) machine, and restarted the SC and CC. They *are* up, and the CLC recognised them. I will now install the NCs, and see what happens.

So, yes, it seems something in packaging (requiring the cloud .jar?).

C de-Avillez (hggdh2) wrote :

(CLC+Walrus), (CC+SC+CLC-down), (NC)+ is up. This CC is running with two NCs. I will run some tests now.

Hmmm. I will get back to you with a fix soon. Hopefully I can manage
to resolve the issue w/o any code changes. More shortly.

On Tue, Aug 10, 2010 at 3:25 PM, C de-Avillez
<email address hidden> wrote:
> OK. I installed eucalyptus-cloud on the (CC+SC) machine, and restarted
> the SC and CC. They *are* up, and the CLC recognised them. I will now
> install the NCs, and see what happens.
>
> So, yes, it seems something in packaging (requiring the cloud .jar?).
>
> --
> eucalyptus-cloud: cloud fails to start on separate install
> https://bugs.launchpad.net/bugs/613033
> You received this bug notification because you are a bug assignee.
>
> Status in Eucalyptus: Fix Committed
> Status in “eucalyptus” package in Ubuntu: Triaged
> Status in “eucalyptus” source package in Maverick: Triaged
>
> Bug description:
> Binary package hint: eucalyptus-cloud
>
> Installing a fully-separate topology (CLC, Walrus, CC, SC, and NCs each on their own machine). After installing the CLC Eucalyptus fails to initialise with:
>
> "Do not ever do whatever it is you did: Failed to lookup required component: walrus".
>
> ii  eucalyptus-cloud                         2.0~bzr1211-0ubuntu1
> ii  eucalyptus-common                        2.0~bzr1211-0ubuntu1
> ii  eucalyptus-java-common                   2.0~bzr1211-0ubuntu1
>
> ProblemType: BugDistroRelease: Ubuntu 10.10
> Package: debian-installer 20100211ubuntu17
> ProcVersionSignature: Ubuntu 2.6.35-14.19-generic 2.6.35
> Uname: Linux 2.6.35-14-generic x86_64
> Architecture: amd64
> CheckboxSubmission: 48216eabc5cef2006b8478621d301a28
> CheckboxSystem: d00f84de8a555815fa1c4660280da308
> Date: Thu Jul  1 15:15:09 2010
> Dependencies:
>
> EcryptfsInUse: YesInstallationMedia: Ubuntu-Server 10.04 "Lucid Lynx" - Beta amd64 (20100406.1)
> MachineType: Dell Inc. Inspiron 1721
> MemoryUsage:
>  total       used       free     shared    buffers     cached
>  Mem:       3796184    3741964      54220          0      66092    2333688
>  -/+ buffers/cache:    1342184    2454000
>  Swap:      3903484      30172    3873312
> ProcCmdLine: BOOT_IMAGE=/vmlinuz-2.6.35-14-generic root=/dev/mapper/hostname-root ro crashkernel=384M-2G:64M,2G-:128M
> ProcEnviron:
>  PATH=(custom, no user)
>  LANG=en_US.UTF-8
>  SHELL=/bin/bashSourcePackage: debian-installer
>
>
>
>

--
Chris Grzegorczyk
Co-founder, Eucalyptus Systems, Inc.
www.eucalyptus.com | 130 Castilian St. | Goleta, CA | 93117 | Mobile:
805-807-8237
________________________________________

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package eucalyptus - 2.0~bzr1224-0ubuntu1

---------------
eucalyptus (2.0~bzr1224-0ubuntu1) maverick; urgency=low

  [ Dave Walker (Daviey) ]
  * New upstream merge, r1224, should fix LP: #610259, #609112, #613033
  * debian/patches/10-disable-iscsi.patch: Removed, we should now be
    supporting iscsi.
  * debian/control: Promoted tgt from Recommends to Depends for
    eucalyptus-sc. (LP: #614503)

  [ Scott Moser ]
  * work around kvm virtio bug with -kernel by using boot floppy LP: #615529
  * remove the listing of dependency on grub-pc for workaround. LP: #613463
 -- Dave Walker (Daviey) <email address hidden> Wed, 11 Aug 2010 13:46:50 +0100

Changed in eucalyptus (Ubuntu Maverick):
status: Triaged → Fix Released
C de-Avillez (hggdh2) wrote :

Reopening, does not seem fixed

Changed in eucalyptus (Ubuntu Maverick):
status: Fix Released → Triaged
chris grzegorczyk (chris-grze) wrote :

@carlos: is that with r1225?

C de-Avillez (hggdh2) wrote :

@chris: yes, it was. I am just now finishing a reinstall, and will be able to confirm in about 20 min.

C de-Avillez (hggdh2) wrote :

Yes, not yet fixed. I am adding a tarball with the logs here

C de-Avillez (hggdh2) wrote :

This was a (CLC+Walrus), (CC+SC) install. (CLC+Walrus) are in cempedak, (CC+SC) in mabolo. I did not even install the NCs.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package eucalyptus - 2.0~bzr1230-0ubuntu1

---------------
eucalyptus (2.0~bzr1230-0ubuntu1) maverick; urgency=low

  * New upstream merge, r1230 - bug fixes:
    - (LP: #617127, #611769, #617032, #613033, #613033, #584497)
  * debian/patches/18-priv_security.patch: Added *_iscsitarget.pl scripts to
    the wrappers.conf whitelist. (LP: #615646)
  * debian/patches/15-fix-default-ramdisk.patch: Dropped, fixed applied upstream.
 -- Dave Walker (Daviey) <email address hidden> Fri, 13 Aug 2010 11:25:25 +0100

Changed in eucalyptus (Ubuntu Maverick):
status: Triaged → Fix Released
Walt Corey (waltc) wrote :
Download full text (4.1 KiB)

I have eucalytus at 2.0~bzr1241-0ubuntu4,1

I receive the following when sourcing eucarc

WARN: Walrus URL is not configured.

In reviewing the cloud-error.log It shows an initial failure with

13:32:55 ERROR [SystemConfiguration:529263462@qtp-248808824-7] com.eucalyptus.util.EucalyptusCloudException: Error locating information for [S
ystemConfiguration id=-1 cloudHost=null defaultKernel=null defaultRamdisk=null registrationId=null maxUserPublicAddresses=null doDynamicPublic
Addresses=null systemReservedPublicAddresses=null dnsDomain=null nameserver=null nameserverAddress=null]
com.eucalyptus.util.EucalyptusCloudException: Error locating information for [SystemConfiguration id=-1 cloudHost=null defaultKernel=null defa
ultRamdisk=null registrationId=null maxUserPublicAddresses=null doDynamicPublicAddresses=null systemReservedPublicAddresses=null dnsDomain=nul
l nameserver=null nameserverAddress=null]
        at com.eucalyptus.entities.EntityWrapper.getUnique(EntityWrapper.java:138)
        at edu.ucsb.eucalyptus.cloud.entities.SystemConfiguration.getSystemConfiguration(SystemConfiguration.java:257)
        at edu.ucsb.eucalyptus.admin.server.Registration.getRegistrationId(Registration.java:142)
        at edu.ucsb.eucalyptus.admin.server.Registration.doGet(Registration.java:148)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:390)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
        at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
        at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.handler.rewrite.RewriteHandler.handle(RewriteHandler.java:230)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:926)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
13:32:57 FATAL [Bootstrap$Stage:main] Fatal error occured during bootstrap: com.eucalyptus.bootstrap.Bootstrap$Stage.start:118
java.lang.IllegalStateExcepti...

Read more...

Neil Soman (neilsoman) wrote :

waltc, did you install eucalyptus-walrus and have you registered Walrus? You will get that warning ("Walrus URL is not configured") if you download credentials before you have Walrus registered.

Marking this bug fix released. Walt if you still have issue, please open a new bug.

Changed in eucalyptus:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers