Autoregistration of eucalyptus-cc sometimes fails

Bug #438602 reported by Thierry Carrez on 2009-09-29
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
eucalyptus (Ubuntu)
High
Dustin Kirkland 
Karmic
High
Dustin Kirkland 
upstart (Ubuntu)
High
Unassigned
Karmic
High
Unassigned

Bug Description

20090929 UEC Cluster install. Some services sometimes fail to autoregister.
Despite upstartification, I still sometimes get "ERROR: you need to be on the CLC host and the CLC needs to be running." in /var/log/eucalyptus/*-registration.log and some components fail to register.
Apparently registration is still a little too fast, the service is not yet ready to register them.

Thierry Carrez (ttx) on 2009-09-29
Changed in eucalyptus (Ubuntu):
assignee: nobody → Thierry Carrez (ttx)
importance: Undecided → High
milestone: none → ubuntu-9.10-beta
status: New → Triaged
Thierry Carrez (ttx) on 2009-09-29
Changed in eucalyptus (Ubuntu Karmic):
status: Triaged → Fix Committed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package eucalyptus - 1.6~bzr854-0ubuntu11

---------------
eucalyptus (1.6~bzr854-0ubuntu11) karmic; urgency=low

  * Restore /etc/init.d/eucalyptus-nc as a quickfix for node startup,
    LP: #438585
  * Wait 5 more seconds after eucalyptus-cloud appears to be started before
    considering it started, to allow registration to succeed, LP: #438602
  * Fix failure to import VNET_PUBLICIPS setting if a space-separated list
    of IPs is entered, LP: #438586

 -- Thierry Carrez <email address hidden> Tue, 29 Sep 2009 11:42:26 +0200

Changed in eucalyptus (Ubuntu Karmic):
status: Fix Committed → Fix Released
Thierry Carrez (ttx) wrote :

This is still failing randomly. Introducing a delay is not enough, sometimes eucalyptus just seems to not accept registration. We'll probably need a fix for bug 438635 to fix that, so it might be postbeta.

Changed in eucalyptus (Ubuntu Karmic):
assignee: Thierry Carrez (ttx) → nobody
status: Fix Released → Triaged
Thierry Carrez (ttx) wrote :

One lead to explore, since walrus is the only one working, would be to use the "external IP" in cc and sc registration as it is used in walrus. If this cannot be fixed today, I'd recommend considering it a known beta bug.

Changed in eucalyptus (Ubuntu Karmic):
assignee: nobody → Dustin Kirkland (kirkland)
Dustin Kirkland  (kirkland) wrote :

I can confirm that walrus did in fact auto register itself successfully, and cc did not. I'm investigating the upstart scripts now...

:-Dustin

Changed in eucalyptus (Ubuntu Karmic):
status: Triaged → In Progress
summary: - Autoregistration sometimes fails
+ Autoregistration of eucalyptus-cc sometimes fails
Dustin Kirkland  (kirkland) wrote :

We're having trouble getting in touch with Scott today, but what we're seeing now is starting to look like an Upstart deficiency. I'm marking a task against Upstart with this bug, in case there's a bug there.

We think we have the registration correctly configured using upstart, specifying the dependencies accordingly. However, the behavior of the jobs are not matching our design expectations. At the very least, we require some help from Scott analyzing the approach.

:-Dustin

Changed in upstart (Ubuntu Karmic):
importance: Undecided → High
Mathias Gug (mathiaz) wrote :

It seems that the eucalyptus-cc-registration job is not always started by upstart.

To check that the jobs have been started, I've added a pre-start section to each -registration job (ex for the walrus-registration job):

pre-start script
         touch /var/tmp/walrus-registration.started
end script

To enable upstart debug messages, I've added a line to the pre-script section of the eucalyptus job (since it's the first one to start the whole eucalyptus system):

        /sbin/initctl log-priority debug

I've attached the syslog generated during boot.

Dustin Kirkland  (kirkland) wrote :

I've made some progress, and autoregistration is now succeeding for me.

I had to ditch the "localhost" registrations, and use a real IP address. I pulled that functionality into a single place, that can be sourced by each upstart file.

As of 1am, this is working for me.

I'm uploading now, and will poke admins about accepting the package and pushing another ISO build.

:-Dustin

Changed in eucalyptus (Ubuntu Karmic):
status: In Progress → Fix Committed
Changed in upstart (Ubuntu Karmic):
status: New → Incomplete
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package eucalyptus - 1.6~bzr854-0ubuntu13

---------------
eucalyptus (1.6~bzr854-0ubuntu13) karmic; urgency=low

  * eucalyptus-common.eucalyptus.upstart: the -L option was in the wrong
    place, breaking the console.log logging, LP: #439161
  * eucalyptus-cc.eucalyptus-cc-registration.upstart,
    eucalyptus-cc.upstart, eucalyptus-cloud.upstart,
    eucalyptus-ipaddr.conf, eucalyptus-common.install,
    eucalyptus-sc.eucalyptus-sc-registration.upstart,
    eucalyptus-sc.upstart, eucalyptus-walrus.upstart: using "localhost"
    is breaking registration; instead, use the real, external ip
    address; consolidate the (nasty) logic for determining this
    into a single, sourcable, overridable location, LP: #437058, #438602
  * 80-eucalyptus-url: use the new $CLOUD_IP_ADDR value and logic

 -- Dustin Kirkland <email address hidden> Wed, 30 Sep 2009 01:16:12 -0500

Changed in eucalyptus (Ubuntu Karmic):
status: Fix Committed → Fix Released
Thierry Carrez (ttx) wrote :

Documented workaround for eucalyptus 1.6~bzr854-0ubuntu12 users:
$ sudo euca_conf --local-sync --register-walrus <IPADDRESSOFTHECLUSTER>
$ sudo euca_conf --local-sync --register-cluster <CLUSTERNAME> <IPADDRESSOFTHECLUSTER>
$ sudo euca_conf --local-sync --register-sc <CLUSTERNAME> <IPADDRESSOFTHECLUSTER>
$ # Registration is done. Now we need to workaround bug 439251
$ sudo stop eucalyptus
$ sudo start eucalyptus
or:
$ sudo apt-get update
$ sudo apt-get upgrade
$ # eucalyptus upgrades to 1.6~bzr854-0ubuntu13
$ sleep 60
$ # autoregistration is done. Now we need to workaround bug 439251
$ sudo stop eucalyptus
$ sudo start eucalyptus

Changed in eucalyptus (Ubuntu Karmic):
milestone: ubuntu-9.10-beta → none

The dbus-reconnect fix that I assume the Upstart task refers to has been uploaded

Changed in upstart (Ubuntu Karmic):
status: Incomplete → Fix Released
tags: added: iso-testing
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Bug attachments