Autoregistration of eucalyptus-cc sometimes fails

[Thierry Carrez]

20090929 UEC Cluster install. Some services sometimes fail to autoregister.
Despite upstartification, I still sometimes get "ERROR: you need to be on the CLC host and the CLC needs to be running." in /var/log/eucalyptus/*-registration.log and some components fail to register.
Apparently registration is still a little too fast, the service is not yet ready to register them.

[Launchpad Janitor (janitor)]

This bug was fixed in the package eucalyptus - 1.6~bzr854-0ubuntu11

---------------
eucalyptus (1.6~bzr854-0ubuntu11) karmic; urgency=low

  * Restore /etc/init.d/eucalyptus-nc as a quickfix for node startup,
    LP: #438585
  * Wait 5 more seconds after eucalyptus-cloud appears to be started before
    considering it started, to allow registration to succeed, LP: #438602
  * Fix failure to import VNET_PUBLICIPS setting if a space-separated list
    of IPs is entered, LP: #438586

 -- Thierry Carrez <email address hidden> Tue, 29 Sep 2009 11:42:26 +0200

[Thierry Carrez (ttx)]

This is still failing randomly. Introducing a delay is not enough, sometimes eucalyptus just seems to not accept registration. We'll probably need a fix for bug 438635 to fix that, so it might be postbeta.

[Thierry Carrez (ttx)]

One lead to explore, since walrus is the only one working, would be to use the "external IP" in cc and sc registration as it is used in walrus. If this cannot be fixed today, I'd recommend considering it a known beta bug.

[Dustin Kirkland  (kirkland)]

I can confirm that walrus did in fact auto register itself successfully, and cc did not. I'm investigating the upstart scripts now...

:-Dustin

[Dustin Kirkland  (kirkland)]

We're having trouble getting in touch with Scott today, but what we're seeing now is starting to look like an Upstart deficiency. I'm marking a task against Upstart with this bug, in case there's a bug there.

We think we have the registration correctly configured using upstart, specifying the dependencies accordingly. However, the behavior of the jobs are not matching our design expectations. At the very least, we require some help from Scott analyzing the approach.

:-Dustin

[Mathias Gug (mathiaz)]

It seems that the eucalyptus-cc-registration job is not always started by upstart.

To check that the jobs have been started, I've added a pre-start section to each -registration job (ex for the walrus-registration job):

pre-start script
         touch /var/tmp/walrus-registration.started
end script

To enable upstart debug messages, I've added a line to the pre-script section of the eucalyptus job (since it's the first one to start the whole eucalyptus system):

        /sbin/initctl log-priority debug

I've attached the syslog generated during boot.

[Dustin Kirkland  (kirkland)]

I've made some progress, and autoregistration is now succeeding for me.

I had to ditch the "localhost" registrations, and use a real IP address. I pulled that functionality into a single place, that can be sourced by each upstart file.

As of 1am, this is working for me.

I'm uploading now, and will poke admins about accepting the package and pushing another ISO build.

:-Dustin

[Launchpad Janitor (janitor)]

This bug was fixed in the package eucalyptus - 1.6~bzr854-0ubuntu13

---------------
eucalyptus (1.6~bzr854-0ubuntu13) karmic; urgency=low

  * eucalyptus-common.eucalyptus.upstart: the -L option was in the wrong
    place, breaking the console.log logging, LP: #439161
  * eucalyptus-cc.eucalyptus-cc-registration.upstart,
    eucalyptus-cc.upstart, eucalyptus-cloud.upstart,
    eucalyptus-ipaddr.conf, eucalyptus-common.install,
    eucalyptus-sc.eucalyptus-sc-registration.upstart,
    eucalyptus-sc.upstart, eucalyptus-walrus.upstart: using "localhost"
    is breaking registration; instead, use the real, external ip
    address; consolidate the (nasty) logic for determining this
    into a single, sourcable, overridable location, LP: #437058, #438602
  * 80-eucalyptus-url: use the new $CLOUD_IP_ADDR value and logic

 -- Dustin Kirkland <email address hidden> Wed, 30 Sep 2009 01:16:12 -0500

[Thierry Carrez (ttx)]

Documented workaround for eucalyptus 1.6~bzr854-0ubuntu12 users:
$ sudo euca_conf --local-sync --register-walrus <IPADDRESSOFTHECLUSTER>
$ sudo euca_conf --local-sync --register-cluster <CLUSTERNAME> <IPADDRESSOFTHECLUSTER>
$ sudo euca_conf --local-sync --register-sc <CLUSTERNAME> <IPADDRESSOFTHECLUSTER>
$ # Registration is done. Now we need to workaround bug 439251
$ sudo stop eucalyptus
$ sudo start eucalyptus
or:
$ sudo apt-get update
$ sudo apt-get upgrade
$ # eucalyptus upgrades to 1.6~bzr854-0ubuntu13
$ sleep 60
$ # autoregistration is done. Now we need to workaround bug 439251
$ sudo stop eucalyptus
$ sudo start eucalyptus

[Scott James Remnant (Canonical) (canonical-scott)]

The dbus-reconnect fix that I assume the Upstart task refers to has been uploaded