eucalyptus-cc and uec-component-listener services are not started

[H.M.]

On Lucid, when the system starts, uec-component-listener and eucalyptus-cc are not running.

List of my packages:

root@srv-uec-qa-cc:/var/log/eucalyptus# dpkg -l|grep -i -e eucalyptus -e uec
ii euca2ools 1.2-0ubuntu10 managing cloud instances for Eucalyptus
ii eucalyptus-cc 1.6.2-0ubuntu30 Elastic Utility Computing Architecture - Clu
ii eucalyptus-cloud 1.6.2-0ubuntu30 Elastic Utility Computing Architecture - Clo
ii eucalyptus-common 1.6.2-0ubuntu30 Elastic Utility Computing Architecture - Com
ii eucalyptus-gl 1.6.2-0ubuntu30 Elastic Utility Computing Architecture - Log
ii eucalyptus-java-common 1.6.2-0ubuntu30 Elastic Utility Computing Architecture - Com
ii eucalyptus-sc 1.6.2-0ubuntu30 Elastic Utility Computing Architecture - Sto
ii eucalyptus-walrus 1.6.2-0ubuntu30 Elastic Utility Computing Architecture - Wal
ii libeucalyptus-commons-ext-java 0.5.0-0ubuntu2 Eucalyptus commons external Java library
ii uec-component-listener 1.6.2-0ubuntu30 Ubuntu Enterprise Cloud - Component listener

Once I manually start uec-component-listener and eucalyptus-cc, I am not able to register any nodes. (Here is some background: http://open.eucalyptus.com/forum/cc-not-able-registersee-nodes) There is also a previous bug on the registration issue: https://bugs.launchpad.net/ubuntu/+source/eucalyptus/+bug/504704 but I believe my issue is slightly different.

I am not sure if the registration problem is independent or related to the startup issue.

[Torsten Spindler (tspindler)]

Is the cluster registered in the cloud at all? Do you see it with
euca-describe-availability-zones?

[H.M. (cybrocop)]

The registration problem is filed as Bug #598186

[Hovik Manucharyan (hovik)]

Yes, I'm able to see it with euca-describe-availability-zones:

$ euca-describe-availability-zones verbose
AVAILABILITYZONE qa-cluster 10.0.10.6
AVAILABILITYZONE |- vm types free / max cpu ram disk
AVAILABILITYZONE |- m1.small 0000 / 0000 1 512 5
AVAILABILITYZONE |- c1.medium 0000 / 0000 1 900 20
AVAILABILITYZONE |- m1.large 0000 / 0000 2 1800 40
AVAILABILITYZONE |- m1.xlarge 0000 / 0000 3 2700 60
AVAILABILITYZONE |- c1.xlarge 0000 / 0000 4 3600 80

[H.M. (cybrocop)]

Since I need to reinstall the system, I have made one final attempt at capturing a clean set of logs:

I have:
-- Stopped euca*
-- Deleted everything under /var/log/eucalyptus
-- Started eucalyptus
-- Ran a series of commands (see everything after ----)
-- Stopped eucalyptus

The logs are attached.

----

root@srv-uec-qa-cc:/var/log/eucalyptus# start eucalyptus
eucalyptus start/running, process 2615
root@srv-uec-qa-cc:/var/log/eucalyptus# euca_conf --list-clusters
registered clusters:
   qa-cluster 10.0.10.6
root@srv-uec-qa-cc:/var/log/eucalyptus# euca_conf --list-nodes

root@srv-uec-qa-cc:/var/log/eucalyptus# euca_conf --deregister-nodes "10.1.0.2 10.1.0.3"
SUCCESS: removed node '10.1.0.2'
SUCCESS: removed node '10.1.0.3'
root@srv-uec-qa-cc:/var/log/eucalyptus# euca_conf --deregister-nodes "10.1.0.2 10.1.0.3"
Node 10.1.0.2 is not known
Node 10.1.0.3 is not known
root@srv-uec-qa-cc:/var/log/eucalyptus# euca_conf --register-nodes "10.1.0.2 10.1.0.3"

INFO: We expect all nodes to have eucalyptus installed in //var/lib/eucalyptus/keys for key synchronization.

Trying rsync to sync keys with "10.1.0.2"...done.

Trying rsync to sync keys with "10.1.0.3"...done.
root@srv-uec-qa-cc:/var/log/eucalyptus# euca_conf --list-nodes

root@srv-uec-qa-cc:/var/log/eucalyptus#

[H.M. (cybrocop)]

Here are my logs from the interaction above:

[H.M. (cybrocop)]

Please ignore comment #4 and comment #5, they were for another bug.

[H.M. (cybrocop)]

I was asked on IRC to capture the log files when the bug occurs. Thus, I have:
-Stopped all euca* services
-Rolled over all log files
-Rebooted system

I logged in and here is proof that eucalyptus-cc and uec-component-listener are still not running.

Attached you will find all my log files:

root@srv-uec-qa-cc:~# cd /var/log/eucalyptus/
root@srv-uec-qa-cc:/var/log/eucalyptus# ls -al
total 12
drwxr-xr-x 2 eucalyptus eucalyptus 4096 2010-06-25 20:37 .
drwxr-xr-x 14 root root 4096 2010-06-25 20:36 ..
-rw-r--r-- 1 eucalyptus eucalyptus 109 2010-06-25 20:37 jetty-request-2010_06_25.log
root@srv-uec-qa-cc:/var/log/eucalyptus# status eucalyptus
eucalyptus start/running, process 931
root@srv-uec-qa-cc:/var/log/eucalyptus# status eucalyptus-cc
eucalyptus-cc stop/waiting
root@srv-uec-qa-cc:/var/log/eucalyptus# status uec-component-listener
uec-component-listener stop/waiting
root@srv-uec-qa-cc:/var/log/eucalyptus# stop eucalyptus
eucalyptus stop/waiting

The /var/log/eucalyptus/ directory only contains jetty-request-2010_06_25.log, it doesn't contain any other euca* log file presumably because eucalyptus-cc and uec-component-listener are not running.

Attached is my syslog file.

[Dave Walker (davewalker)]

@H.M.: Thanks for providing the detail requested, I understand directly following this bug you did a clean reinstall. Can you confirm if the symptoms are still happening in the clean install?

[Thierry Carrez (ttx)]

Also how did you install UEC in the first place ? Using the installer on the ISO, or by manually installing the packages ?

[Hovik Manucharyan (hovik)]

Thierry, I installed using the PackageInstall instructions.

[Hovik Manucharyan (hovik)]

Dave, after a clean reinstall, one time I was able to reboot and raise the bug. But after I rebooted again, the services started up normally on each reboot.

I will try to keep an eye out for it.

[Hovik Manucharyan (hovik)]

I keep getting this bug in a random manner. Some times after a restart, the 2 services start properly and other times I have to manually start them.

[Thierry Carrez (ttx)]

uec-component-listener starts when eucalyptus-cc starts, so the issue is probably in eucalyptus-cc.
Anything in the cc.log that would show why it doesn't start ?

[Launchpad Janitor (janitor)]

[Expired for eucalyptus (Ubuntu) because there has been no activity for 60 days.]