Unable to register nodes in eucalyptus

[H.M.]

My topology is as follows: (Server names & Corresponding coponents)

SRV-UEC-QA-CC: Contains CC, CLC, Walrus, SC
SRV-UEC-QA-NODE01: Contains NC
SRV-UEC-QA-NODE02: Contains NC

NOTE: I have another problem where eucalyptus-cc & uec-component-listener are not started by default. (See Bug 598182) In order to deal with this problem, I start the eucalyptus-cc and uec-component-listener services manually and there are no indications of failure.

-----------------------------------------------------------------------------------------------

PROBLEM: I am unable to get my nodes to register. My Eucalyptus cluster is unusable as a result.

Please visit this for more background: http://open.eucalyptus.com/forum/cc-not-able-registersee-nodes

Here is an example of the issue:

root@srv-uec-qa-cc:/var/log/eucalyptus# euca_conf --list-nodes

root@srv-uec-qa-cc:/var/log/eucalyptus# euca_conf --deregister-nodes "10.1.0.2 10.1.0.3"
SUCCESS: removed node '10.1.0.2'
SUCCESS: removed node '10.1.0.3'
root@srv-uec-qa-cc:/var/log/eucalyptus# euca_conf --deregister-cluster qa-cluster

SUCCESS: cluster 'qa-cluster' successfully deregistered.
root@srv-uec-qa-cc:/var/log/eucalyptus# euca_conf --register-cluster qa-cluster 10.0.10.6

Trying rsync to sync keys with "10.0.10.6"...done.

SUCCESS: new cluster 'qa-cluster' on host '10.0.10.6' successfully registered.
root@srv-uec-qa-cc:/var/log/eucalyptus# euca_conf --register-nodes "10.1.0.2 10.1.0.3"

INFO: We expect all nodes to have eucalyptus installed in //var/lib/eucalyptus/keys for key synchronization.

Trying rsync to sync keys with "10.1.0.2"...done.

Trying rsync to sync keys with "10.1.0.3"...done.
root@srv-uec-qa-cc:/var/log/eucalyptus# euca_conf --list-nodes

root@srv-uec-qa-cc:/var/log/eucalyptus#

-----------------------------------------------------------------------------------------------

Here is a list of my packages:

root@srv-uec-qa-cc:/var/log/eucalyptus# dpkg -l|grep -i -e eucalyptus -e uec
ii euca2ools 1.2-0ubuntu10 managing cloud instances for Eucalyptus
ii eucalyptus-cc 1.6.2-0ubuntu30 Elastic Utility Computing Architecture - Clu
ii eucalyptus-cloud 1.6.2-0ubuntu30 Elastic Utility Computing Architecture - Clo
ii eucalyptus-common 1.6.2-0ubuntu30 Elastic Utility Computing Architecture - Com
ii eucalyptus-gl 1.6.2-0ubuntu30 Elastic Utility Computing Architecture - Log
ii eucalyptus-java-common 1.6.2-0ubuntu30 Elastic Utility Computing Architecture - Com
ii eucalyptus-sc 1.6.2-0ubuntu30 Elastic Utility Computing Architecture - Sto
ii eucalyptus-walrus 1.6.2-0ubuntu30 Elastic Utility Computing Architecture - Wal
ii libeucalyptus-commons-ext-java 0.5.0-0ubuntu2 Eucalyptus commons external Java library
ii uec-component-listener 1.6.2-0ubuntu30 Ubuntu Enterprise Cloud - Component listener

[H.M. (cybrocop)]

I was asked on IRC to capture the log files when the bug occurs. Thus, I have:
-Stopped all euca* services
-Rolled over all log files
-Rebooted system

I logged in and here is proof that eucalyptus-cc and uec-component-listener are still not running.

Attached you will find all my log files:

root@srv-uec-qa-cc:~# cd /var/log/eucalyptus/
root@srv-uec-qa-cc:/var/log/eucalyptus# ls -al
total 12
drwxr-xr-x 2 eucalyptus eucalyptus 4096 2010-06-25 20:37 .
drwxr-xr-x 14 root root 4096 2010-06-25 20:36 ..
-rw-r--r-- 1 eucalyptus eucalyptus 109 2010-06-25 20:37 jetty-request-2010_06_25.log
root@srv-uec-qa-cc:/var/log/eucalyptus# status eucalyptus
eucalyptus start/running, process 931
root@srv-uec-qa-cc:/var/log/eucalyptus# status eucalyptus-cc
eucalyptus-cc stop/waiting
root@srv-uec-qa-cc:/var/log/eucalyptus# status uec-component-listener
uec-component-listener stop/waiting
root@srv-uec-qa-cc:/var/log/eucalyptus# stop eucalyptus
eucalyptus stop/waiting

The /var/log/eucalyptus/ directory only contains jetty-request-2010_06_25.log, it doesn't contain any other euca* log file presumably because eucalyptus-cc and uec-component-listener are not running.

Attached is my syslog file.

[H.M. (cybrocop)]

Please ignore comment #1 it was meant for another bug.

[H.M. (cybrocop)]

Since I need to reinstall the system, I have made one final attempt at capturing a clean set of logs:

I have:
-- Stopped euca*
-- Deleted everything under /var/log/eucalyptus
-- Started eucalyptus
-- Ran a series of commands (see everything after ----)
-- Stopped eucalyptus

The logs are attached.

----

root@srv-uec-qa-cc:/var/log/eucalyptus# start eucalyptus
eucalyptus start/running, process 2615
root@srv-uec-qa-cc:/var/log/eucalyptus# euca_conf --list-clusters
registered clusters:
   qa-cluster 10.0.10.6
root@srv-uec-qa-cc:/var/log/eucalyptus# euca_conf --list-nodes

root@srv-uec-qa-cc:/var/log/eucalyptus# euca_conf --deregister-nodes "10.1.0.2 10.1.0.3"
SUCCESS: removed node '10.1.0.2'
SUCCESS: removed node '10.1.0.3'
root@srv-uec-qa-cc:/var/log/eucalyptus# euca_conf --deregister-nodes "10.1.0.2 10.1.0.3"
Node 10.1.0.2 is not known
Node 10.1.0.3 is not known
root@srv-uec-qa-cc:/var/log/eucalyptus# euca_conf --register-nodes "10.1.0.2 10.1.0.3"

INFO: We expect all nodes to have eucalyptus installed in //var/lib/eucalyptus/keys for key synchronization.

Trying rsync to sync keys with "10.1.0.2"...done.

Trying rsync to sync keys with "10.1.0.3"...done.
root@srv-uec-qa-cc:/var/log/eucalyptus# euca_conf --list-nodes

root@srv-uec-qa-cc:/var/log/eucalyptus#

[H.M. (cybrocop)]

I'm also uploading my configuration files.

[Dave Walker (davewalker)]

@H.M.: Thanks for providing the detail requested, I understand directly following this bug you did a clean reinstall. Can you confirm if the symptoms are still happening in the clean install?

[Thierry Carrez (ttx)]

How did you setup UEC ? From the ISO installer or by manually installing packages ?

[Hovik Manucharyan (hovik)]

@Thiery, I used PackageInstall procedure.

[Thierry Carrez (ttx)]

If you followed the manual instructions, are you sure you distributed the components keys correctly ? In particular, that the NC has the CC's public key in /var/lib/eucalyptus/.ssh/authorized_keys ?

[Mike Guo (hylinux)]

I got this issue too. but I am install the UEC from CD.
I didn't change anything. everything is automatic install.

but when I am run :
euca_conf --list-noes.
there is nothing.
and I try to run: euca_conf --register-nodes 192.168.0.100. and --list-nodes again. there is noting.

so, I ssh into my nc server.
I found :
root@node1:/var/lib/eucalyptus/.ssh# ll
total 12
drwxr-xr-x 2 eucalyptus root 4096 2010-08-13 12:24 ./
drwxr-xr-x 6 eucalyptus eucalyptus 4096 2010-08-13 12:25 ../
-rw------- 1 eucalyptus root 402 2010-08-13 12:24 authorized_keys

not sure this is correct.

[Dave Walker (davewalker)]

@Mike, Can you confirm what Ubuntu CD release this was with?

Thanks.

[C de-Avillez (hggdh2)]

Also, can you all please make sure you have the current version of Eucalyptus for 10.04 (1.6.2-0ubuntu30.3)? If not, please upgrade and try again.

[Stefan Wasilewski (swasilewski)]

I experience the same problem using 2.0+bzr1241-0ubuntu4 on 10.10, also using installation via packages.

[Launchpad Janitor (janitor)]

[Expired for eucalyptus (Ubuntu) because there has been no activity for 60 days.]