Node not added to CLC even though registration is successful

[Mathias Gug]

While installing 5 NCs at the same time registration seemed to have worked correclty:

$ cat /var/log/eucalyptus/registration.log
2010-02-11 20:21:44-05:00 | 2290 -> Calling walrus Walrus 10.55.55.2
2010-02-11 20:21:44-05:00 | 2291 -> Calling storage UEC-TEST1 storage 10.55.55.2
2010-02-11 20:21:44-05:00 | 2292 -> Calling cluster UEC-TEST1 10.55.55.2
2010-02-11 20:21:44-05:00 | 2291 -> Cluster UEC-TEST1 doesn't exist, retrying in 10 seconds.
2010-02-11 20:22:07-05:00 | 2292 -> euca_conf --register-cluster returned 0
2010-02-11 20:22:08-05:00 | 2290 -> euca_conf --register-walrus returned 0
2010-02-11 20:22:15-05:00 | 2291 -> euca_conf --register-sc returned 0
2010-02-11 20:42:35-05:00 | 10378 -> Calling node Cloud node 10.55.55.5
2010-02-11 20:42:57-05:00 | 10378 -> euca_conf --register-nodes returned 0
2010-02-11 20:43:17-05:00 | 10915 -> Calling node Cloud node #2 10.55.55.7
2010-02-11 20:43:39-05:00 | 10915 -> euca_conf --register-nodes returned 0
2010-02-11 20:43:42-05:00 | 11236 -> Calling node Cloud node #3 10.55.55.4
2010-02-11 20:43:42-05:00 | 11236 -> euca_conf --register-nodes returned 0
2010-02-11 20:44:43-05:00 | 12024 -> Calling node Cloud node #4 10.55.55.3
2010-02-11 20:45:04-05:00 | 12024 -> euca_conf --register-nodes returned 0
2010-02-11 20:45:27-05:00 | 12647 -> Calling node Cloud node #5 10.55.55.6
2010-02-11 20:45:48-05:00 | 12647 -> euca_conf --register-nodes returned 0

However only 4 NCs are known to UEC:
ubuntu@cempedak:~$ sudo euca_conf --list-nodes
registered nodes:
   10.55.55.3 UEC-TEST1
   10.55.55.5 UEC-TEST1
   10.55.55.6 UEC-TEST1
   10.55.55.7 UEC-TEST1

ubuntu@cempedak:~$ euca-describe-availability-zones verbose
AVAILABILITYZONE UEC-TEST1 10.55.55.2
AVAILABILITYZONE |- vm types free / max cpu ram disk
AVAILABILITYZONE |- m1.small 0064 / 0064 1 128 2
AVAILABILITYZONE |- c1.medium 0064 / 0064 1 256 5
AVAILABILITYZONE |- m1.large 0032 / 0032 2 512 10
AVAILABILITYZONE |- m1.xlarge 0032 / 0032 2 1024 20
AVAILABILITYZONE |- c1.xlarge 0016 / 0016 4 2048 20

The expected number of available ressources should be 80 instead of 64.

ProblemType: Bug
Architecture: amd64
Date: Thu Feb 11 21:12:18 2010
DistroRelease: Ubuntu 10.04
Package: eucalyptus-cloud 1.6.2~bzr1189-0ubuntu1
ProcEnviron:
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcVersionSignature: Ubuntu 2.6.32-13.18-server
SourcePackage: eucalyptus
Tags: lucid
Uname: Linux 2.6.32-13-server x86_64

[Dustin Kirkland  (kirkland)]

How big is your available PUBLIC_IPS?

[Daniel Nurmi (nurmi)]

Since '--list-nodes' is not reporting '10.55.55.4' as registered, my guess is that the 'euca_conf --register-nodes 10.55.55.4' did not complete successfully. Can you check your nodes.list on the CC to see if '10.55.55.4' was added to the list? If so, then we should check to make sure the CC is trying to call the node (cc.log) , and if so, whether the node is being successfully contacted (cc.log).

[Thierry Carrez (ttx)]

Registration does two things: sync keys from CC to NC's /var/lib/eucalyptus/keys/ and add the node IP to the CC /var/lib/eucalyptus/nodes.list. Could you check what completed successfully on the failing node, if any ?

If both appear to have completed successfully, then something prevents the CC to "see" the latest IP. If one (or all) of those appear to have failed, then at the very minimum, there is a bug in euca_conf --register-nodes (returned 0 while registration was not complete).

[Mathias Gug (mathiaz)]

The failing node (10.55.55.4) hasn't been added to nodes.list:

ubuntu@cempedak:~$ sudo cat /var/lib/eucalyptus/nodes.list

10.55.55.5
10.55.55.7
10.55.55.3
10.55.55.6

On the failing node (10.55.55.4) there isn't any key available:

ubuntu@mamoncillo:~$ sudo ls -l /var/lib/eucalyptus/keys/
total 0
ubuntu@mamoncillo:~$

So it seems that there is a bug in euca_conf --register-nodes.

[Thierry Carrez (ttx)]

Looking at the code, it seems that if sync_keys fails, euca_conf --register-nodes will still return 0 (and will skip nodes.list update).