node registration breakage due to missing keys

[Dustin Kirkland ]

Binary package hint: upstart

ubuntu@x61:~$ sudo euca_conf --no-rsync --register-nodes 192.168.1.136
[sudo] password for ubuntu:

INFO: We expect all nodes to have eucalyptus installed in //var/lib/eucalyptus/keys for key synchronization.
warning: //var/lib/eucalyptus/keys///node-cert.pem doesn't exists!
warning: //var/lib/eucalyptus/keys///cluster-cert.pem doesn't exists!
warning: //var/lib/eucalyptus/keys///node-pk.pem doesn't exists!

Trying scp to sync keys to: eucalyptus@192.168.1.136://var/lib/eucalyptus/keys/...
done.

[Dustin Kirkland  (kirkland)]

I think this should fix it:

=== modified file 'tools/euca_conf.in'
--- tools/euca_conf.in 2009-09-26 06:55:06 +0000
+++ tools/euca_conf.in 2009-09-30 00:53:21 +0000
@@ -1303,6 +1303,8 @@
                echo "INFO: We expect all nodes to have eucalyptus installed in $EUCALYPTUS/var/lib/eucalyptus/keys for key synchronization."
        fi

+ # Get the CC_NAME
+ . /etc/eucalyptus/eucalyptus.conf
        # adding (or removing) nodes
        for NEWNODE in ${NEWNODES} ; do
                # remove is simpler: just remove the node name
@@ -1318,7 +1320,7 @@
                fi

                # let's sync keys with the nodes
- if ! sync_keys ${SOURCEDIR} ${DESTDIR} ${NEWNODE} node-cert.pem cluster-cert.pem node-pk.pem cloud-cert.pem; then
+ if ! sync_keys ${SOURCEDIR}/${CC_NAME} ${DESTDIR} ${NEWNODE} node-cert.pem cluster-cert.pem node-pk.pem cloud-cert.pem; then
                        echo
                        echo "ERROR: could not synchronize keys with $NEWNODE!"
                        echo "The configuration will not have this node."

[Dustin Kirkland  (kirkland)]

The above isn't quite right; there's more to it...

:-Dustin

[Dustin Kirkland  (kirkland)]

Marking invalid.

The cause of this problem is actually due to CC registration failing. This copies the keys into the right location.

:-Dustin

[Philipp C. Heckel (binwiederhier)]

Hey guys,

I'm having the exact same problem on Ubuntu 10.04 Alpha 3. My CLC/CC/SC are on the same host (.39), one NC on a different one (.40). Are there any known solutions or workarounds?

root@cloutop2:~# euca_conf --register-nodes 192.168.1.40

INFO: We expect all nodes to have eucalyptus installed in //var/lib/eucalyptus/keys for key synchronization.
Warning: cannot file file node-cert.pem in //var/lib/eucalyptus/keys/
Warning: cannot file file cluster-cert.pem in //var/lib/eucalyptus/keys/
Warning: cannot file file node-pk.pem in //var/lib/eucalyptus/keys/
Warning: cannot file file cloud-cert.pem in //var/lib/eucalyptus/keys/

Trying rsync to sync keys with "192.168.1.40"...done.
root@cloutop2:~# euca_conf --list-nodes
// empty

Regards,
Philipp

[Adrian Bridgett (adrian-bridgett)]

Confirming Dustin's comment (but went through a bit of debug too and I think that the rsyncing is a bit flawed - not a single server that I admin has remote root access! It also silently fails on the web GUI.

Philipp - copy your keys from /var/lib/eucalyptus/keys/(clustername)/* on your cloudcontroller to /var/lib/eucalyptus/keys on your clustercontroller.

I've hit this too (1.6.2 on debian though, fresh install):
euca_conf --register-nodes 192.168.101.10

INFO: We expect all nodes to have eucalyptus installed in / for key synchronization.
Warning: cannot file file node-cert.pem in //var/lib/eucalyptus/keys/
Warning: cannot file file cluster-cert.pem in //var/lib/eucalyptus/keys/
Warning: cannot file file node-pk.pem in //var/lib/eucalyptus/keys/

I wonder if it's because I tried to use the GUI rather than the CLI to register Walrus and the Cluster?

I've written up my fairly detailed notes here:
http://smop.co.uk/mediawiki/index.php/Eucalyptus

.:
total 36
drwxr-xr-x 2 eucalyptus eucalyptus 4096 Mar 10 18:25 ashcluster
-rw-r--r-- 1 eucalyptus eucalyptus 1257 Mar 10 18:25 cloud-cert.pem
-rw-r--r-- 1 eucalyptus eucalyptus 21578 Mar 10 18:13 euca.p12
-rwxr-xr-x 1 eucalyptus eucalyptus 2834 Feb 15 22:03 nc-client-policy.xml

./examplecluster:
total 20
-rw-r--r-- 1 eucalyptus eucalyptus 1265 Mar 10 18:25 cluster-cert.pem
-rw-r--r-- 1 eucalyptus eucalyptus 1675 Mar 10 18:25 cluster-pk.pem
-rw-r--r-- 1 eucalyptus eucalyptus 1265 Mar 10 18:25 node-cert.pem
-rw-r--r-- 1 eucalyptus eucalyptus 1679 Mar 10 18:25 node-pk.pem
-rw-r--r-- 1 eucalyptus eucalyptus 512 Mar 10 18:25 vtunpass

... ah, well, kinda. (god Eucalyptus's error handling is absolute shite)

I deregistered (and then saved) the cluster from the web gui, then did this:
euca_conf --register-cluster example cluster euca.smop.co.uk
Trying rsync to sync keys with "euca.smop.co.uk"...The authenticity of host '[euca.smop.co.uk]:1022 ([192.168.101.10]:1022)' can't be established.
RSA key fingerprint is 05:88:ec:6b:ea:a0:10:25:7a:62:15:75:2d:19:9a:03.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[euca.smop.co.uk]:1022,[192.168.101.10]:1022' (RSA) to the list of known hosts.
<email address hidden>'s password:
Permission denied, please try again.
...
ERROR: failed to sync keys with euca.smop.co.uk; registration will not be complete until keys can be synced, please try again.

I have this issue with adding nodes too - basically the fact that it tries to rsync as root to remote hosts fails in _all_ my setups (for obvious security reasons). At least the node registration gives details of how to do this manually.

[Steffen Möller (moeller-debian)]

I was very happy to have found this bug report. Please consider to mention already in the error message the likely cause of a missing <code>euca_conf --register-cluster ... </code> and/or a problem with the forwarding of the such created keys to the nodes. Thanks, SM