juju setup fails, ERROR Invalid SSH key - 12.04 LTS

[Erick Wipprecht]

When creating a new Cloud environment following MAAS documentation, setup fails when configuring juju (per https://wiki.ubuntu.com/ServerTeam/MAAS/Juju). environments.yaml has been created and ssh keys (both RSA & DSA) have been created. Juju bootstrap appears to work correctly but subsequent attempts to communicate with member servers results in SSH key errors:

root@n7cldmaas01:~# juju bootstrap
2012-06-19 09:55:59,436 INFO Bootstrapping environment 'maas' (origin: distro type: maas)...
2012-06-19 09:56:02,405 INFO 'bootstrap' command finished successfully
root@n7cldmaas01:~# juju status
2012-06-19 09:56:13,688 INFO Connecting to environment...
The authenticity of host 'n7cldpoc01.blah.com (10.194.142.11)' can't be established.
ECDSA key fingerprint is 5f:2a:0e:e7:b0:0e:10:fa:cf:d8:43:90:80:2f:d0:3d.
Are you sure you want to continue connecting (yes/no)? yes
2012-06-19 09:56:15,322 ERROR Invalid SSH key
2012-06-19 09:56:44,093 ERROR Invalid SSH key
2012-06-19 09:57:14,213 ERROR Invalid SSH key
...

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Julian Edwards (julian-edwards)]

Bootstrapping is not instant, you need to wait for the master node to install before status works.

[Erick Wipprecht (erick-wipprecht)]

How long? A week? I've had one instance sitting three days after a bootstrap and it still doesn't work.

[Julian Edwards (julian-edwards)]

Please see the first two questions on https://answers.launchpad.net/maas/+faqs

[Erick Wipprecht (erick-wipprecht)]

Time synchronization doesn't seem to be my issue here, unfortunately. All servers were within 1.5 seconds of each other.

[Erick Wipprecht (erick-wipprecht)]

Given that this looked like it might just be an ssh key issue (which is certainly a problem -- I wasn't able to ssh to the member servers as the ubuntu user) I set a password in maas.preseed and added keys to authorized_keys post-installation. That fixed the initial key problem but juju seems to not be able to communicate with these servers:

root@maas01:~# juju -v status
2012-06-25 15:04:57,585 DEBUG Initializing juju status runtime
2012-06-25 15:04:57,593 INFO Connecting to environment...
2012-06-25 15:04:57,715 DEBUG Connecting to environment using server02.blah.com...
2012-06-25 15:04:57,715 DEBUG Spawning SSH process with remote_user="ubuntu" remote_host="server02.blah.com" remote_port="2181" local_port="39073".
2012-06-25 15:04:58,220:11766(0x7fbdcbf5d700):ZOO_INFO@log_env@658: Client environment:zookeeper.version=zookeeper C client 3.3.5
2012-06-25 15:04:58,220:11766(0x7fbdcbf5d700):ZOO_INFO@log_env@662: Client environment:host.name=maas01
2012-06-25 15:04:58,220:11766(0x7fbdcbf5d700):ZOO_INFO@log_env@669: Client environment:os.name=Linux
2012-06-25 15:04:58,220:11766(0x7fbdcbf5d700):ZOO_INFO@log_env@670: Client environment:os.arch=3.2.0-25-generic
2012-06-25 15:04:58,220:11766(0x7fbdcbf5d700):ZOO_INFO@log_env@671: Client environment:os.version=#40-Ubuntu SMP Wed May 23 20:30:51 UTC 2012
2012-06-25 15:04:58,221:11766(0x7fbdcbf5d700):ZOO_INFO@log_env@679: Client environment:user.name=root
2012-06-25 15:04:58,221:11766(0x7fbdcbf5d700):ZOO_INFO@log_env@687: Client environment:user.home=/root
2012-06-25 15:04:58,221:11766(0x7fbdcbf5d700):ZOO_INFO@log_env@699: Client environment:user.dir=/root
2012-06-25 15:04:58,221:11766(0x7fbdcbf5d700):ZOO_INFO@zookeeper_init@727: Initiating client connection, host=localhost:39073 sessionTimeout=10000 watcher=0x7fbdc9f016b0 sessionId=0 sessionPasswd=<null> context=0x2c07630 flags=0
2012-06-25 15:04:58,221:11766(0x7fbdc6c5b700):ZOO_INFO@check_events@1585: initiated connection to server [127.0.0.1:39073]
2012-06-25 15:04:58,222:11766(0x7fbdc6c5b700):ZOO_ERROR@handle_socket_error_msg@1603: Socket [127.0.0.1:39073] zk retcode=-4, errno=112(Host is down): failed while receiving a server response
2012-06-25 15:05:01,558:11766(0x7fbdc6c5b700):ZOO_ERROR@handle_socket_error_msg@1579: Socket [127.0.0.1:39073] zk retcode=-4, errno=111(Connection refused): server refused to accept the client
...

This seems less like a problem with juju itself but the MAAS setup before that -- cloud-init, perhaps?

[Julian Edwards (julian-edwards)]

That looks like a connection refused to zookeeper, nothing to do with maas. I suspect your zk node hasn't come up properly, which might be related to bug 992075

[Erick Wipprecht (erick-wipprecht)]

Well... there *isn't* a zookeeper node. Perhaps this is just a dependency issue in one of the packages?

[Julian Edwards (julian-edwards)]

The ZK node is the first one to be booted by Juju when you run "juju bootstrap". Juju installs ZK on it.

Since ZK is not there, you have a problem at that stage. Since you managed to log in, can you examine the cloud-init logs please? If there's nothing sensitive in there perhaps attach to the bug.

[Erick Wipprecht (erick-wipprecht)]

Curious. I was able to get one of the servers to actually function correctly as the ZK node -- not sure why that one is different since it was built just like the other two nodes. Anyway, logs are attached for all three as well as the cloud-init-output.log log from that one functioning server (server03; neither of the other two servers have this logfile).

[Marcel Miersebach (keamas)]

I have the same issue here and looking for a working solution...

I also would like to setup a MAAS for OpenStack with Ubuntu 12.04

Everything workes great to this point
When i would like to bootstrap juju i get a error of a invalid ssh key
I did everything like in the ubuntu MAAS wiki documentation

[QUOTE]marcel@ubuntu20:~$ juju status 2012-07-09 13:00:02,559 INFO Connecting to environment... The authenticity of host 'node1 (10.110.11.71)' can't be established. ECDSA key fingerprint is 29:42:2c:7a:ef:52:d4:f8:63:51:d8:a1:1e:a9:16:0e. Are you sure you want to continue connecting (yes/no)? yes 2012-07-09 13:00:05,883 ERROR Invalid SSH key[/QUOTE]

I read some more people have this issu not sure if it is a BUG but I couldn't find a working solution... I read this Posting [url]http://askubuntu.com/questions/147714/invalid-ssh-key-error-in-juju-when-using-it-with-maas[/url]

I added a Root User to this File that I can Access the Nodes: /var/lib/cobbler/kickstarts/maas.preseed:

dns time everything is correct. I can also ssh to the machine
It seems that there is no ubuntu user so i added one And created a ssh key with ssh- without passphrase
Than i copied the keys and now i can ssh with ubuntu user without a pw

Did i messed something up with the keys ? Or the user pw ?

[QUOTE]marcel@ubuntu20:~$ juju status
2012-07-09 15:43:12,130 INFO Connecting to environment...
Warning: the ECDSA host key for 'node1' differs from the key for the IP address '10.110.11.71'
Offending key for IP in /home/marcel/.ssh/known_hosts:1
Matching host key in /home/marcel/.ssh/known_hosts:8
Are you sure you want to continue connecting (yes/no)? yes
A clouser look showes me this issues:[/QUOTE]

[QUOTE]marcel@ubuntu20:~$ juju -v status
2012-07-09 15:36:49,458 DEBUG Initializing juju status runtime
2012-07-09 15:36:49,469 INFO Connecting to environment...
2012-07-09 15:36:49,588 DEBUG Connecting to environment using node1...
2012-07-09 15:36:49,589 DEBUG Spawning SSH process with remote_user="ubuntu" remote_host="node1" remote_port="2181" local_port="43964".
Warning: the ECDSA host key for 'node1' differs from the key for the IP address '10.110.11.71'
Offending key for IP in /home/marcel/.ssh/known_hosts:1
Matching host key in /home/marcel/.ssh/known_hosts:8
Are you sure you want to continue connecting (yes/no)? yes
2012-07-09 15:36:53,098:16474(0x7f0c16d58700):ZOO_INFO@log_env@658: Client environment:zookeeper.version=zookeeper C client 3.3.5
2012-07-09 15:36:53,098:16474(0x7f0c16d58700):ZOO_INFO@log_env@662: Client environment:host.name=ubuntu20
2012-07-09 15:36:53,098:16474(0x7f0c16d58700):ZOO_INFO@log_env@669: Client environment:os.name=Linux
2012-07-09 15:36:53,098:16474(0x7f0c16d58700):ZOO_INFO@log_env@670: Client environment:os.arch=3.2.0-23-generic
2012-07-09 15:36:53,098:16474(0x7f0c16d58700):ZOO_INFO@log_env@671: Client environment:os.version=#36-Ubuntu SMP Tue Apr 10 20:39:51 UTC 2012
2012-07-09 15:36:53,099:16474(0x7f0c16d58700):ZOO_INFO@log_env@679: Client environment:user.name=marcel
2012-07-09 15:36:53,099:16474(0x7f0c16d58700):ZOO_INFO@log_env@687: Client environment:user.home=/home/marcel
2012-07-09 15:36:53,099:16474(0x7f0c16d58700):ZOO_INFO@log_env@699: ...

[Marcel Miersebach (keamas)]

I think the Problem is a bug because there is no ubuntu User on my installation:

root@node5:~# cat /etc/passwd
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/bin/sh
bin:x:2:2:bin:/bin:/bin/sh
sys:x:3:3:sys:/dev:/bin/sh
sync:x:4:65534:sync:/bin:/bin/sync
games:x:5:60:games:/usr/games:/bin/sh
man:x:6:12:man:/var/cache/man:/bin/sh
lp:x:7:7:lp:/var/spool/lpd:/bin/sh
mail:x:8:8:mail:/var/mail:/bin/sh
news:x:9:9:news:/var/spool/news:/bin/sh
uucp:x:10:10:uucp:/var/spool/uucp:/bin/sh
proxy:x:13:13:proxy:/bin:/bin/sh
www-data:x:33:33:www-data:/var/www:/bin/sh
backup:x:34:34:backup:/var/backups:/bin/sh
list:x:38:38:Mailing List Manager:/var/list:/bin/sh
irc:x:39:39:ircd:/var/run/ircd:/bin/sh
gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/bin/sh
nobody:x:65534:65534:nobody:/nonexistent:/bin/sh
libuuid:x:100:101::/var/lib/libuuid:/bin/sh
syslog:x:101:103::/home/syslog:/bin/false
messagebus:x:102:105::/var/run/dbus:/bin/false
whoopsie:x:103:108::/nonexistent:/bin/false
avahi:x:104:109:Avahi mDNS daemon,,,:/var/run/avahi-daemon:/bin/false
landscape:x:105:111::/var/lib/landscape:/bin/false
sshd:x:106:65534::/var/run/sshd:/usr/sbin/nologin

[Julian Edwards (julian-edwards)]

On Wednesday 27 June 2012 15:45:17 you wrote:
> Curious. I was able to get one of the servers to actually function
> correctly as the ZK node -- not sure why that one is different since it
> was built just like the other two nodes. Anyway, logs are attached for
> all three as well as the cloud-init-output.log log from that one
> functioning server (server03; neither of the other two servers have this
> logfile).
>
> ** Attachment added: "cloud-init-logs.tgz"
>
> https://bugs.launchpad.net/ubuntu/+source/juju/+bug/1015207/+attachment/320
> 5714/+files/cloud-init-logs.tgz

The critical difference in the logs is here:

server02:
Jun 21 13:08:36 server02 [CLOUDINIT] __init__.py[DEBUG]: handling scripts-user
with freq=None and args=[]
Jun 21 13:08:36 server02 [CLOUDINIT] cc_scripts_user.py[WARNING]: failed to
run-parts in /var/lib/cloud/instance/scripts

Did you put your own cloudinit scripts in which are now failing?

[Julian Edwards (julian-edwards)]

> I think the Problem is a bug because there is no ubuntu User on my installation:

This is because it didn't finish commissioning. You need to examine the logs as above to find out where the problem is.

[Erick Wipprecht (erick-wipprecht)]

In my case, at least, there aren't any non-standard scripts in play -- this is a bone-stock, by the book install per the Ubuntu docs. The only thing that really deviates at all is the cobbler maas.preseed password addition. Failure behavior is not different with that in place; that was added simply to allow me to log onto the servers in question for troubleshooting purposes. So far I haven't been able to spot the cause in the logs.

[Scott Moser (smoser)]

If there is no ubuntu user, then the install actually failed. The logs for that are in /var/log/maas/rsyslog you might be able to find some errors in the installation. The fact that you edited the preseed with regard to the ubuntu user, and the ubuntu user didn't get created seem likely to be related.

Also, /var/lib/cloud on server02 would likely be useful. I'd like to see what is in /var/lib/cloud/instance/scripts. but the whole thing would be nice to have attached.

[Erick Wipprecht (erick-wipprecht)]

The ubuntu user is being created for me fine. Nothing really leaped out at me in the logs -- they're attached.

[Ondrej Koch (o-koch)]

What helped us was simply rebooting the node using PXE (the node should be in allocated state). The node in ready state ends up shut down. So basically what juju does is sending a wake-on-lan signal to start it. If you (like us) powered the node up in ready state, you end up with this particular problem.
I hope it doen't matter we hacked the images a little so we could ssh to them to see what's going on and imported ssh keys manually... it shouldn't...

[Shawn Rapp (shawn-rapp)]

I think Ondrej is on to something.... I had problems earlier attaching nodes to maas by trying to read between the lines on the instructions instead of doing literal. One problem on my setup of nodes is i didnt set pxe as first boot order instead just left the default. So it would pxe boot the initial install but not commission gracefully. Further on at the juju step i too manually powered the unit on as well as still not using pxe as first boot order.
I think maybe the bug is on documentation not telling us what we shouldnt be doing correcting us from thinking we know what we are doing. It just seems like its a id10t error on my part by not doing a step right or more likely over complicating something.

[Francis J. Lacoste (flacoste)]

Have we identified if this was a problem with Juju or with MAAS yet?

[Clint Byrum (clint-fewbar)]

Can somebody please explain clearly what the bug in juju is? Thanks!

[Julian Edwards (julian-edwards)]

I am closing the maas task because a completely updated version of maas is about to be SRUed which will be likely to make this go away.

[Sonali Jadhav (sonali-m)]

hi julian, i am sorry but i am still facing this same issue.

Are you sure you want to continue connecting (yes/no)? The authenticity of host 'a3qcg.master (172.19.178.50)' can't be established.
ECDSA key fingerprint is 3b:9a:c4:1c:3f:e8:28:0e:82:e9:74:f9:22:bc:5a:5a.
Are you sure you want to continue connecting (yes/no)? The authenticity of host 'a3qcg.master (172.19.178.50)' can't be established.
ECDSA key fingerprint is 3b:9a:c4:1c:3f:e8:28:0e:82:e9:74:f9:22:bc:5a:5a.
Are you sure you want to continue connecting (yes/no)?

[Julian Edwards (julian-edwards)]

Sonali, that is a standard message the first time you connect to a new host with ssh.