ensemble deploy needs better error messages

[Gustavo Niemeyer]

ensemble deploy will report in an incomprehensible manner when the user tries to deploy something and the machine hasn't finished initializing yet. It should detect that situation, and report the problem appropriately.

Note that this problem may happen in three situations:

a) The machine hasn't yet finished booted (SSH isn't available)
b) The machine has booted, but zookeeper isn't running
c) The machine has booted, zookeeper is running, but has not been initialized

All the three cases should return a nice error message to the user, explaining that initialization is still in progress.

[Kapil Thangavelu (hazmat)]

Example out as currently identified by the this bug

(py)kapil@realms-laptop:~/canonical/ensemble/trunk$ ./bin/ensemble --verbose=junk deploy --repository=../formula-repo cassandra
2010-09-17 16:10:58,855 DEBUG Entering run-loop...
usage: ensemble deploy [-h] --repository REPOSITORY [--environment ENVIRONMENT] formula [service_name]
ensemble deploy: error: [Failure instance: Traceback (failure with no frames): <class 'ensemble.errors.NoConnection'>: No Ensemble Nodes
]
(py)kapil@realms-laptop:~/canonical/ensemble/trunk$ ./bin/ensemble --verbose=junk deploy --repository=../formula-repo cassandra
2010-09-17 16:11:37,997 DEBUG Entering run-loop...
2010-09-17 16:11:39,595 DEBUG Spawning SSH process with remote_user="ubuntu" remote_host="ec2-184-72-197-49.compute-1.amazonaws.com" remote_\
port="2181" local_port="22181".
2010-09-17 16:11:46,609:3927(0xb78546c0):ZOO_INFO@log_env@658: Client environment:zookeeper.version=zookeeper C client 3.4.0
2010-09-17 16:11:46,609:3927(0xb78546c0):ZOO_INFO@log_env@662: Client environment:host.name=realms-laptop
2010-09-17 16:11:46,609:3927(0xb78546c0):ZOO_INFO@log_env@669: Client environment:os.name=Linux
2010-09-17 16:11:46,609:3927(0xb78546c0):ZOO_INFO@log_env@670: Client environment:os.arch=2.6.32-24-generic
2010-09-17 16:11:46,609:3927(0xb78546c0):ZOO_INFO@log_env@671: Client environment:os.version=#42-Ubuntu SMP Fri Aug 20 14:24:04 UTC 2010
2010-09-17 16:11:46,609:3927(0xb78546c0):ZOO_INFO@log_env@679: Client environment:user.name=(null)
2010-09-17 16:11:46,609:3927(0xb78546c0):ZOO_INFO@log_env@687: Client environment:user.home=/home/kapil
2010-09-17 16:11:46,609:3927(0xb78546c0):ZOO_INFO@log_env@699: Client environment:user.dir=/home/kapil/canonical/ensemble/trunk
2010-09-17 16:11:46,609:3927(0xb78546c0):ZOO_INFO@zookeeper_init@727: Initiating client connection, host=localhost:22181 sessionTimeout=1000\
0 watcher=0xebbf50 sessionId=0 sessionPasswd=<null> context=0x9d6c7b0 flags=0
2010-09-17 16:11:46,613:3927(0xb6ea5b70):ZOO_INFO@check_events@1585: initiated connection to server [127.0.0.1:22181]
2010-09-17 16:11:53,282:3927(0xb6ea5b70):ZOO_ERROR@handle_socket_error_msg@1528: Socket [127.0.0.1:22181] zk retcode=-7, errno=110(Connectio\
n timed out): connection timed out (exceeded timeout by 3ms)
2010-09-17 16:11:53,282:3927(0xb6ea5b70):ZOO_INFO@check_events@1585: initiated connection to server [127.0.0.1:22181]
2010-09-17 16:11:59,950:3927(0xb6ea5b70):ZOO_ERROR@handle_socket_error_msg@1528: Socket [127.0.0.1:22181] zk retcode=-7, errno=110(Connectio\
n timed out): connection timed out (exceeded timeout by 1ms)
2010-09-17 16:12:03,285:3927(0xb6ea5b70):ZOO_WARN@zookeeper_interest@1461: Exceeded deadline by 3337ms
2010-09-17 16:12:03,285:3927(0xb6ea5b70):ZOO_INFO@check_events@1585: initiated connection to server [127.0.0.1:22181]
2010-09-17 16:12:05,804:3927(0xb6ea5b70):ZOO_INFO@check_events@1632: session establishment complete on server [127.0.0.1:22181], sessionId=0\
x12b211c28ae0001, negotiated timeout=10000
usage: ensemble deploy [-h] --repository REPOSITORY [--environment ENVIRONMENT] formula [service_name]
ensemble deploy: error: [Failure instance: Traceback (failure with no frames): <class 'zookeeper.No...

[Gustavo Niemeyer (niemeyer)]

Here are a few other error variants, out of some sequential attempts at deploying the same thing:

[niemeyer@gopher ..ay-sprint-hacks]% ensemble deploy --repository ensemble/formula/tests/repository dummy
usage: ensemble deploy [-h] --repository REPOSITORY
                       [--environment ENVIRONMENT]
                       formula [service_name]
ensemble deploy: error: No Ensemble Nodes
[niemeyer@gopher ..ay-sprint-hacks]% ensemble deploy --repository ensemble/formula/tests/repository dummy
usage: ensemble deploy [-h] --repository REPOSITORY
                       [--environment ENVIRONMENT]
                       formula [service_name]
ensemble deploy: error: No Ensemble Nodes
[niemeyer@gopher ..ay-sprint-hacks]% ensemble deploy --repository ensemble/formula/tests/repository dummy
2010-09-21 14:36:23,402 ERROR ssh tunnel error 'ssh: connect to host ec2-67-202-50-8.compute-1.amazonaws.com port 22: Connection refused\r\n', closing client
usage: ensemble deploy [-h] --repository REPOSITORY
                       [--environment ENVIRONMENT]
                       formula [service_name]
ensemble deploy: error: ProviderError: Interaction with machine provider failed: ConnectionTimeoutException('could not connect',)
[niemeyer@gopher ..ay-sprint-hacks]% ensemble deploy --repository ensemble/formula/tests/repository dummy
(worked)

[Gustavo Niemeyer (niemeyer)]

The tuesday-sprint-hacks branch fixes most of these problems, besides the last one (connection succeeded, but zookeeper wasn't initialized yet). I'll handle that one in a different branch.

[Gustavo Niemeyer (niemeyer)]

The wednesday-sprint-hacks branch takes care of it.