deployer doesn't always honor global deployment timeout

[Paul Czarkowski]

deploying the following never finishes.

https://github.com/paulczar/charm-championship/blob/master/monitoringstack.yaml

```
juju-deployer -v -W -c monitoringstack.yaml monitoringstack
2013-10-18 10:59:03 Using runtime GoEnvironment
2013-10-18 10:59:03 Starting deployment of monitoringstack
2013-10-18 10:59:07 Connected to environment
2013-10-18 10:59:07 Getting charms...
2013-10-18 10:59:07 Cache dir /home/pczarkowski/.juju/.deployer-store-cache/cs_~paulcz_precise_sensu-agent-1
2013-10-18 10:59:07 Cache dir /home/pczarkowski/.juju/.deployer-store-cache/cs_~paulcz_precise_kibana-7
2013-10-18 10:59:07 Cache dir /home/pczarkowski/.juju/.deployer-store-cache/cs_~paulcz_precise_logstash-indexer-8
2013-10-18 10:59:07 Cache dir /home/pczarkowski/.juju/.deployer-store-cache/cs_~paulcz_precise_graphite-1
2013-10-18 10:59:07 Cache dir /home/pczarkowski/.juju/.deployer-store-cache/cs_precise_rabbitmq-server-14
2013-10-18 10:59:07 Cache dir /home/pczarkowski/.juju/.deployer-store-cache/cs_~paulcz_precise_elasticsearch-4
2013-10-18 10:59:07 Cache dir /home/pczarkowski/.juju/.deployer-store-cache/cs_precise_juju-gui-78
2013-10-18 10:59:07 Retrieving store charm cs:precise/juju-gui-78
2013-10-18 10:59:13 Cache dir /home/pczarkowski/.juju/.deployer-store-cache/cs_~paulcz_precise_logstash-agent-5
2013-10-18 10:59:13 Retrieving store charm cs:~paulcz/precise/logstash-agent-5
2013-10-18 10:59:13 Cache dir /home/pczarkowski/.juju/.deployer-store-cache/cs_~paulcz_precise_sensu-server-1
2013-10-18 10:59:13 Retrieving store charm cs:~paulcz/precise/sensu-server-1
2013-10-18 10:59:14 Resolving configuration
2013-10-18 10:59:14 Deploying services...
2013-10-18 10:59:15 Deploying service sensu-agent using cs:~paulcz/precise/sensu-agent-1
2013-10-18 10:59:21 Deploying service kibana using cs:~paulcz/precise/kibana-7
2013-10-18 10:59:32 Deploying service logstash-indexer using cs:~paulcz/precise/logstash-indexer-8
2013-10-18 10:59:43 Deploying service graphite using cs:~paulcz/precise/graphite-1
2013-10-18 10:59:55 Deploying service rabbit using cs:precise/rabbitmq-server-14
2013-10-18 11:00:07 Deploying service elasticsearch using cs:~paulcz/precise/elasticsearch-4
2013-10-18 11:00:18 Deploying service juju-gui using cs:precise/juju-gui-78
2013-10-18 11:00:53 Deploying service logstash-agent using cs:~paulcz/precise/logstash-agent-5
2013-10-18 11:00:59 Deploying service sensu-server using cs:~paulcz/precise/sensu-server-1
2013-10-18 11:03:30 Delta machine: 2 change:started
2013-10-18 11:03:31 Delta unit: logstash-indexer/0 change:pending
2013-10-18 11:04:15 Delta machine: 1 change:started
2013-10-18 11:04:20 Delta unit: kibana/0 change:pending
2013-10-18 11:04:21 Delta unit: logstash-indexer/0 change:started
2013-10-18 11:04:31 Delta unit: kibana/0 change:pending
2013-10-18 11:04:31 Delta unit: kibana/0 change:started
2013-10-18 11:04:51 Delta machine: 3 change:started
2013-10-18 11:04:51 Delta unit: graphite/0 change:pending
2013-10-18 11:04:56 Delta unit: graphite/0 change:pending
2013-10-18 11:05:51 Delta machine: 5 change:started
2013-10-18 11:05:51 Delta unit: elasticsearch/0 change:pending
2013-10-18 11:06:31 Delta machine: 6 change:started
2013-10-18 11:06:31 Delta unit: juju-gui/0 change:pending
2013-10-18 11:06:36 Delta unit: juju-gui/0 change:pending
2013-10-18 11:07:06 Delta unit: elasticsearch/0 change:installed
2013-10-18 11:07:11 Delta unit: elasticsearch/0 change:installed
2013-10-18 11:07:16 Delta unit: elasticsearch/0 change:started
2013-10-18 11:07:31 Delta unit: juju-gui/0 change:installed
2013-10-18 11:07:36 Delta unit: juju-gui/0 change:installed
2013-10-18 11:07:36 Delta unit: juju-gui/0 change:started
2013-10-18 11:08:11 Delta unit: graphite/0 change:installed
2013-10-18 11:08:11 Delta unit: graphite/0 change:installed
2013-10-18 11:08:16 Delta unit: graphite/0 change:started
2013-10-18 11:09:16 Delta machine: 7 change:started
2013-10-18 11:09:21 Delta unit: sensu-server/0 change:pending
2013-10-18 11:13:36 Delta unit: sensu-server/0 change:pending
2013-10-18 11:13:36 Delta unit: sensu-server/0 change:started
```

and then it just sits there.... current time writing this is ` date
Fri Oct 18 11:40:34 CDT 2013` and the deploy hasn't progressed since 11:13.

[Paul Czarkowski (paulcz)]

juju status
environment: amazon
machines:
  "0":
    agent-state: started
    agent-version: 1.16.0
    dns-name: ec2-54-224-88-249.compute-1.amazonaws.com
    instance-id: i-67196f1f
    instance-state: running
    series: precise
    hardware: arch=amd64 cpu-cores=1 cpu-power=100 mem=1740M root-disk=8192M
  "1":
    agent-state: started
    agent-version: 1.16.0
    dns-name: ec2-54-205-165-46.compute-1.amazonaws.com
    instance-id: i-d2c521aa
    instance-state: running
    series: precise
    hardware: arch=amd64 cpu-cores=1 cpu-power=100 mem=1740M root-disk=8192M
  "2":
    agent-state: started
    agent-version: 1.16.0
    dns-name: ec2-54-224-84-112.compute-1.amazonaws.com
    instance-id: i-2d8d4657
    instance-state: running
    series: precise
    hardware: arch=amd64 cpu-cores=1 cpu-power=100 mem=1740M root-disk=8192M
  "3":
    agent-state: started
    agent-version: 1.16.0
    dns-name: ec2-50-17-68-0.compute-1.amazonaws.com
    instance-id: i-723d9a15
    instance-state: running
    series: precise
    hardware: arch=amd64 cpu-cores=1 cpu-power=100 mem=1740M root-disk=8192M
  "4":
    agent-state-info: '(error: invalid URL "http://cloud-images.ubuntu.com/releases/streams/v1/index.sjson"
      not found)'
    instance-id: pending
    series: precise
  "5":
    agent-state: started
    agent-version: 1.16.0
    dns-name: ec2-54-205-194-197.compute-1.amazonaws.com
    instance-id: i-4ec52136
    instance-state: running
    series: precise
    hardware: arch=amd64 cpu-cores=1 cpu-power=100 mem=1740M root-disk=8192M
  "6":
    agent-state: started
    agent-version: 1.16.0
    dns-name: ec2-54-234-215-130.compute-1.amazonaws.com
    instance-id: i-ca2591ac
    instance-state: running
    series: precise
    hardware: arch=amd64 cpu-cores=1 cpu-power=100 mem=1740M root-disk=8192M
  "7":
    agent-state: started
    agent-version: 1.16.0
    dns-name: ec2-54-205-113-224.compute-1.amazonaws.com
    instance-id: i-0e3e9969
    instance-state: running
    series: precise
    hardware: arch=amd64 cpu-cores=1 cpu-power=100 mem=1740M root-disk=8192M
services:
  elasticsearch:
    charm: cs:~paulcz/precise/elasticsearch-4
    exposed: false
    units:
      elasticsearch/0:
        agent-state: started
        agent-version: 1.16.0
        machine: "5"
        open-ports:
        - 9200/tcp
        - 9300/tcp
        public-address: ec2-54-205-194-197.compute-1.amazonaws.com
  graphite:
    charm: cs:~paulcz/precise/graphite-1
    exposed: false
    units:
      graphite/0:
        agent-state: started
        agent-version: 1.16.0
        machine: "3"
        open-ports:
        - 80/tcp
        public-address: ec2-50-17-68-0.compute-1.amazonaws.com
  juju-gui:
    charm: cs:precise/juju-gui-78
    exposed: false
    units:
      juju-gui/0:
        agent-state: started
        agent-version: 1.16.0
        machine: "6"
        open-ports:
        - 80/tcp
        - 443/tcp
        public-address: ec2-54-234-215-130.compute-1.amazonaws.com
  kibana:
    charm: cs:~paulcz/precise/kibana-7
    exposed: false
    units:
      kibana/0:
        agent-state: started
        agent-version: 1.16.0
 ...

[Paul Czarkowski (paulcz)]

juju-deployer finally finishes ... but all services still not brought up. From where it left off above ....

2013-10-18 11:09:21 Delta unit: sensu-server/0 change:pending
2013-10-18 11:13:36 Delta unit: sensu-server/0 change:pending
2013-10-18 11:13:36 Delta unit: sensu-server/0 change:started

2013-10-18 11:56:24 Adding units...
2013-10-18 11:56:25 Config specifies num units for subordinate: sensu-agent
2013-10-18 11:56:25 Service 'kibana' does not need any more units added.
2013-10-18 11:56:25 Service 'logstash-indexer' does not need any more units added.
2013-10-18 11:56:25 Service 'graphite' does not need any more units added.
2013-10-18 11:56:25 Service 'rabbit' does not need any more units added.
2013-10-18 11:56:25 Service 'elasticsearch' does not need any more units added.
2013-10-18 11:56:25 Service 'juju-gui' does not need any more units added.
2013-10-18 11:56:25 Config specifies num units for subordinate: logstash-agent
2013-10-18 11:56:25 Service 'sensu-server' does not need any more units added.
2013-10-18 11:56:25 Adding relations...
2013-10-18 11:56:25 Adding relation elasticsearch:cluster <-> logstash-indexer:cluster
2013-10-18 11:56:25 Waiting 5s before next relation
2013-10-18 11:56:30 Adding relation logstash-agent:juju-info <-> rabbit:juju-info
2013-10-18 11:56:30 Waiting 5s before next relation
2013-10-18 11:56:35 Adding relation logstash-agent:juju-info <-> juju-gui:juju-info
2013-10-18 11:56:36 Waiting 5s before next relation
2013-10-18 11:56:41 Adding relation logstash-agent:juju-info <-> sensu-server:juju-info
2013-10-18 11:56:41 Waiting 5s before next relation
2013-10-18 11:56:46 Adding relation logstash-agent:juju-info <-> graphite:juju-info
2013-10-18 11:56:46 Waiting 5s before next relation
2013-10-18 11:56:51 Adding relation logstash-agent:juju-info <-> kibana:juju-info
2013-10-18 11:56:51 Waiting 5s before next relation
2013-10-18 11:56:56 Adding relation logstash-agent:juju-info <-> elasticsearch:juju-info
2013-10-18 11:56:56 Waiting 5s before next relation
2013-10-18 11:57:01 Adding relation sensu-agent:juju-info <-> logstash-indexer:juju-info
2013-10-18 11:57:01 Waiting 5s before next relation
2013-10-18 11:57:06 Adding relation sensu-agent:juju-info <-> elasticsearch:juju-info
2013-10-18 11:57:06 Waiting 5s before next relation
2013-10-18 11:57:11 Adding relation sensu-agent:juju-info <-> juju-gui:juju-info
2013-10-18 11:57:11 Waiting 5s before next relation
2013-10-18 11:57:16 Adding relation sensu-server:amqp <-> rabbit:amqp
2013-10-18 11:57:16 Waiting 5s before next relation
2013-10-18 11:57:21 Adding relation graphite:amqp <-> rabbit:amqp
2013-10-18 11:57:21 Waiting 5s before next relation
2013-10-18 11:57:26 Adding relation logstash-indexer:amqp <-> rabbit:amqp
2013-10-18 11:57:26 Waiting 5s before next relation
2013-10-18 11:57:31 Adding relation logstash-agent:amqp <-> rabbit:amqp
2013-10-18 11:57:31 Waiting 5s before next relation
2013-10-18 11:57:36 Adding relation sensu-agent:juju-info <-> rabbit:juju-info
2013-10-18 11:57:36 Waiting 5s before next relation
2013-10-18 11:57:41 Adding relation sensu-agent:amqp <-> rabbit:amqp
2013-10-18 11:57:41 Waitin...

[Martin Packman (gz)]

 "4":
     agent-state-info: '(error: invalid URL "http://cloud-images.ubuntu.com/releases/streams/v1/index.sjson"
       not found)'
     instance-id: pending
     series: precise

That url resolves for me, but it's probably spurious. I suspect machine-4 just failed to provision due to EC2 instability, deleting it and retrying the rabbit deploy may be enough. Unfortunately the debug-log doesn't contain anything incriminating about the actual start instance failure.

[Kapil Thangavelu (hazmat)]

afaics this isn't a deployer issue but a juju-core one. closing.

[Kapil Thangavelu (hazmat)]

well technically it could be a deployment timeout issue.