Bootstrapping to Openstack Environment fails with "no instances found"

[Sean Feole]

I'm trying to Bootstrap to my Openstack Juno Enviornment and running into failures with the following messages:

Logging to /var/log/cloud-init-output.log on remote host
Running apt-get update
Running apt-get upgrade
Installing package: curl
Installing package: cpu-checker
Installing package: bridge-utils
Installing package: rsyslog-gnutls
Bootstrapping Juju machine agent

2015-01-22 17:38:32 INFO juju.cmd supercommand.go:37 running jujud [1.21-rc1.1-trusty-amd64 gc]
2015-01-22 17:38:32 DEBUG juju.agent agent.go:400 read agent config, format "1.18"
2015-01-22 17:38:32 INFO juju.network network.go:106 setting prefer-ipv6 to false
2015-01-22 17:38:32 INFO juju.provider.openstack provider.go:248 opening environment "openstack"
2015-01-22 17:39:32 ERROR juju.cmd supercommand.go:323 no instances found
2015-01-22 17:39:31 ERROR juju.cmd supercommand.go:323 failed to bootstrap environment: subprocess encountered error code 1

Full output available here: http://pastebin.ubuntu.com/9824751/

Output of my enviornments.yaml file: http://pastebin.ubuntu.com/9824787/

I have generated my own image metadata using: juju metadata generate-image -i <image-id> -s trusty
and launch the bootstrap command with: juju bootstrap --debug --metadata-source /home/sfeole/.juju/environments
I have tried removing and letting juju recreate its own secgroups as well as, forcing the instance to use the default secgroups both changes did not appear to have an affect on the error.

Note that after the failure the instance does fail to shutdown completely, so I am able to triage logs from this if required:

The /var/log/juju directory is completely empty on the vm after the failure and there does not appear to be anything interesting in the syslog on the juju-openstack-machine-0 node: http://pastebin.ubuntu.com/9824892/

cloud-init-output.log: http://pastebin.ubuntu.com/9825001/

sfeole@sfmadmax:~/.juju$ nova list
+--------------------------------------+--------------------------+--------+------------+-------------+--------------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+--------------------------+--------+------------+-------------+--------------------------------------+
| e2abffb7-f079-4d6f-a4d5-3f1068e1d2cc | juju-openstack-machine-0 | ACTIVE | - | Running | admin_net=192.168.1.40, 10.229.0.153 |
+--------------------------------------+--------------------------+--------+------------+-------------+--------------------------------------+

I can supply more information about openstack env if required. But it's pretty close to the following bundle: https://jujucharms.com/u/openstack-charmers/openstack/

 with added swift-proxy and swift storage relations.

I have tried both juju stable 1.20 and devel 1.21-rc1

[Curtis Hovey (sinzui)]

Comparing the failed cloud-init log to a successful one in ci,

INFO juju.provider.openstack provider.go:249 opening environment "canonistack-deploy-trusty-amd64"
INFO juju.agent identity.go:22 writing system identity file
juju.cmd.jujud bootstrap.go:240 starting mongo
juju.cmd.jujud bootstrap.go:265 calling ensureMongoServer

I believe juju tried to open a file named after the env to learn what it needed to do. The file wasn't there or the information was incomplete so the bootstrap was aborted. I think this file is /var/lib/juju/agents/machine-0/agent.conf. We see the format is reported so some or all the file was read.

Maybe /var/lib/cloud/instances was empty so there was no instructions to setup a machine.

[Curtis Hovey (sinzui)]

Hi Tim, Can you or someone one your team triage this issue.

[Tim Penhey (thumper)]

The command that is failing is 'jujud bootstrap' which is run over ssh from the client machine.

The openstack provider is obviously creating the machine, which is then ssh'ed into and the bootstrap command run. One of the first things this command does is try to get the instance id and addresses for the machine.
    cmd/jujud/bootstrap.go:126

It appears that the instance id is incorrect, but we don't have enough logging here. I have filed bug 1414785 as this is severely hampering any debugging here.

[Sean Feole (sfeole)]

Hey Curtis / Tim, Can I do anything else to help? Or are we soley waiting on bug 1414785 ?

[Sean Feole (sfeole)]

I can try again and double check the following files:
 /var/lib/juju/agents/machine-0/agent.conf. to validate the sanity of this file, as it was read
and

 /var/lib/cloud/instances : is this in fact empty or does it contain valuable info

[Curtis Hovey (sinzui)]

Hi Sean, please try again to learn those values.

You may want to add --keep-broken to the bootstrap command to give yourself time to do an autopsy., or you can invite an engineer to visit the machine and poke around.

[Sean Feole (sfeole)]

Well today I stood up the openstack env again to try and debug this issue.

It works now, I can't seem to reproduce this :(

In a way I'm happy however if it happens again i'll refer to the notes from Tim and will notify you guys.

http://pastebin.ubuntu.com/9947070/

Lets close this bug out