juju 1.17.0 bootstrap on Hp fails

[Curtis Hovey]

Hp fails to bootstrap. We see it run forever trying to connect to the newly started machine over ssh:

Waiting for DNS name.
 - 10.2.82.113
Attempting to connect to 10.2.82.113:22...............................................................................................................................Build step 'Execute shell' marked build as failure
Description set: lp:juju-core r2122

See http://162.213.35.54:8080/job/hp-upgrade-and-deploy/85/console

Note that this is not an issue for 1.16.5 which was tested hours earlier.

While CI was waiting, I installed the same deb and ran juju bootstrap using the same tools-url to use the matching agent. I experienced the same issue. Looking at the Hp console, I can see cloud-init and ssh came up fine and came up quickly. I could not ssh in another terminal, and it is not surprising given that the IP address is private (10.2.83.17). I could ssh into the public address. I ran "nova list". I used the same creds as CI, we can see CI's and my own machine listed. The private address is clearly marked, but mgz suspects the rules to match the public address failed.

$ nova list
+---------+--------------------------------+--------+------------+-------------+-------------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+---------+--------------------------------+--------+------------+-------------+-------------------------------------+
| 2631867 | juju-test-hp-machine-0 | ACTIVE | N/A | N/A | private=10.2.83.17, 15.185.241.251 |
| 2631947 | juju-test-release-hp-machine-0 | ACTIVE | N/A | N/A | private=10.3.96.219, 15.185.250.248 |
+---------+--------------------------------+--------+------------+-------------+-------------------------------------+

[Aaron Bentley (abentley)]

This is related to bug #1257427, but that bug was focused on the fact that hp waited forever.

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2013-12-05 21:53, Curtis Hovey wrote:
> Public bug reported:
>
> Hp fails to bootstrap. We see it run forever trying to connect to
> the newly started machine over ssh:
>
> Waiting for DNS name. - 10.2.82.113 Attempting to connect to
> 10.2.82.113:22...............................................................................................................................Build
> step 'Execute shell' marked build as failure Description set:
> lp:juju-core r2122
>
> See http://162.213.35.54:8080/job/hp-upgrade-and-deploy/85/console
>
> Note that this is not an issue for 1.16.5 which was tested hours
> earlier.

IIRC, the issue is that as the machine is coming up, it first only
gets a private IP address, we see that and assume that is what we want
to connect to.

And then a second later it gets the public IP address, but we never go
back and ask for the DNSName again.

The current loop spins until it gets a DNSName and then spins until it
can connect there.

We should change it to spin over the whole process, it can just
shortcut the loop back to the beginning when the DNSName doesn't even
return correct data yet.

I think this is pretty straightforward how to implement it, and should
be done soon. I'll see who I can schedule it to.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.13 (Cygwin)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlKlprcACgkQJdeBCYSNAANYYgCgjZGUs0GuxhYBm33M9MkZgTXf
mfEAoKGRaqys9UBxyLKk24tXfwyvHTEu
=MAHt
-----END PGP SIGNATURE-----

[Andrew Wilkins (axwalk)]

I don't have an HP Cloud account, but I've verified that bootstrap continues to work for other providers, and verified the new behaviour with a simulated address change in a unit test. I'll carry on with this, and we can see what CI says.