Use ipAddress instead of dnsName now that txaws supports it

[Roman Yepishev]

lp:txaws now exposes ip_address, we should start using it instead of dns_name for ssh connection, makes private openstack users happier.

In particular, when deploying a service to a private openstack, one gets the following errors:

2012-07-23 19:17:08,804 ERROR Invalid host for SSH forwarding: ssh: Could not resolve hostname server-13898: Name or service not known
2012-07-23 19:17:36,693 ERROR Invalid host for SSH forwarding: ssh: Could not resolve hostname server-13898: Name or service not known
...

This repeats many times slowly at roughly 30s intervals. On the 6th time I hit control-c.

[Robert Collins (lifeless)]

rev 134 of txaws has this, which isn't in precise. We need to backport it, or backport the whole last release. http://bazaar.launchpad.net/~txaws-dev/txaws/trunk/revision/134

[Robert Collins (lifeless)]

The attached branch addresses this. We don't need to fiddle with private-dns, because inside openstack private-dns does work, its just public that is fraught.

[Gustavo Niemeyer (niemeyer)]

I'm -1 on changing this as suggested. If a DNS name is available, it should be provided instead of the IP address, and we shouldn't be changing such an API without better reasoning. What's breaking at the moment? Do we have cases where the DNS name does not exist? If so, we could fallback to the IP adddress.

[Patrick Hetu (patrick-hetu)]

This is the kind of bug that having access to an IP address would solve more cleanly:

  https://bugs.launchpad.net/charms/+source/postgresql/+bug/902672

I suspect that this kind of hack will be needed often in charms without having access to an IP address.

[Gustavo Niemeyer (niemeyer)]

There's nothing wrong with resolving the domain name so that you get the ip address when you need it.

[Gustavo Niemeyer (niemeyer)]

Rob Collins provider further information about the problem in the merge proposal:

You can talk to me, you can talk to everyone using canonistack. I'd be
delighted to step you through whats going on, including exactly what
private openstack instances hand out and what the challenges are. I
will note that Martin Packman has used IP in the openstack provider
*for exactly this reason*. So if you are -1 on this proposal because
of the way it uses DNS, it would be consistent to be -1 on the
openstack provider for the same reason.

This is whats going on - I thought I detailed it elsewhere already,
but apparently not :)

Openstack hands out DNS entries of the form 'Server-N' (see Ted
Gould's post to canonical-tech, for instance). for *both* its public
and private dns entries.

Thats not resolvable at all, unless you manually expose dnsmasq on
your nova-network controller *and* add that as your primary DNS
server. Which won't work at all when dealing with multi-region
openstack.

It is extensible by writing a custom 'public dns api provider' for
nova, which brokers DNS names, either setting them to be like the ec2
ones (which embed the IP address in them so that amazon can bulk-load
their entire zone at once for a region), or letting you do arbitrary
custom funky stuff like using an LDAP backed DNS server. There is no
sensible-default shipped, or (that I know of) publically available.
Certainly a helper for bind to do amazon's trick would be useful for
folk, but not very useful for people that throw maas at 5 machines
under their desk and bring up openstack on top of that.

The private dns name isn't extensible at all today. And its also just
'server-N' not 'server-N.$customzone'. euca-describe-instances for
canonistack, for instance:
INSTANCE i-00000080 ami-000000bf server-128
server-128 running 0 m1.small
2012-06-27T03:09:18.000Z nova
monitoring-disabled 10.55.63.14 10.55.63.14
instance-store

Note that the only way to tell that this is canonistack2 vs
canonistack1 is the ip address.

This gets worse though (and my patch doesn't help with the next bit -
though its on my list (to let juju ssh $unit work for private
openstack clouds): The machines themselves report hostnames that are
only resolvable within the cloud itself without special configuration:
'server-128.canonistack2' (hostname == server128', but unless you have
specially configured DNS (or for Canonistack users, ~/.ssh.config),
thats not resolvable at all.

Using IP addresses would provide a single point to configure to let
this work, and work with vpns, ssh etc with equal ease. I don't think
the same can be said for dns.

-Rob

[Gustavo Niemeyer (niemeyer)]

I understand we have an issue, and I apologize for perhaps increasing your frustration around this issue a bit, but if OpenStack has an implementation that hands off information that doesn't work, we have a problem with OpenStack, that we can either fix in OpenStack, or that we can workaround in Juju, and I'm very happy to discuss possible workarounds we can have to relieve the end user from any frustration around this problem.

That said, Juju is not just OpenStack. The right thing for this parameter is to be a DNS name. DNS is resolvable to whatever the infrastructure decides it to resolve to. It can be an IPv4 address, it can be an IPv6 address, and it can also be a different address depending on whether you are within the network of the provider or outside of it.

That's not a theoretical problem. Amazon does resolve today differently if you are inside the network, so that packets are routed more efficiently. We shouldn't break just that because OpenStack and Canonistack are offering garbage through their API.

As a side note, I wasn't part of the recent addition of the OpenStack provider code to the Python implementation.

[Clint Byrum (clint-fewbar)]

Perhaps we can make the code pragmatic and work around the issue when it is obvious that we have run into it:

# In pseudo-python
try:
    address = gethostbyname(api_response['publicHostname'])
except DnsError:
    address = api_response['publicIpAddress']

This works in all cases, as it takes advantage of Amazon's split DNS views for optimal addresses, and when the hostname is not resolvable, falls back to whatever the API told us the "most accessible" address was.

Is there any reason not to be more robust like this?

[Kapil Thangavelu (hazmat)]

why not keep it as is and do provider specific behavior, ie ec2 we return
dns names, in openstack use ips and deprecate use of ec2 api for openstack.

-k

On Wed, Jun 27, 2012 at 10:13 AM, Clint Byrum <email address hidden> wrote:

> Perhaps we can make the code pragmatic and work around the issue when
> it is obvious that we have run into it:
>
> # In pseudo-python
> try:
> address = gethostbyname(api_response['publicHostname'])
> except DnsError:
> address = api_response['publicIpAddress']
>
> This works in all cases, as it takes advantage of Amazon's split DNS
> views for optimal addresses, and when the hostname is not resolvable,
> falls back to whatever the API told us the "most accessible" address
> was.
>
> Is there any reason not to be more robust like this?
>
> --
> You received this bug notification because you are subscribed to juju.
> https://bugs.launchpad.net/bugs/945505
>
> Title:
> Use ipAddress instead of dnsName now that txaws supports it
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/945505/+subscriptions
>

[Gustavo Niemeyer (niemeyer)]

Clint's suggestion sounds great. Doing provider-specific is not enough because one can use OpenStack via the ec2 provider.

[Eric Williams (eric-canonical)]

This issue crops up if using nova-network multi_host mode, multiple nova-network instances, and multiple nova-compute nodes.

Instances can only resolve instance hostnames on the same compute node, as dnsmasq is local to that node.

Running with a single nova-network service in your cloud doesn't seem to reveal this issue.

Eric

[Thomas Leonard (tal-it-innovation)]

So is this a reasonable work-around for now?

=== modified file 'juju/providers/ec2/machine.py'
--- juju/providers/ec2/machine.py 2011-10-11 10:59:04 +0000
+++ juju/providers/ec2/machine.py 2012-11-08 15:38:17 +0000
@@ -17,6 +17,6 @@
     """
     return EC2ProviderMachine(
         instance.instance_id,
- instance.dns_name,
- instance.private_dns_name,
+ instance.ip_address,
+ instance.private_ip_address,
         instance.instance_state)

[Martin Packman (gz)]

Thomas: yes, or use the openstack provider rather than the ec2 one when running against openstack which effectively does exactly that.

[Thomas Leonard (tal-it-innovation)]

OK, so an update on this:

Using "type: openstack" or "type: openstack_s3" didn't work initially. Juju status would just hang without displaying anything after bootstrap.

On the VM, I saw that the cloud init script failed with:

juju-admin: error: unrecognized arguments: <head> <title>401 Unauthorized</title> </head> <body> <h1>401 Unauthorized</h1> This server could not verify that you are authorized to access the document you requested. Either you supplied the wrong credentials (e.g., bad password), or your browser does not understand how to supply the credentials required.<br /><br /> Authentication required </body> </html>

The problem seems to be a lack of quoting in the curl shell command:

juju-admin initialize --instance-id=$(curl http://...:8080/juju-openstack-experimedia-95ec-8c2083e67721/juju_master_id?Signature=...HkA%3D&Expires=1668074520&AWSAccessKeyId=...d26b) ...

Adding quotes in juju/providers/openstack/launch.py fixed it for me:

            cloud_init.set_instance_id_accessor("$(%s '%s')" % (
                curl, filestorage.get_url(id_name),))

[Thomas Leonard (tal-it-innovation)]

Note: the above fix is only for "openstack_s3" (which now works).
The "openstack" provider already adds the quotes (but still gives Unauthorized).

[Martin Packman (gz)]

Thomas: can you file that url quoting issues as a new bug please? It's worth addressing seperately.

[Kapil Thangavelu (hazmat)]

fwiw openstack providres use ip addresses, and ec2 continues to use dns name, which is likely the further resolution this will see. nova multi host without interhost vm access is imo broken away for real usage. thomas comments point to a separate openstack swift/s3 defiency wrt to authenticated anon access (tempurl middleware must be deployed, and is not discoverable via extensions mechanism afaik).