RABBITMQ_HOSTNAME configuration borked by revno 100 of the charm

[Daniel Manrique]

This was reported here: https://bugs.launchpad.net/capomastro/+bug/1486615

The spec in question tracks trunk so when revno 100 landed, runs with that spec broke.

Rabbitmq can't start because it has a bogus RABBITMQ_NODENAME:

2015-08-20 01:55:53 INFO config-changed - unable to connect to epmd on ps45-10-25-62-20: nxdomain (non-existing domain)

This is done earlier during the process via apparently a juju set:

2015-08-20 01:55:46 INFO juju-log local nodename: ps45-10-25-62-20
2015-08-20 01:55:46 INFO juju-log configuring nodename
2015-08-20 01:55:46 INFO juju-log forcing nodename=ps45-10-25-62-20
2015-08-20 01:55:46 INFO juju-log Stopping rabbitmq-server.
2015-08-20 01:55:46 INFO config-changed * Stopping message broker rabbitmq-server
2015-08-20 01:55:49 INFO config-changed ...done.
2015-08-20 01:55:49 INFO juju-log Updating /etc/rabbitmq/rabbitmq-env.conf, RABBITMQ_NODENAME=rabbit@ps45-10-25-62-20
2015-08-20 01:55:49 INFO juju-log Starting rabbitmq-server.
2015-08-20 01:55:49 INFO config-changed * Restarting message broker rabbitmq-server
2015-08-20 01:55:51 INFO config-changed ...fail!

Note that initially, the value of RABBITMQ_NODENAME is something like rabbit@$HOSTNAME, and $HOSTNAME Usually resolves correctly.

This looks like some evil dance between prodstack host naming and the rabbitmq charm.

See in the log how it does this:

2015-08-20 01:55:46 INFO juju-log local nodename: ps45-10-25-62-20
2015-08-20 01:55:46 INFO juju-log configuring nodename
2015-08-20 01:55:46 INFO juju-log forcing nodename=ps45-10-25-62-20
2015-08-20 01:55:46 INFO juju-log Stopping rabbitmq-server.
2015-08-20 01:55:46 INFO config-changed * Stopping message broker rabbitmq-server
2015-08-20 01:55:49 INFO config-changed ...done.
2015-08-20 01:55:49 INFO juju-log Updating /etc/rabbitmq/rabbitmq-env.conf, RABBITMQ_NODENAME=rabbit@ps45-10-25-62-20
2015-08-20 01:55:49 INFO juju-log Starting rabbitmq-server.
2015-08-20 01:55:49 INFO config-changed * Restarting message broker rabbitmq-server
2015-08-20 01:55:51 INFO config-changed ...fail!

The first three lines come from the rabbitmq charm's configure_nodename method:

def configure_nodename():
    '''Set RABBITMQ_NODENAME to something that's resolvable by my peers'''
    nodename = get_local_nodename()
    log('configuring nodename', level=INFO)
    if (nodename and
            rabbit.get_node_name() != 'rabbit@%s' % nodename):
        log('forcing nodename=%s' % nodename, level=INFO)
        # would like to have used the restart_on_change decorator, but
        # need to stop it under current nodename prior to updating env
        log('Stopping rabbitmq-server.')
        service_stop('rabbitmq-server')
        rabbit.update_rmq_env_conf(hostname='rabbit@%s' % nodename,
                                   ipv6=config('prefer-ipv6'))
        log('Starting rabbitmq-server.')
        service_restart('rabbitmq-server')

get_local_nodename has this:
def get_local_nodename():
    '''Resolve local nodename into something that's universally addressable'''
    ip_addr = get_host_ip(unit_get('private-address'))
    log('getting local nodename for ip address: %s' % ip_addr, level=INFO)
    try:
        nodename = get_hostname(ip_addr, fqdn=False)
    except:
        log('Cannot resolve hostname for %s using DNS servers' % ip_addr,
            level='WARNING')
        log('Falling back to use socket.gethostname()',
            level='WARNING')
        # If the private-address is not resolvable using DNS
        # then use the current hostname
        nodename = socket.gethostname()
    log('local nodename: %s' % nodename, level=INFO)
    return nodename

I'm not sure how to fix this: maybe a reverse check ensuring that the nodename resolves back to ip_addr, and falling back to $SOMETHING if it doesn't.

[Paul Collins (pjdc)]

The second MP-less branch is vs. 15.07 and contains the same changes.

[Nick Moffitt (nick-moffitt)]

In particular, the output of the `hostname` command in prodstack usually never resolves, and has never done to my memory. If you want DNS, you associate records to floating IPs. All connections between units tends to be by IP address.