Guest agent fails if it can't resolve it's own hostname

[Sam Morrison]

When using heat to build instances heat will create an instance in nova with some large random hostname.

After taskmanager creates the instance then it creates a record in designate, this hostname is different that the hostname assigned to the instance.

Trove guest agent will fail with:

2014-03-28 03:16:11.079 2590 INFO trove.guestagent.datastore.mysql.service [req-9bf949a2-dacf-44e7-a5b7-e6ee8b3f7dc7 d9646718471b46aeb5fd94c702336ca9 0bdf024c921848c4b74d9e69af9edf08 - - -] Dbaas secure complete.
2014-03-28 03:16:11.080 2590 DEBUG trove.openstack.common.processutils [req-9bf949a2-dacf-44e7-a5b7-e6ee8b3f7dc7 d9646718471b46aeb5fd94c702336ca9 0bdf024c921848c4b74d9e69af9edf08 - - -] Running cmd (subprocess): sudo awk /password\t=/{print $3; exit} /etc/mysql/my.cnf execute /usr/local/lib/python2.7/dist-packages/trove/openstack/common/processutils.py:137
2014-03-28 03:16:11.111 2590 ERROR trove.guestagent.datastore.mysql.service [req-9bf949a2-dacf-44e7-a5b7-e6ee8b3f7dc7 d9646718471b46aeb5fd94c702336ca9 0bdf024c921848c4b74d9e69af9edf08 - - -] sudo: unable to resolve host tr-8-4f75-ac74-46fca512f4b7-baseinstance-hub53wtgiftf

2014-03-28 03:16:11.112 2590 ERROR trove.openstack.common.rpc.amqp [req-9bf949a2-dacf-44e7-a5b7-e6ee8b3f7dc7 d9646718471b46aeb5fd94c702336ca9 0bdf024c921848c4b74d9e69af9edf08 - - -] Exception during message handling
2014-03-28 03:16:11.112 2590 TRACE trove.openstack.common.rpc.amqp Traceback (most recent call last):
2014-03-28 03:16:11.112 2590 TRACE trove.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/trove/openstack/common/rpc/amqp.py", line 440, in _process_data
2014-03-28 03:16:11.112 2590 TRACE trove.openstack.common.rpc.amqp **args)
2014-03-28 03:16:11.112 2590 TRACE trove.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/trove/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2014-03-28 03:16:11.112 2590 TRACE trove.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2014-03-28 03:16:11.112 2590 TRACE trove.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/trove/guestagent/datastore/mysql/manager.py", line 139, in prepare
2014-03-28 03:16:11.112 2590 TRACE trove.openstack.common.rpc.amqp app.secure_root(secure_remote_root=True)
2014-03-28 03:16:11.112 2590 TRACE trove.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/trove/guestagent/datastore/mysql/service.py", line 620, in secure_root
2014-03-28 03:16:11.112 2590 TRACE trove.openstack.common.rpc.amqp with LocalSqlClient(get_engine()) as client:
2014-03-28 03:16:11.112 2590 TRACE trove.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/trove/guestagent/datastore/mysql/service.py", line 117, in get_engine
2014-03-28 03:16:11.112 2590 TRACE trove.openstack.common.rpc.amqp pwd = get_auth_password()
2014-03-28 03:16:11.112 2590 TRACE trove.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/trove/guestagent/datastore/mysql/service.py", line 105, in get_auth_password
2014-03-28 03:16:11.112 2590 TRACE trove.openstack.common.rpc.amqp raise RuntimeError("Problem reading my.cnf! : %s" % err)
2014-03-28 03:16:11.112 2590 TRACE trove.openstack.common.rpc.amqp RuntimeError: Problem reading my.cnf! : sudo: unable to resolve host tr-8-4f75-ac74-46fca512f4b7-baseinstance-hub53wtgiftf
2014-03-28 03:16:11.112 2590 TRACE trove.openstack.common.rpc.amqp
2014-03-28 03:16:11.112 2590 TRACE trove.openstack.common.rpc.amqp

[Sushil Kumar (sushil-kumar2)]

Hi Sam

2 things,
1. Is this problem still on.
2. it would be helpfull, if you can please provide an example name which gets registered on dns in your case

[Sam Morrison (sorrison)]

Hi Sushil,
Looking at the latest code in master this is still an issue.

The generated hostname is in the stacktrace:

unable to resolve host tr-8-4f75-ac74-46fca512f4b7-baseinstance-hub53wtgiftf

[Sushil Kumar (sushil-kumar2)]

Not the hostname in stack trace, if possible i wanted to know the name on designate

[Sam Morrison (sorrison)]

Yeah so in trove the hostname is different, it's mydzaue5atb.ourdomain

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/96552

[OpenStack Infra (hudson-openstack)]

Change abandoned by Nikhil Manchanda (<email address hidden>) on branch: master
Review: https://review.openstack.org/96552
Reason: This patch is being abandoned as part of routine cleanup (patches older than 4 weeks with a negative review).

Please feel free to restore the patch if this is something you're still actively working on.

Thanks!

[Kevin Fox (kevpn)]

I've run across this too. I think. It may be a similar bug instead. I'm using heat but not designate. The instance creates just fine. So does creating a database I think. But listing users fails.

Oddly, the node can resolve itself:
ping -c 1 `hostname`
PING tr-8-4012-b391-7d1795565f9b-baseinstance-ltfsb6b4y3gx.novalocal (127.0.0.1) 56(84) bytes of data.
64 bytes from tr-8-4012-b391-7d1795565f9b-baseinstance-ltfsb6b4y3gx.novalocal (127.0.0.1): icmp_seq=1 ttl=64 time=0.103 ms

[kfox@mantis mgmt]$ trove list
+--------------------------------------+--------+-----------+-------------------+--------+--------------------------------------+------+
| id | name | datastore | datastore_version | status | flavor_id | size |
+--------------------------------------+--------+-----------+-------------------+--------+--------------------------------------+------+
| c39eab36-1138-4012-b391-7d1795565f9b | testdb | mysql | mysql-5.5 | ACTIVE | baadf037-6304-4dde-8454-a5a833e1ae89 | 1 |
+--------------------------------------+--------+-----------+-------------------+--------+--------------------------------------+------+
[kfox@mantis mgmt]$ trove user-list c39eab36-1138-4012-b391-7d1795565f9b
ERROR: An error occurred communicating with the guest: 'tr-8-4012-b391-7d1795565f9b-baseinstance-ltfsb6b4y3gx' is not a valid hostname.
Traceback (most recent call last):

  File "/usr/lib/python2.7/site-packages/trove/openstack/common/rpc/amqp.py", line 440, in _process_data
    **args)

  File "/usr/lib/python2.7/site-packages/trove/openstack/common/rpc/dispatcher.py", line 172, in dispatch
    result = getattr(proxyobj, method)(ctxt, **kwargs)

  File "/usr/lib/python2.7/site-packages/trove/guestagent/datastore/mysql/manager.py", line 93, in list_users
    include_marker)

  File "/usr/lib/python2.7/site-packages/trove/guestagent/datastore/mysql/service.py", line 513, in list_users
    mysql_user.host = row['Host']

  File "/usr/lib/python2.7/site-packages/trove/guestagent/db/models.py", line 438, in host
    raise ValueError("'%s' is not a valid hostname." % value)

ValueError: 'tr-8-4012-b391-7d1795565f9b-baseinstance-ltfsb6b4y3gx' is not a valid hostname.
.

[Amrith Kumar (amrith)]

Retained importance at Medium, bugscrub 2014-10-23

[Rohit Jaiswal (rohit-jaiswal-3)]

I got this error in my trove-guest.log -


2015-03-06 00:20:38.341 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-03-06 00:20:38.341 TRACE oslo_messaging.rpc.dispatcher  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-03-06 00:20:38.341 TRACE oslo_messaging.rpc.dispatcher  executor_callback))
2015-03-06 00:20:38.341 TRACE oslo_messaging.rpc.dispatcher  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-03-06 00:20:38.341 TRACE oslo_messaging.rpc.dispatcher  executor_callback)
2015-03-06 00:20:38.341 TRACE oslo_messaging.rpc.dispatcher  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2015-03-06 00:20:38.341 TRACE oslo_messaging.rpc.dispatcher  result = func(ctxt, **new_args)
2015-03-06 00:20:38.341 TRACE oslo_messaging.rpc.dispatcher  File "/home/ubuntu/trove/trove/guestagent/datastore/mysql/manager.py", line 149, in prepare
2015-03-06 00:20:38.341 TRACE oslo_messaging.rpc.dispatcher  app.secure_root(secure_remote_root=True)
2015-03-06 00:20:38.341 TRACE oslo_messaging.rpc.dispatcher  File "/home/ubuntu/trove/trove/guestagent/datastore/mysql/service.py", line 633, in secure_root
2015-03-06 00:20:38.341 TRACE oslo_messaging.rpc.dispatcher  with LocalSqlClient(get_engine()) as client:
2015-03-06 00:20:38.341 TRACE oslo_messaging.rpc.dispatcher  File "/home/ubuntu/trove/trove/guestagent/datastore/mysql/service.py", line 125, in get_engine
2015-03-06 00:20:38.341 TRACE oslo_messaging.rpc.dispatcher  pwd = get_auth_password()
2015-03-06 00:20:38.341 TRACE oslo_messaging.rpc.dispatcher  File "/home/ubuntu/trove/trove/guestagent/datastore/mysql/service.py", line 113, in get_auth_password
2015-03-06 00:20:38.341 TRACE oslo_messaging.rpc.dispatcher  raise RuntimeError("Problem reading my.cnf! : %s" % err)
2015-03-06 00:20:38.341 TRACE oslo_messaging.rpc.dispatcher RuntimeError: Problem reading my.cnf! : sudo: unable to resolve host ubuntu
2015-03-06 00:20:38.341 TRACE oslo_messaging.rpc.dispatcher 
2015

[Launchpad Janitor (janitor)]

[Expired for OpenStack DBaaS (Trove) because there has been no activity for 60 days.]