memcached_servers doesn't use proper IPv6 bracketing in nova.conf without pacemaker

[Attila Darazs]

If you deploy without pacemaker using IPv6 network isolation, the result will be something like this in /etc/nova/nova.conf on the controller:

memcached_servers=fd00:fd00:fd00:2000::13:11211

It should look like this instead:

memcached_servers=inet6:[fd00:fd00:fd00:2000::13]:11211

The incorrect configuration results in this error during the IPv6 gate run:

http://logs.openstack.org/45/289445/19/check-tripleo/gate-tripleo-ci-f22-nonha/026bed8/console.html#_2016-03-21_12_31_31_396

And the error from /var/log/nova/nova-api.log on the controller:

2016-03-21 12:22:22.562 25440 INFO nova.metadata.wsgi.server [req-f7ffd105-15bb-42cb-8ebf-8dd502aa0798 - - - - -] (25440) wsgi starting up on http://fd00:fd00:fd00:2000::13:8775/
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack [-] Caught error: Unable to parse connection string: "fd00:fd00:fd00:2000::13:11211"
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack Traceback (most recent call last):
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/api/openstack/__init__.py", line 140, in __call__
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack return req.get_response(self.application)
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/request.py", line 1317, in send
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack application, catch_exc_info=False)
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/request.py", line 1281, in call_application
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack app_iter = application(self.environ, start_response)
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack return resp(environ, start_response)
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack return self.func(req, *args, **kwargs)
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 464, in __call__
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack response = self.process_request(req)
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 732, in process_request
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack resp = super(AuthProtocol, self).process_request(request)
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 492, in process_request
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack data, user_auth_ref = self._do_fetch_token(request.user_token)
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 531, in _do_fetch_token
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack data = self.fetch_token(token)
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 835, in fetch_token
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack cached = self._cache_get_hashes(token_hashes)
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 818, in _cache_get_hashes
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack cached = self._token_cache.get(token)
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/_cache.py", line 222, in get
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack with self._cache_pool.reserve() as cache:
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack return self.gen.next()
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/_cache.py", line 77, in reserve
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack c = memorycache.get_client(self._memcached_servers)
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/keystonemiddleware/openstack/common/memorycache.py", line 47, in get_client
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack return client_cls(memcached_servers, debug=0)
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/memcache.py", line 229, in __init__
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack self.debug = debug
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/eventlet/corolocal.py", line 48, in __setattr__
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack _patch(self)
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/eventlet/corolocal.py", line 38, in _patch
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack thrl.__init__(*args, **kw)
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/memcache.py", line 233, in __init__
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack self.set_servers(servers)
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/memcache.py", line 284, in set_servers
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack for s in servers]
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/memcache.py", line 1223, in __init__
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack raise ValueError('Unable to parse connection string: "%s"' % host)
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack ValueError: Unable to parse connection string: "fd00:fd00:fd00:2000::13:11211"
2016-03-21 12:31:30.095 25401 ERROR nova.api.openstack

Full logs are available at http://logs.openstack.org/45/289445/19/check-tripleo/gate-tripleo-ci-f22-nonha/026bed8/

Probably the nova puppet module should be fixed when deploying with IPv6.

[Emilien Macchi (emilienm)]

I'm also investigating the bug, but in a first glance, I think this patch should avoid this bug: https://review.openstack.org/#/c/270110/

[Emilien Macchi (emilienm)]

After some code investigation, I think the HA scenario has the right config but not the non-ha, if you look at the patch. I'm working on it now.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/295449

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/295449
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=99c3db0ea146b55005848fd5230ca0bb63874f58
Submitter: Jenkins
Branch: master

commit 99c3db0ea146b55005848fd5230ca0bb63874f58
Author: Emilien Macchi <email address hidden>
Date: Mon Mar 21 14:42:52 2016 -0400

    nonha: fix memcached servers string in nova.conf for ipv6

    Full context is described here:
    https://review.openstack.org/#/c/270110/

    The patch that was supposed to fix [1] was not fixing non-ha scenario.
    [1] https://launchpad.net/bugs/1536103

    This patch aims to fix it.

    Change-Id: Iaf4608de1894ce03f35925939e83230abb9f5207
    Closes-Bug: #1560063

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/297066

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/297066
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=2bd0d3d18ced10af3b9f6a3706e2731ccf0c7031
Submitter: Jenkins
Branch: stable/liberty

commit 2bd0d3d18ced10af3b9f6a3706e2731ccf0c7031
Author: Emilien Macchi <email address hidden>
Date: Mon Mar 21 14:42:52 2016 -0400

    nonha: fix memcached servers string in nova.conf for ipv6

    Full context is described here:
    https://review.openstack.org/#/c/270110/

    The patch that was supposed to fix [1] was not fixing non-ha scenario.
    [1] https://launchpad.net/bugs/1536103

    This patch aims to fix it.

    Change-Id: Iaf4608de1894ce03f35925939e83230abb9f5207
    Closes-Bug: #1560063
    (cherry picked from commit 99c3db0ea146b55005848fd5230ca0bb63874f58)