mitaka ksclient fails to connect to v6 keystone

[Edward Hope-Morley]

[Impact]
urllib3 is not stripping [] from IPv6 URL's, resulting in double square brackets which breaks connectivity to targets.

[Test Case]
<relying on Ed for actual testing>
Deploy OpenStack with IPv6 preferred; keystone service is unable to self configure as extra brackets are inserted to the URL hostname resulting in failed connections.

[Regression Potential]
Change in behaviour was reverted upstream; regression potential minimal.

[Original Bug Report]
2016-05-04 15:02:13 INFO shared-db-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 449, in _send_request
2016-05-04 15:02:13 INFO shared-db-relation-changed raise exceptions.ConnectionRefused(msg)
2016-05-04 15:02:13 INFO shared-db-relation-changed keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://[2001:db8:0:1:f816:3eff:fe2d:9445]:35347/v2.0/OS-KSADM/services

Full traceback is here http://paste.ubuntu.com/16224193/

This works fine with Liberty

[Edward Hope-Morley (hopem)]

root@juju-hopem-machine-9:~# ip a s| grep 2001:db8:0:1:f816:3eff:fe2d:9445
    inet6 2001:db8:0:1:f816:3eff:fe2d:9445/64 scope global dynamic
root@juju-hopem-machine-9:~# netstat -tulpn| grep 35347
tcp6 0 0 :::35347 :::* LISTEN 25463/apache2

[Edward Hope-Morley (hopem)]

Having analysed the requests going to keystone it appears that something is adding an extra set of square brackets to the host address:

Liberty GET request has "HOST: [2001:db8:0:1:f816:3eff:fe67:6978]:35347"

Mitaka GET request has "HOST: [[2001:db8:0:1:f816:3eff:fe67:6978]]:35347"

Whether this is a charm issue or not is debatable since the correct address appears to get passed to the keystoneclient (albeit bracketed). Something further down is inferring that it has an ipv6 address and adds more brackets without first checking that they have been added. Another way of looking at this is that the newer client wants unbracketed so we should pass it as such. Either way this is a change of behaviour in the keystoneclient.

[Edward Hope-Morley (hopem)]

Looks like it's a python-urllib3 issue - https://github.com/shazow/urllib3/commit/95275687c805d8129ab4d10f6000410dd0303023

[Edward Hope-Morley (hopem)]

Not sure if ^^ is the only patch required to fix this. Will try more recent version of urllib3 (than 1.13.1) to see if it fixes things.

[Daniele Tricoli (eriol.v)]

Hello, Debian Maintainer here. I'm updating both urllib3 and requests, I hope to complete this at max tomorrow. Right now I'm finishing to test urllib3 before pushing on the git repository.

[Edward Hope-Morley (hopem)]

Ok some more info/test results. It seems both urllib3 AND the keystone charm need fixing:

1. urlib3 needed this to get working with v6 addresses - https://github.com/shazow/urllib3/commit/a3b9a34be9dccfc42df65b2403bc0c6666fd3392

2. the keystone charm does not re-run ensure_intial_admin() once this error is overcome causing subsequent juju resolves to fail since admin creds are not setup. Should be an easy fix.

[Edward Hope-Morley (hopem)]

Keystone charm patch: https://review.openstack.org/314247

[James Page (james-page)]

@Daniele - thanks for the update - I'm going to patch into yakkety so we can get moving asap on the xenial SRU - will resync with your updates in Debian as soon as they land.

[James Page (james-page)]

Uploads made for Xenial and Yakkety; Xenial upload will require SRU team review.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-urllib3 - 1.13.1-2ubuntu1

---------------
python-urllib3 (1.13.1-2ubuntu1) yakkety; urgency=medium

  * d/p/06_revert_square_brackets_httplib.patch: Cherry pick revert of
    change to behaviour with httplib which breaks IPv6 square bracket
    handling (LP: #1578351).

 -- James Page <email address hidden> Tue, 10 May 2016 09:41:48 +0100

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/314247
Committed: https://git.openstack.org/cgit/openstack/charm-keystone/commit/?id=216a5c9e39be799ca0e82e45750465539bbc891e
Submitter: Jenkins
Branch: master

commit 216a5c9e39be799ca0e82e45750465539bbc891e
Author: Edward Hope-Morley <email address hidden>
Date: Mon May 9 20:20:48 2016 +0100

    Ensure we can recover from error during db init

    Currently if a failure occurs during the shared-db
    hook intialisation sequence after db is inited but
    before admin creds have been setup, there is no way
    to re-run ensure_admin_credentials. This patch
    resolves that issue.

    Change-Id: Iad80a0eeae6f94dc89ff994f8e5794c60c272e16
    Closes-Bug: 1578351

[Martin Pitt (pitti)]

Hello Edward, or anyone else affected,

Accepted python-urllib3 into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-urllib3/1.13.1-2ubuntu0.16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[James Page (james-page)]

Hello Edward, or anyone else affected,

Accepted python-urllib3 into mitaka-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:mitaka-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-mitaka-needed to verification-mitaka-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-mitaka-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Edward Hope-Morley (hopem)]

xenial-mitaka-proposed tested and confirmed all good.

[Edward Hope-Morley (hopem)]

trusty-mitaka-proposed tested and confirmed all good.

[Brian Murray (brian-murray)]

I've changed the tag back to verification-need as its not clear from comments 14 and 15 that the package in xenial-proposed was tested.

[Edward Hope-Morley (hopem)]

@brian-murray i tested both xenial-proposed and trusty-proposed/mitaka (UCA)

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-urllib3 - 1.13.1-2ubuntu0.16.04.1

---------------
python-urllib3 (1.13.1-2ubuntu0.16.04.1) xenial; urgency=medium

  * d/p/06_revert_square_brackets_httplib.patch: Cherry pick revert of
    change to behaviour with httplib which breaks IPv6 square bracket
    handling (LP: #1578351).

 -- James Page <email address hidden> Tue, 10 May 2016 10:00:32 +0100

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for python-urllib3 has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[James Page (james-page)]

The verification of the Stable Release Update for python-urllib3 has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[James Page (james-page)]

This bug was fixed in the package python-urllib3 - 1.13.1-2ubuntu0.16.04.1~cloud0
---------------

 python-urllib3 (1.13.1-2ubuntu0.16.04.1~cloud0) trusty-mitaka; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 python-urllib3 (1.13.1-2ubuntu0.16.04.1) xenial; urgency=medium
 .
   * d/p/06_revert_square_brackets_httplib.patch: Cherry pick revert of
     change to behaviour with httplib which breaks IPv6 square bracket
     handling (LP: #1578351).