mitaka ksclient fails to connect to v6 keystone

Bug #1578351 reported by Edward Hope-Morley on 2016-05-04
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu Cloud Archive
High
Unassigned
Mitaka
High
Unassigned
keystone (Juju Charms Collection)
High
Edward Hope-Morley
python-urllib3 (Ubuntu)
High
James Page
Xenial
High
James Page
Yakkety
High
James Page

Bug Description

[Impact]
urllib3 is not stripping [] from IPv6 URL's, resulting in double square brackets which breaks connectivity to targets.

[Test Case]
<relying on Ed for actual testing>
Deploy OpenStack with IPv6 preferred; keystone service is unable to self configure as extra brackets are inserted to the URL hostname resulting in failed connections.

[Regression Potential]
Change in behaviour was reverted upstream; regression potential minimal.

[Original Bug Report]
2016-05-04 15:02:13 INFO shared-db-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 449, in _send_request
2016-05-04 15:02:13 INFO shared-db-relation-changed raise exceptions.ConnectionRefused(msg)
2016-05-04 15:02:13 INFO shared-db-relation-changed keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://[2001:db8:0:1:f816:3eff:fe2d:9445]:35347/v2.0/OS-KSADM/services

Full traceback is here http://paste.ubuntu.com/16224193/

This works fine with Liberty

Changed in keystone (Juju Charms Collection):
importance: Undecided → High
milestone: none → 16.07
Edward Hope-Morley (hopem) wrote :

root@juju-hopem-machine-9:~# ip a s| grep 2001:db8:0:1:f816:3eff:fe2d:9445
    inet6 2001:db8:0:1:f816:3eff:fe2d:9445/64 scope global dynamic
root@juju-hopem-machine-9:~# netstat -tulpn| grep 35347
tcp6 0 0 :::35347 :::* LISTEN 25463/apache2

Edward Hope-Morley (hopem) wrote :

Having analysed the requests going to keystone it appears that something is adding an extra set of square brackets to the host address:

Liberty GET request has "HOST: [2001:db8:0:1:f816:3eff:fe67:6978]:35347"

Mitaka GET request has "HOST: [[2001:db8:0:1:f816:3eff:fe67:6978]]:35347"

Whether this is a charm issue or not is debatable since the correct address appears to get passed to the keystoneclient (albeit bracketed). Something further down is inferring that it has an ipv6 address and adds more brackets without first checking that they have been added. Another way of looking at this is that the newer client wants unbracketed so we should pass it as such. Either way this is a change of behaviour in the keystoneclient.

Edward Hope-Morley (hopem) wrote :
Changed in keystone (Juju Charms Collection):
status: New → Invalid
Changed in python-urllib3 (Ubuntu):
importance: Undecided → High
Edward Hope-Morley (hopem) wrote :

Not sure if ^^ is the only patch required to fix this. Will try more recent version of urllib3 (than 1.13.1) to see if it fixes things.

Daniele Tricoli (eriol.v) wrote :

Hello, Debian Maintainer here. I'm updating both urllib3 and requests, I hope to complete this at max tomorrow. Right now I'm finishing to test urllib3 before pushing on the git repository.

Edward Hope-Morley (hopem) wrote :

Ok some more info/test results. It seems both urllib3 AND the keystone charm need fixing:

1. urlib3 needed this to get working with v6 addresses - https://github.com/shazow/urllib3/commit/a3b9a34be9dccfc42df65b2403bc0c6666fd3392

2. the keystone charm does not re-run ensure_intial_admin() once this error is overcome causing subsequent juju resolves to fail since admin creds are not setup. Should be an easy fix.

Changed in keystone (Juju Charms Collection):
status: Invalid → Triaged
tags: added: ipv6 openstack sts
Edward Hope-Morley (hopem) wrote :
Changed in keystone (Juju Charms Collection):
assignee: nobody → Edward Hope-Morley (hopem)
status: Triaged → In Progress
James Page (james-page) on 2016-05-10
Changed in python-urllib3 (Ubuntu Yakkety):
assignee: nobody → James Page (james-page)
Changed in python-urllib3 (Ubuntu Xenial):
assignee: nobody → James Page (james-page)
importance: Undecided → Medium
Changed in python-urllib3 (Ubuntu Yakkety):
status: New → Triaged
Changed in python-urllib3 (Ubuntu Xenial):
importance: Medium → High
status: New → Triaged
Changed in python-urllib3 (Ubuntu Yakkety):
status: Triaged → In Progress
James Page (james-page) wrote :

@Daniele - thanks for the update - I'm going to patch into yakkety so we can get moving asap on the xenial SRU - will resync with your updates in Debian as soon as they land.

description: updated
Changed in python-urllib3 (Ubuntu Xenial):
status: Triaged → In Progress
James Page (james-page) wrote :

Uploads made for Xenial and Yakkety; Xenial upload will require SRU team review.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-urllib3 - 1.13.1-2ubuntu1

---------------
python-urllib3 (1.13.1-2ubuntu1) yakkety; urgency=medium

  * d/p/06_revert_square_brackets_httplib.patch: Cherry pick revert of
    change to behaviour with httplib which breaks IPv6 square bracket
    handling (LP: #1578351).

 -- James Page <email address hidden> Tue, 10 May 2016 09:41:48 +0100

Changed in python-urllib3 (Ubuntu Yakkety):
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/314247
Committed: https://git.openstack.org/cgit/openstack/charm-keystone/commit/?id=216a5c9e39be799ca0e82e45750465539bbc891e
Submitter: Jenkins
Branch: master

commit 216a5c9e39be799ca0e82e45750465539bbc891e
Author: Edward Hope-Morley <email address hidden>
Date: Mon May 9 20:20:48 2016 +0100

    Ensure we can recover from error during db init

    Currently if a failure occurs during the shared-db
    hook intialisation sequence after db is inited but
    before admin creds have been setup, there is no way
    to re-run ensure_admin_credentials. This patch
    resolves that issue.

    Change-Id: Iad80a0eeae6f94dc89ff994f8e5794c60c272e16
    Closes-Bug: 1578351

Changed in keystone (Juju Charms Collection):
status: In Progress → Fix Committed
tags: added: sts-sru

Hello Edward, or anyone else affected,

Accepted python-urllib3 into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-urllib3/1.13.1-2ubuntu0.16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in python-urllib3 (Ubuntu Xenial):
status: In Progress → Fix Committed
tags: added: verification-needed
James Page (james-page) wrote :

Hello Edward, or anyone else affected,

Accepted python-urllib3 into mitaka-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:mitaka-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-mitaka-needed to verification-mitaka-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-mitaka-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-mitaka-needed
Edward Hope-Morley (hopem) wrote :

xenial-mitaka-proposed tested and confirmed all good.

tags: added: verification-done
removed: verification-needed
Edward Hope-Morley (hopem) wrote :

trusty-mitaka-proposed tested and confirmed all good.

tags: added: verification-mitaka-done
removed: verification-mitaka-needed
Brian Murray (brian-murray) wrote :

I've changed the tag back to verification-need as its not clear from comments 14 and 15 that the package in xenial-proposed was tested.

tags: added: verification-needed
removed: verification-done
Edward Hope-Morley (hopem) wrote :

@brian-murray i tested both xenial-proposed and trusty-proposed/mitaka (UCA)

tags: added: verification-done
removed: verification-needed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-urllib3 - 1.13.1-2ubuntu0.16.04.1

---------------
python-urllib3 (1.13.1-2ubuntu0.16.04.1) xenial; urgency=medium

  * d/p/06_revert_square_brackets_httplib.patch: Cherry pick revert of
    change to behaviour with httplib which breaks IPv6 square bracket
    handling (LP: #1578351).

 -- James Page <email address hidden> Tue, 10 May 2016 10:00:32 +0100

Changed in python-urllib3 (Ubuntu Xenial):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for python-urllib3 has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

James Page (james-page) wrote :

The verification of the Stable Release Update for python-urllib3 has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

James Page (james-page) wrote :

This bug was fixed in the package python-urllib3 - 1.13.1-2ubuntu0.16.04.1~cloud0
---------------

 python-urllib3 (1.13.1-2ubuntu0.16.04.1~cloud0) trusty-mitaka; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 python-urllib3 (1.13.1-2ubuntu0.16.04.1) xenial; urgency=medium
 .
   * d/p/06_revert_square_brackets_httplib.patch: Cherry pick revert of
     change to behaviour with httplib which breaks IPv6 square bracket
     handling (LP: #1578351).

Changed in cloud-archive:
status: Fix Committed → Invalid
Liam Young (gnuoy) on 2016-07-29
Changed in keystone (Juju Charms Collection):
status: Fix Committed → Fix Released
Louis Bouchard (louis) on 2016-11-09
tags: removed: sts-sru
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers