metadata agent throwing AttributeError: 'HTTPClient' object has no attribute 'auth_tenant_id' with latest release

Bug #1280941 reported by Robert Collins on 2014-02-17
36
This bug affects 6 people
Affects Status Importance Assigned to Milestone
Ubuntu Cloud Archive
Critical
Unassigned
neutron
Critical
Ihar Hrachyshka
Havana
Critical
Ihar Hrachyshka
python-neutronclient
Critical
Mark McClain
tripleo
Critical
Robert Collins
python-neutronclient (Ubuntu)
Critical
James Page
Saucy
Critical
Chuck Short
Trusty
Critical
James Page

Bug Description

So we need a new release - this is fixed in:
commit 02baef46968b816ac544b037297273ff6a4e8e1b

but until a new release is done, anyone running trunk Neutron will have the metadata agent fail.

And neutron itself is missing a versioned dep on the fixed client (but obviously that has to wait for the client release to be done)

description: updated
Changed in tripleo:
status: New → Triaged
importance: Undecided → Critical

Fix proposed to branch: master
Review: https://review.openstack.org/73904

Changed in tripleo:
assignee: nobody → Robert Collins (lifeless)
status: Triaged → In Progress

Not trunk only. The fix for neutron was backported to stable/havana, so stable is broken too.

Reviewed: https://review.openstack.org/73904
Committed: https://git.openstack.org/cgit/openstack/tripleo-image-elements/commit/?id=4c16ec1eca50e2e5f561420141ad268c2970c99e
Submitter: Jenkins
Branch: master

commit 4c16ec1eca50e2e5f561420141ad268c2970c99e
Author: Robert Collins <email address hidden>
Date: Mon Feb 17 16:24:36 2014 +1300

    Workaround broken Neutron requirements.

    Neutron have introduced another hard dependency on unreleased
    python-neutronclient :(.

    Change-Id: I10702098192c0393a91efbd707c61f5b0e627de6
    Partial-Bug: 1280941

Update:

As per @markmcclain, there's some delay publishing a new signed package to PyPI (keys republishing is in progress), that's why we haven't seen a new client release yet. As quick as we get this done, a new package will be pushed to PyPI (by @mark), and requirements.txt for global and neutron will be updated (by me).

Changed in python-neutronclient:
importance: Undecided → Critical
assignee: nobody → Mark McClain (markmcclain)
status: New → In Progress
no longer affects: neutron
Changed in neutron:
status: New → Confirmed
status: Confirmed → In Progress
importance: Undecided → Critical
assignee: nobody → Mark McClain (markmcclain)
milestone: none → icehouse-3

Fix proposed to branch: master
Review: https://review.openstack.org/75240

Changed in neutron:
assignee: Mark McClain (markmcclain) → Justin Lund (justin-lund)

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/75410

Changed in neutron:
assignee: Justin Lund (justin-lund) → Ihar Hrachyshka (ihar-hrachyshka)
Changed in python-neutronclient:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/75240
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ff2f4dd4363919d49e07eedb25123ac804a7cdc3
Submitter: Jenkins
Branch: master

commit ff2f4dd4363919d49e07eedb25123ac804a7cdc3
Author: Justin Lund <email address hidden>
Date: Thu Feb 20 18:53:39 2014 -0500

    Sync minimum requirements

    Change-Id: I1a0653058f1f8085918d7c43147d6b2631acb013
    Closes-bug:1280941

Changed in neutron:
status: In Progress → Fix Committed

We are facing following issue with the metadata agent.

Errors in neutron-metadata-agent.log

neutron.agent.metadata.agent AttributeError: 'HTTPClient' object has no attribute 'auth_tenant_id'

Errors at the vm side

HTTPError: HTTP Error 500: Internal Server Error
2014-02-22 19:24:30,169 - utils.py[ERROR]: Caught exception reading instance data
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/boto/utils.py", line 219, in retry_url
    r = opener.open(req)
  File "/usr/lib64/python2.6/urllib2.py", line 397, in open
    response = meth(req, response)
  File "/usr/lib64/python2.6/urllib2.py", line 510, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib64/python2.6/urllib2.py", line 435, in error
    return self._call_chain(*args)
  File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.6/urllib2.py", line 518, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 500: Internal Server Error
2014-02-22 19:25:34,285 - utils.py[ERROR]: Caught exception reading instance data
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/boto/utils.py", line 219, in retry_url
    r = opener.open(req)
  File "/usr/lib64/python2.6/urllib2.py", line 397, in open
    response = meth(req, response)
  File "/usr/lib64/python2.6/urllib2.py", line 510, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib64/python2.6/urllib2.py", line 435, in error
    return self._call_chain(*args)
  File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.6/urllib2.py", line 518, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 500: Internal Server Error

The original rdo installed version was python-neutronclient-2.3.1-2.el6.noarch and we tried to upgrade this to python-neutronclient-2.3.3-1.el6.noarch.rpm

Ting (tul75123) wrote :

Hi, is it solved in stable havana? After I updated code today from yum. I'm facing the same problem....

It's not solved in havana. See: https://review.openstack.org/#/c/75410/ Jenkins task failed because gate is almost broken for neutron stable (not master).

Adil, please report the issue downstream, in RDO.

In RDO, they should either package new client release (2.3.4) or backport this particular fix to their code.

Pádraig Brady (p-draigbrady) wrote :

Fix is being backported to python-neutronclient-2.3.1-3 in RDO

Thierry Carrez (ttx) on 2014-03-05
Changed in neutron:
status: Fix Committed → Fix Released
James Page (james-page) on 2014-03-07
Changed in python-neutronclient (Ubuntu):
importance: Undecided → Critical
Changed in python-neutronclient (Ubuntu Saucy):
importance: Undecided → Critical
Changed in python-neutronclient (Ubuntu):
status: New → Fix Released
Changed in python-neutronclient (Ubuntu Saucy):
status: New → Triaged
James Page (james-page) wrote :

SRU'ing the required fix for neutronclient to Ubuntu 13.10 and the Havana Cloud Archive.

Changed in cloud-archive:
status: New → Triaged
importance: Undecided → Critical
James Page (james-page) on 2014-03-07
Changed in python-neutronclient (Ubuntu Saucy):
status: Triaged → In Progress
assignee: nobody → Chuck Short (zulcss)

Hello Robert, or anyone else affected,

Accepted python-neutronclient into saucy-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/python-neutronclient/1:2.3.0-0ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in python-neutronclient (Ubuntu Saucy):
status: In Progress → Fix Committed
tags: added: verification-needed
James Page (james-page) on 2014-03-07
Changed in cloud-archive:
status: Triaged → Fix Committed
Ante Karamatić (ivoks) wrote :

I confirm proposed package solves the problem:

cloud-init start-local running: Fri, 07 Mar 2014 19:14:22 +0000. up 5.24 seconds
no instance data found in start-local
ci-info: lo : 1 127.0.0.1 255.0.0.0 .
ci-info: eth0 : 1 192.168.100.2 255.255.255.0 fa:16:3e:30:b4:f7
ci-info: route-0: 0.0.0.0 192.168.100.1 0.0.0.0 eth0 UG
ci-info: route-1: 192.168.100.0 0.0.0.0 255.255.255.0 eth0 U
cloud-init start running: Fri, 07 Mar 2014 19:14:23 +0000. up 6.82 seconds
found data source: DataSourceEc2
2014-03-07 19:14:27,960 - __init__.py[WARNING]: Unhandled non-multipart userdata ''
Generating public/private rsa key pair.
Your identification has been saved in /etc/ssh/ssh_host_rsa_key.
Your public key has been saved in /etc/ssh/ssh_host_rsa_key.pub.

While previously it was:

cloud-init start-local running: Fri, 07 Mar 2014 19:08:32 +0000. up 5.00 seconds
no instance data found in start-local
ci-info: lo : 1 127.0.0.1 255.0.0.0 .
ci-info: eth0 : 1 192.168.100.2 255.255.255.0 fa:16:3e:0f:ed:de
ci-info: route-0: 0.0.0.0 192.168.100.1 0.0.0.0 eth0 UG
ci-info: route-1: 192.168.100.0 0.0.0.0 255.255.255.0 eth0 U
cloud-init start running: Fri, 07 Mar 2014 19:08:34 +0000. up 6.35 seconds
2014-03-07 19:08:36,915 - utils.py[ERROR]: Caught exception reading instance data
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/boto/utils.py", line 177, in retry_url
    resp = urllib2.urlopen(req)
  File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 406, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 519, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 444, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 527, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 500: Internal Server Error

tags: added: verification-done
removed: verification-needed
James Page (james-page) wrote :

Confirmed verification of fix resolved original problem.

The verification of the Stable Release Update for python-neutronclient has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-neutronclient - 1:2.3.0-0ubuntu1.1

---------------
python-neutronclient (1:2.3.0-0ubuntu1.1) saucy-proposed; urgency=medium

  * debian/patches/fix-get-auth-info.patch: Fix regression introduced by
    stable/havanna update. (LP: #1280941)
 -- Chuck Short <email address hidden> Fri, 07 Mar 2014 12:04:11 -0500

Changed in python-neutronclient (Ubuntu Saucy):
status: Fix Committed → Fix Released
James Page (james-page) wrote :

CA update for havana

--
 python-neutronclient (1:2.3.0-0ubuntu1.1~cloud0) precise-havana; urgency=low

   * New update for the Ubuntu Cloud Archive.

 python-neutronclient (1:2.3.0-0ubuntu1.1) saucy-proposed; urgency=medium

   * debian/patches/fix-get-auth-info.patch: Fix regression introduced by
     stable/havanna update. (LP: #1280941)

Changed in cloud-archive:
status: Fix Committed → Fix Released
James Page (james-page) on 2014-03-13
Changed in python-neutronclient (Ubuntu Trusty):
status: Fix Released → In Progress
assignee: nobody → James Page (james-page)
James Page (james-page) on 2014-03-13
Changed in python-neutronclient (Ubuntu Trusty):
status: In Progress → Fix Committed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-neutronclient - 1:2.3.4-0ubuntu1

---------------
python-neutronclient (1:2.3.4-0ubuntu1) trusty; urgency=medium

  * New upstream point release:
    - [02baef4] Fix get_auth_info() for pre-authenticated clients
      (LP: #1280941).
 -- James Page <email address hidden> Thu, 13 Mar 2014 11:02:06 +0000

Changed in python-neutronclient (Ubuntu Trusty):
status: Fix Committed → Fix Released

Reviewed: https://review.openstack.org/82154
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a75435dda8154914f6d6355d9ea6368b53a737e3
Submitter: Jenkins
Branch: stable/havana

commit a75435dda8154914f6d6355d9ea6368b53a737e3
Author: Ihar Hrachyshka <email address hidden>
Date: Fri Feb 21 14:50:51 2014 +0100

    Updated neutronclient dependency

    This is needed by: 15a912b1ca3c24ba8851b8b77d6de8027e120d78

    Change-Id: If0eab1b5dd102cc5ceb7720a8465ee47a708839d
    Closes-Bug: #1280941

TripleO should revert their workaround and update requirements.txt from globals to fix the issue.

Robert Collins (lifeless) wrote :

@ihar please do submit a patch :). Note that the requirements.txt in question is neutrons one.

@robert, the patch for requirements in neutron already arrived to both stable and master, tripleo workaround is not needed now.

Aha, tripleo already reverted the workaround. So nothing should be done in tripleo, we may mark the bug as fixed.

Changed in tripleo:
status: In Progress → Fix Committed
Thierry Carrez (ttx) on 2014-04-17
Changed in neutron:
milestone: icehouse-3 → 2014.1
Changed in tripleo:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers