Nova Metadata server in Mitaka can not work with Liberty config

[Matt Riedemann]

http://logs.openstack.org/59/265759/24/experimental/gate-grenade-dsvm-neutron-multinode/8f1deec/logs/new/screen-n-api.txt.gz?level=INFO#_2016-02-12_16_28_16_860

2016-02-12 16:28:16.860 20168 INFO nova.metadata.wsgi.server [-] Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 470, in handle_one_response
    result = self.application(self.environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/paste/urlmap.py", line 216, in __call__
    return app(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/opt/stack/new/nova/nova/api/ec2/__init__.py", line 32, in __call__
    return webob.exc.HTTPException(message=_DEPRECATION_MESSAGE)
TypeError: __init__() takes exactly 3 arguments (2 given)

This only shows up in the gate-grenade-dsvm-neutron-multinode job which is not running the n-api-meta service but is running the neutron metadata service, which has a bunch of warnings because it's not getting valid responses back from the nova metadata API (b/c it's not running):

http://logs.openstack.org/59/265759/24/experimental/gate-grenade-dsvm-neutron-multinode/8f1deec/logs/new/screen-q-meta.txt.gz?level=TRACE

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/279721

[Sean M. Collins (scollins)]

+Neutron and myself since it's the grenade multinode job that is being hit

[Sean Dague (sdague)]

There is still a 404 on getting the metadata after this patch, there is a logging change out here - https://review.openstack.org/#/c/282025/ for nova to try to expose what's going on, as in reading the code, there are a number of things that might be passed that would make this fail.

[Armando Migliaccio (armando-migliaccio)]

Results from run [1] still didn't yield any result. So that's my analysis: the api-paste file is not upgraded and that still contains an ec2 middleware (ec2faultwrap). This is the one that responds with 404, inline with what patch [3] does. That's the reason why we don't see the trace from Sean's patch [4]. I think the bug lies in grenade: if we get rid of ec2faultwrap we should get past the 404 error.

[1] http://logs.openstack.org/00/281600/4/experimental/gate-grenade-dsvm-neutron-multinode/28b0630//logs/new/
[2] http://logs.openstack.org/00/281600/4/experimental/gate-grenade-dsvm-neutron-multinode/28b0630//logs/etc/nova/api-paste.ini.txt.gz
[3] https://review.openstack.org/#/c/279721/3/nova/api/ec2/__init__.py
[4] https://review.openstack.org/#/c/282025/

[Armando Migliaccio (armando-migliaccio)]

I'll try to post a patch in grenade to corroborate my claim

[Armando Migliaccio (armando-migliaccio)]

the WIP patch here: https://review.openstack.org/#/c/282110/

[Sean Dague (sdague)]

Interesting. If that's the case, then I think the thing we have to do is create a stub ec2faultwrap middleware which prints out a warning that this needs to be removed. I can dig into it in the morning. We've hit this kind of issue in the past in adjusting middleware drops.

[Armando Migliaccio (armando-migliaccio)]

Looks like my grenade fix that removed the deprecated middleware [1] did the trick. We got a clean pass in [2].

[1] http://logs.openstack.org/00/281600/6/experimental/gate-grenade-dsvm-neutron-multinode/40e16c8/logs/etc/nova/api-paste.ini.txt.gz
[2] http://logs.openstack.org/00/281600/6/experimental/gate-grenade-dsvm-neutron-multinode/40e16c8/logs/testr_results.html.gz

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/282161

[OpenStack Infra (hudson-openstack)]

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/282110
Reason: No longer needed. Nova decided to keep the middleware for yet another cycle.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/282161
Reason: sorted by keeping the middleware for yet another cycle

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/279721
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=f2c1cfae53e7d9efb36b1a1b3a8d6334d9ca688f
Submitter: Jenkins
Branch: master

commit f2c1cfae53e7d9efb36b1a1b3a8d6334d9ca688f
Author: Matt Riedemann <email address hidden>
Date: Fri Feb 12 10:46:13 2016 -0800

    Partial revert of ec2 removal patch

    In Id7936be290b6febd18deb4c2db8ea4d678d4d9b1 and
    I8bf7cbaa7015bb61656ab90ccc8f944aaeebb095 parts of the ec2
    infrastructure were removed that had been deprecated and warned about
    in Liberty.

    However, changes were made to the metadata server in these changes as
    well, none of which had been warned about, and arguably should not be
    removed ever. The metadata server is an AWS construct, and given the
    implementation order of APIs in Nova (ec2 was the first API) it shares
    code with the ec2 base classes. But it's the ec2 API we were
    deprecating out and deleting.

    Removing these breaks anyone upgrading using the metadata server, as
    was seen by the neutron multinode job.

    In addition, paste filters should *never* return a response code
    directly. Doing so means they block processing the rest of the
    pipeline. Which means that the deprecation approach that was used here
    actually forcably broke anyone with existing deploys.

    Change-Id: I22eb3a3fcd8e74a1d9085acde15c25a927ae12cb
    Closes-Bug: #1545101

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/nova 13.0.0.0b3 development milestone.