OpenStack Nova Cloud Controller Charm

PY3: nova meta-data 404

Bug #1796405 reported by Corey Bryant
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Nova Cloud Controller Charm
Fix Released
High
Liam Young
OpenStack Nova Cloud Controller Charm 19.04

Bug Description

Running tempest smoke against a full py3 workload deployment results in some tempest failures due to metadata 404 errors.

To recreate
-----------
(clients) ubuntu@coreycb-bastion:~/openstack-charm-testing/tempest$ ssh cirros@10.5.150.17
Warning: Permanently added '10.5.150.17' (RSA) to the list of known hosts.
cirros@10.5.150.17's password:
$ curl http://169.254.169.254/latest/meta-data/public-ipv4
<html>
 <head>
  <title>404 Not Found</title>
 </head>
 <body>
  <h1>404 Not Found</h1>
  The resource could not be found.<br /><br />
 </body>
</html>$

neutron-gateway/0
-----------------
/var/log/neutron/neutron-metadata-agent.log:

2018-10-05 20:27:37.504 2341 INFO eventlet.wsgi.server [-] 192.168.21.22,<local> "GET /latest/meta-data/public-ipv4 HTTP/1.1" status: 404 len: 297 time: 5.1055355

neutron-gateway/0
-----------------
systemctl status nova-api-metadata # service is down -- makes sense, recently was moved to nova-cc

nova-cloud-controller/0
-----------------------
/etc/apache2/sites-enabled/wsgi-openstack-metadata.conf # it runs under apache2 now

/var/log/apache2/nova_meta_access.log:

10.5.0.66 - - [05/Oct/2018:20:27:32 +0000] "GET /latest/meta-data/public-ipv4 HTTP/1.1" 404 366 "-" "python-requests/2.18.4"

/var/log/nova/nova-metadata-wsgi.log:

2018-10-05 20:27:36.896 13122 WARNING nova.api.metadata.handler [req-a03d0daa-7e86-4073-8c4e-c1062ff8b39b - - - - -] X-Instance-ID present in request headers. The 'service_metadata_proxy' option must be enabled to process this header.

2018-10-05 20:27:37.497 13122 ERROR nova.api.metadata.handler [req-a03d0daa-7e86-4073-8c4e-c1062ff8b39b - - - - -] Failed to get metadata for IP 10.5.0.66: no metadata

sudo grep service_metadata_proxy /etc/nova/nova.conf # no hits - service_metadata_proxy missing from nova.conf

templates/parts/section-neutron has:

{% if enable_metadata -%}
service_metadata_proxy = True
metadata_proxy_shared_secret = {{ metadata_proxy_shared_secret }}
{% endif -%}

hooks/nova_cc_contexts.py has:
class NovaMetadataContext():
    ....
    if cmp_os_release >= 'rocky':
        ctxt['enable_metadata'] = True

So it seems as though the metadata options should be rendered to nova.conf since the deployment is >= rocky.

See original description
Corey Bryant (corey.bryant)
description: updated
description: updated
summary: - PY3: meta-data 404
+ PY3: nova meta-data 404
Liam Young (gnuoy)
Changed in charm-nova-cloud-controller:
assignee: nobody → Liam Young (gnuoy)
Liam Young (gnuoy)
Changed in charm-nova-cloud-controller:
status: New → Confirmed
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-nova-cloud-controller (master)

Fix proposed to branch: master
Review: https://review.openstack.org/608641

Changed in charm-nova-cloud-controller:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-nova-cloud-controller (master)

Reviewed: https://review.openstack.org/608641
Committed: https://git.openstack.org/cgit/openstack/charm-nova-cloud-controller/commit/?id=fabc0c9dbb4d517eee88827f90c9588993cedd9d
Submitter: Zuul
Branch: master

commit fabc0c9dbb4d517eee88827f90c9588993cedd9d
Author: Liam Young <email address hidden>
Date: Mon Oct 8 11:04:54 2018 +0000

    Use shared neutron template fragment.

    The neutron template fragment in charmhelpers has support for
    metadata, so bring the pike template inline witt all the other
    nova.conf templates in the charm and use parts/section-neutron.

    Change-Id: I35514988aa9d2c289ec20dd3394846386cb1559e
    Closes-Bug: 1796405

Changed in charm-nova-cloud-controller:
status: In Progress → Fix Committed
Revision history for this message
Corey Bryant (corey.bryant) wrote :

I just tested with this fix and I'm still hitting the following. Looks like service_metadata_proxy needs to be set.

2018-10-09 14:14:05.730 3354 WARNING nova.api.metadata.handler [req-30e1ccb4-e2c2-45f0-8a82-f942c030b51b - - - - -] X-Instance-ID present in request headers. The 'service_metadata_proxy' option must be enabled to process this header.
2018-10-09 14:14:05.875 3354 ERROR nova.api.metadata.handler [req-30e1ccb4-e2c2-45f0-8a82-f942c030b51b - - - - -] Failed to get metadata for IP 10.5.0.39: no metadata

Revision history for this message
Corey Bryant (corey.bryant) wrote :

A diff with templates/parts/section-neutron shows that templates/pike/nova.conf doesn't include the following:

# {{ enable_metadata }}
{% if enable_metadata -%}
service_metadata_proxy = True
metadata_proxy_shared_secret = {{ metadata_proxy_shared_secret }}
{% endif -%}

Revision history for this message
Corey Bryant (corey.bryant) wrote :

Adding those bits to templates/pike/nova.conf seems to fix it.

Before:

$ curl http://169.254.169.254/latest/meta-data/public-ipv4
<html>
 <head>
  <title>404 Not Found</title>
 </head>
 <body>
  <h1>404 Not Found</h1>
  The resource could not be found.<br /><br />
 </body>
</html>$

After:

$ curl http://169.254.169.254/latest/meta-data/public-ipv4
10.5.150.6$

Revision history for this message
Corey Bryant (corey.bryant) wrote :

Ok duh, I was reading the code review backwards and wasn't running with the fix.

Revision history for this message
Corey Bryant (corey.bryant) wrote :

Tested this out today for real and it works like a charm!

David Ames (thedac)
Changed in charm-nova-cloud-controller:
milestone: none → 19.04
David Ames (thedac)
Changed in charm-nova-cloud-controller:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.