cloud init metadata injection failed while booting VM

[sutefun]

Hello,

when we start a VM the process of cloud init failed with:

Inside the VM
url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [62/120s]: bad status code [500]

The VM is reachable and you can curl http://169.254.169.254 and get a valid result but when you curl http://169.254.169.254/latest we get: Remote metadata server experienced an internal server error.

In nova-api log I see: Unauthorized: The request you have made requires authentication. (HTTP 401)

Environment

OS:
Ubuntu 16.04.1 LTS

packages on nova:
ii nova-api 2:13.1.0-0ubuntu1 all OpenStack Compute - API frontend
ii nova-common 2:13.1.0-0ubuntu1 all OpenStack Compute - common files
ii nova-conductor 2:13.1.0-0ubuntu1 all OpenStack Compute - conductor service
ii nova-consoleauth 2:13.1.0-0ubuntu1 all OpenStack Compute - Console Authenticator
ii nova-novncproxy 2:13.1.0-0ubuntu1 all OpenStack Compute - NoVNC proxy
ii nova-scheduler 2:13.1.0-0ubuntu1 all OpenStack Compute - virtual machine scheduler
ii nova-spiceproxy 2:13.1.0-0ubuntu1 all OpenStack Compute - spice html5 proxy
ii python-nova 2:13.1.0-0ubuntu1 all OpenStack Compute Python libraries
ii python-novaclient 2:3.3.1-2 all client library for OpenStack Compute API - Python 2.7
ii python-neutronclient 1:4.1.1-2 all client API library for Neutron - Python 2.7

packages on neutron:

ii neutron-common 2:8.1.2-0ubuntu1 all Neutron is a virtual network service for Openstack - common
ii neutron-dhcp-agent 2:8.1.2-0ubuntu1 all Neutron is a virtual network service for Openstack - DHCP agent
ii neutron-l3-agent 2:8.1.2-0ubuntu1 all Neutron is a virtual network service for Openstack - l3 agent
ii neutron-metadata-agent 2:8.1.2-0ubuntu1 all Neutron is a virtual network service for Openstack - metadata agent
ii neutron-openvswitch-agent 2:8.1.2-0ubuntu1 all Neutron is a virtual network service for Openstack - Open vSwitch plugin agent
ii neutron-plugin-ml2 2:8.1.2-0ubuntu1 all Neutron is a virtual network service for Openstack - ML2 plugin
ii neutron-plugin-openvswitch-agent 2:8.1.2-0ubuntu1 all Transitional package for neutron-openvswitch-agent
ii neutron-server 2:8.1.2-0ubuntu1 all Neutron is a virtual network service for Openstack - server
ii python-neutron 2:8.1.2-0ubuntu1 all Neutron is a virtual network service for Openstack - Python library
ii python-neutron-fwaas 1:8.0.0-0ubuntu1 all Firewall-as-a-Service driver for OpenStack Neutron
ii python-neutron-lib 0.0.2-2 all Neutron shared routines and utilities - Python 2.7
ii python-neutronclient 1:4.1.1-2 all client API library for Neutron - Python 2.7

Hypervisor is KVM

ii nova-compute-kvm 2:13.1.0-0ubuntu1 all OpenStack Compute - compute node (KVM)
ii libvirt-bin 1.3.1-1ubuntu10.1 amd64 programs for the libvirt library
ii libvirt0:amd64 1.3.1-1ubuntu10.1 amd64 library for interfacing with different virtualization systems
ii nova-compute-libvirt 2:13.1.0-0ubuntu1 all OpenStack Compute - compute node libvirt support
ii python-libvirt 1.3.1-1ubuntu1 amd64 libvirt Python bindings

Storage is CEPH
root@openstack11:~# ceph version
ceph version 10.2.2

Network is Neutron with openvswitch

I tried a lot to get around this issue, I hope I didn't oversee something.

Maybe someone can help or fix this bug.

Thanks in advance

[Dr. Jens Harbott (j-harbott)]

Can you confirm that you did set metadata_proxy_shared_secret consistently in nova.conf and neutron/metadata_agent.ini on the controller and network nodes?

[Sylvain Bauza (sylvain-bauza)]

So, I'm suspecting some bad configuration on the Nova side for Keystone admin authentication from neutronclient called in the metadata API service.

Since Nova recreates a new auth token for the metadata calls, we need to have the right opts in nova.conf for using the keystone strategy.

Check out nova.conf on your nova-metadata-api service and look at [keystone_authtoken] section.

Marking the bug as Invalid as I'm suspecting some configuration issue.

[sutefun (stefan-bujack)]

Hello,

I set the metadata_proxy_shared_secret in the config files. At first I had a different secret and changed it to test but that didnt't work either.

I have attached the config files. Maybe I 've overseen something.

Thanks for looking.

[sutefun (stefan-bujack)]

Hy,

this solved my problem:

https://ask.openstack.org/en/question/95889/domainnotfound-could-not-find-domain-default/

Greets