Nova Docker: Metadata service doesn't work

[Daniel Kuffner]

I was playing around with cloud-init. I wanted to use cloud init as substitute for the missing environment variables feature.
The basic idea is to define variables in the user data and inject the them before starting the service.

Following:
docker run -e "MY_Variable=MyValue" -d centos

would look like in openstack:

nova boot --image centos:latest --user-data "MY_Variable=MyValue" myinstance

Unfortunately does the metadata service not work inside of a docker container. After some testing I figured out that the reason for that is that a container uses as default gateway the docker network (docker ip address). The metadata service simple rejects the call since the IP address of docker container is not associated with the nova instance.

Note: The metadata service itself can be accessed (http://169.254.169.254) but it is not possible to access the actual data (http://169.254.169.254/2009-04-04 - Status 404)

I was able to work around the issue by simply changing the route inside the container:

# Hack: In order to receive data from the metadata service we must make sure we resolve the data via our nova network.
#
# A docker container in openstack has two NICs.
# - eth0 has a IP address on the docker0 bridge which is usually an e.g. 172.0.0.0 IP address.
# - pvnetXXXX is a IP address assigned by nova.
#
# Extract the NIC name of the nova network.
#
NOVA_NIC=$(ip a | grep pvnet | head -n 1 | cut -d: -f2)

while [ "$NOVA_NIC" == "" ] ; do
   echo "Find nova NIC..."
   sleep 1
   NOVA_NIC=$(ip a | grep pvnet | head -n 1 | cut -d: -f2)
done

echo "Device $NOVA_NIC found. Wait until ready."
sleep 3

# Setup a network route to insure we use the nova network.
#
echo "[INFO] Create default route for $NOVA_NIC. Gateway 10.0.0.1"
ip r r default via 10.0.0.1 dev $NOVA_NIC

# Shutdown eth0 since icps will fetch enabled enterface for streaming.
ip l set down dev eth0

This approach is obviously a poor solution since it has certain expectation of the network.

Another solution might be extend the docker driver to add a firewall rule which will masquerade requests on 169.254.169.254 with the actual nova network IP address

I third solution would need improvements in docker. If docker would have a network mode which allows to assign the IP from outside this issue would be solved. That of course is a just which must be accepted by the docker community.

Network Threads:
- https://groups.google.com/forum/#!topic/docker-dev/YfCeX8TBweA

Simple script to test metadata inside the container:

#!/bin/bash
status=$(curl -I -s -o /dev/null -w "%{http_code}" http://169.254.169.254/2009-04-04)

while [ $status != '200' ]; do
   echo "Cannot access metadata, status: '$status', try again..."
   date # easier to see in docker logs that loop is still running.
   status=$(curl -I -s -o /dev/null -w "%{http_code}" http://169.254.169.254/2009-04-04)
   sleep 1
done

echo "Yes we got some user data:"
curl http://169.254.169.254/2009-04-04

[Daniel Kuffner (daniel-kuffner)]

Change Set: https://review.openstack.org/#/c/62220/
It seems to be good enough to just change the default route of the container to the nova network.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/62220
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=709410d243a97d35c3da314b41bab039eac75736
Submitter: Jenkins
Branch: master

commit 709410d243a97d35c3da314b41bab039eac75736
Author: Daniel Kuffner <email address hidden>
Date: Sun Dec 15 09:53:17 2013 +0100

    Nova Docker: Metadata service doesn't work

    The metadata services rejects the request because the remote IP address
    belongs to the docker network. The remote IP address is correct by using
    the nova network as default route.

    Change-Id: I410a9bfea560f669515b31db7f007515b4d5c4e7
    Closes-Bug: #1259267
    Closes-Bug: #1261021