Nova Docker: Metadata service doesn't work
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Fix Released
|
Undecided
|
Daniel Kuffner |
Bug Description
I was playing around with cloud-init. I wanted to use cloud init as substitute for the missing environment variables feature.
The basic idea is to define variables in the user data and inject the them before starting the service.
Following:
docker run -e "MY_Variable=
would look like in openstack:
nova boot --image centos:latest --user-data "MY_Variable=
Unfortunately does the metadata service not work inside of a docker container. After some testing I figured out that the reason for that is that a container uses as default gateway the docker network (docker ip address). The metadata service simple rejects the call since the IP address of docker container is not associated with the nova instance.
Note: The metadata service itself can be accessed (http://
I was able to work around the issue by simply changing the route inside the container:
# Hack: In order to receive data from the metadata service we must make sure we resolve the data via our nova network.
#
# A docker container in openstack has two NICs.
# - eth0 has a IP address on the docker0 bridge which is usually an e.g. 172.0.0.0 IP address.
# - pvnetXXXX is a IP address assigned by nova.
#
# Extract the NIC name of the nova network.
#
NOVA_NIC=$(ip a | grep pvnet | head -n 1 | cut -d: -f2)
while [ "$NOVA_NIC" == "" ] ; do
echo "Find nova NIC..."
sleep 1
NOVA_NIC=$(ip a | grep pvnet | head -n 1 | cut -d: -f2)
done
echo "Device $NOVA_NIC found. Wait until ready."
sleep 3
# Setup a network route to insure we use the nova network.
#
echo "[INFO] Create default route for $NOVA_NIC. Gateway 10.0.0.1"
ip r r default via 10.0.0.1 dev $NOVA_NIC
# Shutdown eth0 since icps will fetch enabled enterface for streaming.
ip l set down dev eth0
This approach is obviously a poor solution since it has certain expectation of the network.
Another solution might be extend the docker driver to add a firewall rule which will masquerade requests on 169.254.169.254 with the actual nova network IP address
I third solution would need improvements in docker. If docker would have a network mode which allows to assign the IP from outside this issue would be solved. That of course is a just which must be accepted by the docker community.
Network Threads:
- https:/
Simple script to test metadata inside the container:
#!/bin/bash
status=$(curl -I -s -o /dev/null -w "%{http_code}" http://
while [ $status != '200' ]; do
echo "Cannot access metadata, status: '$status', try again..."
date # easier to see in docker logs that loop is still running.
status=$(curl -I -s -o /dev/null -w "%{http_code}" http://
sleep 1
done
echo "Yes we got some user data:"
curl http://
description: | updated |
description: | updated |
description: | updated |
description: | updated |
description: | updated |
summary: |
- Docker: Metadata service doesn't work + Nova Docker: Metadata service doesn't work |
Changed in nova: | |
milestone: | none → icehouse-3 |
Changed in nova: | |
status: | Fix Committed → Fix Released |
Changed in nova: | |
milestone: | icehouse-3 → 2014.1 |
Change Set: https:/ /review. openstack. org/#/c/ 62220/
It seems to be good enough to just change the default route of the container to the nova network.