| 2013-12-09 18:07:57 |
Daniel Kuffner |
bug |
|
|
added bug |
| 2013-12-09 18:08:24 |
Daniel Kuffner |
description |
I was playing around with cloud-init. I wanted to use cloud init as substitute for the mission environment variables.
The basic idea is to define variables in the user data and inject the them before starting the service.
Following:
docker run -e "MY_Variable=MyValue" -d centos
would look like in openstack:
nova boot --image centos:latest --userdata "MY_Variable=MyValue" myinstance
Unfortunately does the metadata service not work inside of a docker container.
After some testing I figured out that the reason for that is that a container
uses as default gateway the docker network (docker ip address). The metadata service
simple rejects the call since the IP address of docker container is not associated with
the nova instance.
I was able to work around the issue by simply changing the route inside the container:
# Hack: In order to receive data from the metadata service we must make sure we resolve the data via our nova network.
#
# A docker container in openstack has two NICs.
# - eth0 has a IP address on the docker0 bridge which is usually an e.g. 172.0.0.0 IP address.
# - pvnetXXXX is a IP address assigned by nova.
#
# Extract the NIC name of the nova network.
#
NOVA_NIC=$(ip a | grep pvnet | head -n 1 | cut -d: -f2)
while [ "$NOVA_NIC" == "" ] ; do
echo "Find nova NIC..."
sleep 1
NOVA_NIC=$(ip a | grep pvnet | head -n 1 | cut -d: -f2)
done
echo "Device $NOVA_NIC found. Wait until ready."
sleep 3
# Setup a network route to insure we use the nova network.
#
echo "[INFO] Create default route for $NOVA_NIC. Gateway 10.0.0.1"
ip r r default via 10.0.0.1 dev $NOVA_NIC
# Shutdown eth0 since icps will fetch enabled enterface for streaming.
ip l set down dev eth0
This approach is obviously a poor solution since it has certain expectation of the network.
Another solution might be extend the docker driver to add a firewall rule which will masquerade requests on 169.254.169.254 with the actual nova network IP address |
I was playing around with cloud-init. I wanted to use cloud init as substitute for the missing environment variables feature.
The basic idea is to define variables in the user data and inject the them before starting the service.
Following:
docker run -e "MY_Variable=MyValue" -d centos
would look like in openstack:
nova boot --image centos:latest --userdata "MY_Variable=MyValue" myinstance
Unfortunately does the metadata service not work inside of a docker container.
After some testing I figured out that the reason for that is that a container
uses as default gateway the docker network (docker ip address). The metadata service
simple rejects the call since the IP address of docker container is not associated with
the nova instance.
I was able to work around the issue by simply changing the route inside the container:
# Hack: In order to receive data from the metadata service we must make sure we resolve the data via our nova network.
#
# A docker container in openstack has two NICs.
# - eth0 has a IP address on the docker0 bridge which is usually an e.g. 172.0.0.0 IP address.
# - pvnetXXXX is a IP address assigned by nova.
#
# Extract the NIC name of the nova network.
#
NOVA_NIC=$(ip a | grep pvnet | head -n 1 | cut -d: -f2)
while [ "$NOVA_NIC" == "" ] ; do
echo "Find nova NIC..."
sleep 1
NOVA_NIC=$(ip a | grep pvnet | head -n 1 | cut -d: -f2)
done
echo "Device $NOVA_NIC found. Wait until ready."
sleep 3
# Setup a network route to insure we use the nova network.
#
echo "[INFO] Create default route for $NOVA_NIC. Gateway 10.0.0.1"
ip r r default via 10.0.0.1 dev $NOVA_NIC
# Shutdown eth0 since icps will fetch enabled enterface for streaming.
ip l set down dev eth0
This approach is obviously a poor solution since it has certain expectation of the network.
Another solution might be extend the docker driver to add a firewall rule which will masquerade requests on 169.254.169.254 with the actual nova network IP address |
|
| 2013-12-09 18:08:54 |
Daniel Kuffner |
description |
I was playing around with cloud-init. I wanted to use cloud init as substitute for the missing environment variables feature.
The basic idea is to define variables in the user data and inject the them before starting the service.
Following:
docker run -e "MY_Variable=MyValue" -d centos
would look like in openstack:
nova boot --image centos:latest --userdata "MY_Variable=MyValue" myinstance
Unfortunately does the metadata service not work inside of a docker container.
After some testing I figured out that the reason for that is that a container
uses as default gateway the docker network (docker ip address). The metadata service
simple rejects the call since the IP address of docker container is not associated with
the nova instance.
I was able to work around the issue by simply changing the route inside the container:
# Hack: In order to receive data from the metadata service we must make sure we resolve the data via our nova network.
#
# A docker container in openstack has two NICs.
# - eth0 has a IP address on the docker0 bridge which is usually an e.g. 172.0.0.0 IP address.
# - pvnetXXXX is a IP address assigned by nova.
#
# Extract the NIC name of the nova network.
#
NOVA_NIC=$(ip a | grep pvnet | head -n 1 | cut -d: -f2)
while [ "$NOVA_NIC" == "" ] ; do
echo "Find nova NIC..."
sleep 1
NOVA_NIC=$(ip a | grep pvnet | head -n 1 | cut -d: -f2)
done
echo "Device $NOVA_NIC found. Wait until ready."
sleep 3
# Setup a network route to insure we use the nova network.
#
echo "[INFO] Create default route for $NOVA_NIC. Gateway 10.0.0.1"
ip r r default via 10.0.0.1 dev $NOVA_NIC
# Shutdown eth0 since icps will fetch enabled enterface for streaming.
ip l set down dev eth0
This approach is obviously a poor solution since it has certain expectation of the network.
Another solution might be extend the docker driver to add a firewall rule which will masquerade requests on 169.254.169.254 with the actual nova network IP address |
I was playing around with cloud-init. I wanted to use cloud init as substitute for the missing environment variables feature.
The basic idea is to define variables in the user data and inject the them before starting the service.
Following:
docker run -e "MY_Variable=MyValue" -d centos
would look like in openstack:
nova boot --image centos:latest --userdata "MY_Variable=MyValue" myinstance
Unfortunately does the metadata service not work inside of a docker container. After some testing I figured out that the reason for that is that a container uses as default gateway the docker network (docker ip address). The metadata service simple rejects the call since the IP address of docker container is not associated with the nova instance.
I was able to work around the issue by simply changing the route inside the container:
# Hack: In order to receive data from the metadata service we must make sure we resolve the data via our nova network.
#
# A docker container in openstack has two NICs.
# - eth0 has a IP address on the docker0 bridge which is usually an e.g. 172.0.0.0 IP address.
# - pvnetXXXX is a IP address assigned by nova.
#
# Extract the NIC name of the nova network.
#
NOVA_NIC=$(ip a | grep pvnet | head -n 1 | cut -d: -f2)
while [ "$NOVA_NIC" == "" ] ; do
echo "Find nova NIC..."
sleep 1
NOVA_NIC=$(ip a | grep pvnet | head -n 1 | cut -d: -f2)
done
echo "Device $NOVA_NIC found. Wait until ready."
sleep 3
# Setup a network route to insure we use the nova network.
#
echo "[INFO] Create default route for $NOVA_NIC. Gateway 10.0.0.1"
ip r r default via 10.0.0.1 dev $NOVA_NIC
# Shutdown eth0 since icps will fetch enabled enterface for streaming.
ip l set down dev eth0
This approach is obviously a poor solution since it has certain expectation of the network.
Another solution might be extend the docker driver to add a firewall rule which will masquerade requests on 169.254.169.254 with the actual nova network IP address |
|
| 2013-12-09 19:20:01 |
Daniel Kuffner |
description |
I was playing around with cloud-init. I wanted to use cloud init as substitute for the missing environment variables feature.
The basic idea is to define variables in the user data and inject the them before starting the service.
Following:
docker run -e "MY_Variable=MyValue" -d centos
would look like in openstack:
nova boot --image centos:latest --userdata "MY_Variable=MyValue" myinstance
Unfortunately does the metadata service not work inside of a docker container. After some testing I figured out that the reason for that is that a container uses as default gateway the docker network (docker ip address). The metadata service simple rejects the call since the IP address of docker container is not associated with the nova instance.
I was able to work around the issue by simply changing the route inside the container:
# Hack: In order to receive data from the metadata service we must make sure we resolve the data via our nova network.
#
# A docker container in openstack has two NICs.
# - eth0 has a IP address on the docker0 bridge which is usually an e.g. 172.0.0.0 IP address.
# - pvnetXXXX is a IP address assigned by nova.
#
# Extract the NIC name of the nova network.
#
NOVA_NIC=$(ip a | grep pvnet | head -n 1 | cut -d: -f2)
while [ "$NOVA_NIC" == "" ] ; do
echo "Find nova NIC..."
sleep 1
NOVA_NIC=$(ip a | grep pvnet | head -n 1 | cut -d: -f2)
done
echo "Device $NOVA_NIC found. Wait until ready."
sleep 3
# Setup a network route to insure we use the nova network.
#
echo "[INFO] Create default route for $NOVA_NIC. Gateway 10.0.0.1"
ip r r default via 10.0.0.1 dev $NOVA_NIC
# Shutdown eth0 since icps will fetch enabled enterface for streaming.
ip l set down dev eth0
This approach is obviously a poor solution since it has certain expectation of the network.
Another solution might be extend the docker driver to add a firewall rule which will masquerade requests on 169.254.169.254 with the actual nova network IP address |
I was playing around with cloud-init. I wanted to use cloud init as substitute for the missing environment variables feature.
The basic idea is to define variables in the user data and inject the them before starting the service.
Following:
docker run -e "MY_Variable=MyValue" -d centos
would look like in openstack:
nova boot --image centos:latest --userdata "MY_Variable=MyValue" myinstance
Unfortunately does the metadata service not work inside of a docker container. After some testing I figured out that the reason for that is that a container uses as default gateway the docker network (docker ip address). The metadata service simple rejects the call since the IP address of docker container is not associated with the nova instance.
I was able to work around the issue by simply changing the route inside the container:
# Hack: In order to receive data from the metadata service we must make sure we resolve the data via our nova network.
#
# A docker container in openstack has two NICs.
# - eth0 has a IP address on the docker0 bridge which is usually an e.g. 172.0.0.0 IP address.
# - pvnetXXXX is a IP address assigned by nova.
#
# Extract the NIC name of the nova network.
#
NOVA_NIC=$(ip a | grep pvnet | head -n 1 | cut -d: -f2)
while [ "$NOVA_NIC" == "" ] ; do
echo "Find nova NIC..."
sleep 1
NOVA_NIC=$(ip a | grep pvnet | head -n 1 | cut -d: -f2)
done
echo "Device $NOVA_NIC found. Wait until ready."
sleep 3
# Setup a network route to insure we use the nova network.
#
echo "[INFO] Create default route for $NOVA_NIC. Gateway 10.0.0.1"
ip r r default via 10.0.0.1 dev $NOVA_NIC
# Shutdown eth0 since icps will fetch enabled enterface for streaming.
ip l set down dev eth0
This approach is obviously a poor solution since it has certain expectation of the network.
Another solution might be extend the docker driver to add a firewall rule which will masquerade requests on 169.254.169.254 with the actual nova network IP address
I third solution would need improvements in docker. If docker would have a network mode which allows to assign the IP from outside this issue would be solved. That of course is a just which must be accepted by the docker community.
Network Threads:
- https://groups.google.com/forum/#!topic/docker-dev/YfCeX8TBweA |
|
| 2013-12-09 22:33:36 |
Daniel Kuffner |
description |
I was playing around with cloud-init. I wanted to use cloud init as substitute for the missing environment variables feature.
The basic idea is to define variables in the user data and inject the them before starting the service.
Following:
docker run -e "MY_Variable=MyValue" -d centos
would look like in openstack:
nova boot --image centos:latest --userdata "MY_Variable=MyValue" myinstance
Unfortunately does the metadata service not work inside of a docker container. After some testing I figured out that the reason for that is that a container uses as default gateway the docker network (docker ip address). The metadata service simple rejects the call since the IP address of docker container is not associated with the nova instance.
I was able to work around the issue by simply changing the route inside the container:
# Hack: In order to receive data from the metadata service we must make sure we resolve the data via our nova network.
#
# A docker container in openstack has two NICs.
# - eth0 has a IP address on the docker0 bridge which is usually an e.g. 172.0.0.0 IP address.
# - pvnetXXXX is a IP address assigned by nova.
#
# Extract the NIC name of the nova network.
#
NOVA_NIC=$(ip a | grep pvnet | head -n 1 | cut -d: -f2)
while [ "$NOVA_NIC" == "" ] ; do
echo "Find nova NIC..."
sleep 1
NOVA_NIC=$(ip a | grep pvnet | head -n 1 | cut -d: -f2)
done
echo "Device $NOVA_NIC found. Wait until ready."
sleep 3
# Setup a network route to insure we use the nova network.
#
echo "[INFO] Create default route for $NOVA_NIC. Gateway 10.0.0.1"
ip r r default via 10.0.0.1 dev $NOVA_NIC
# Shutdown eth0 since icps will fetch enabled enterface for streaming.
ip l set down dev eth0
This approach is obviously a poor solution since it has certain expectation of the network.
Another solution might be extend the docker driver to add a firewall rule which will masquerade requests on 169.254.169.254 with the actual nova network IP address
I third solution would need improvements in docker. If docker would have a network mode which allows to assign the IP from outside this issue would be solved. That of course is a just which must be accepted by the docker community.
Network Threads:
- https://groups.google.com/forum/#!topic/docker-dev/YfCeX8TBweA |
I was playing around with cloud-init. I wanted to use cloud init as substitute for the missing environment variables feature.
The basic idea is to define variables in the user data and inject the them before starting the service.
Following:
docker run -e "MY_Variable=MyValue" -d centos
would look like in openstack:
nova boot --image centos:latest --user-data "MY_Variable=MyValue" myinstance
Unfortunately does the metadata service not work inside of a docker container. After some testing I figured out that the reason for that is that a container uses as default gateway the docker network (docker ip address). The metadata service simple rejects the call since the IP address of docker container is not associated with the nova instance.
Note: The metadata service itself can be accessed (http://169.254.169.254) but it is not possible to access the actual data (http://169.254.169.254/2009-04-04 - Status 404)
I was able to work around the issue by simply changing the route inside the container:
# Hack: In order to receive data from the metadata service we must make sure we resolve the data via our nova network.
#
# A docker container in openstack has two NICs.
# - eth0 has a IP address on the docker0 bridge which is usually an e.g. 172.0.0.0 IP address.
# - pvnetXXXX is a IP address assigned by nova.
#
# Extract the NIC name of the nova network.
#
NOVA_NIC=$(ip a | grep pvnet | head -n 1 | cut -d: -f2)
while [ "$NOVA_NIC" == "" ] ; do
echo "Find nova NIC..."
sleep 1
NOVA_NIC=$(ip a | grep pvnet | head -n 1 | cut -d: -f2)
done
echo "Device $NOVA_NIC found. Wait until ready."
sleep 3
# Setup a network route to insure we use the nova network.
#
echo "[INFO] Create default route for $NOVA_NIC. Gateway 10.0.0.1"
ip r r default via 10.0.0.1 dev $NOVA_NIC
# Shutdown eth0 since icps will fetch enabled enterface for streaming.
ip l set down dev eth0
This approach is obviously a poor solution since it has certain expectation of the network.
Another solution might be extend the docker driver to add a firewall rule which will masquerade requests on 169.254.169.254 with the actual nova network IP address
I third solution would need improvements in docker. If docker would have a network mode which allows to assign the IP from outside this issue would be solved. That of course is a just which must be accepted by the docker community.
Network Threads:
- https://groups.google.com/forum/#!topic/docker-dev/YfCeX8TBweA |
|
| 2013-12-09 22:36:27 |
Daniel Kuffner |
description |
I was playing around with cloud-init. I wanted to use cloud init as substitute for the missing environment variables feature.
The basic idea is to define variables in the user data and inject the them before starting the service.
Following:
docker run -e "MY_Variable=MyValue" -d centos
would look like in openstack:
nova boot --image centos:latest --user-data "MY_Variable=MyValue" myinstance
Unfortunately does the metadata service not work inside of a docker container. After some testing I figured out that the reason for that is that a container uses as default gateway the docker network (docker ip address). The metadata service simple rejects the call since the IP address of docker container is not associated with the nova instance.
Note: The metadata service itself can be accessed (http://169.254.169.254) but it is not possible to access the actual data (http://169.254.169.254/2009-04-04 - Status 404)
I was able to work around the issue by simply changing the route inside the container:
# Hack: In order to receive data from the metadata service we must make sure we resolve the data via our nova network.
#
# A docker container in openstack has two NICs.
# - eth0 has a IP address on the docker0 bridge which is usually an e.g. 172.0.0.0 IP address.
# - pvnetXXXX is a IP address assigned by nova.
#
# Extract the NIC name of the nova network.
#
NOVA_NIC=$(ip a | grep pvnet | head -n 1 | cut -d: -f2)
while [ "$NOVA_NIC" == "" ] ; do
echo "Find nova NIC..."
sleep 1
NOVA_NIC=$(ip a | grep pvnet | head -n 1 | cut -d: -f2)
done
echo "Device $NOVA_NIC found. Wait until ready."
sleep 3
# Setup a network route to insure we use the nova network.
#
echo "[INFO] Create default route for $NOVA_NIC. Gateway 10.0.0.1"
ip r r default via 10.0.0.1 dev $NOVA_NIC
# Shutdown eth0 since icps will fetch enabled enterface for streaming.
ip l set down dev eth0
This approach is obviously a poor solution since it has certain expectation of the network.
Another solution might be extend the docker driver to add a firewall rule which will masquerade requests on 169.254.169.254 with the actual nova network IP address
I third solution would need improvements in docker. If docker would have a network mode which allows to assign the IP from outside this issue would be solved. That of course is a just which must be accepted by the docker community.
Network Threads:
- https://groups.google.com/forum/#!topic/docker-dev/YfCeX8TBweA |
I was playing around with cloud-init. I wanted to use cloud init as substitute for the missing environment variables feature.
The basic idea is to define variables in the user data and inject the them before starting the service.
Following:
docker run -e "MY_Variable=MyValue" -d centos
would look like in openstack:
nova boot --image centos:latest --user-data "MY_Variable=MyValue" myinstance
Unfortunately does the metadata service not work inside of a docker container. After some testing I figured out that the reason for that is that a container uses as default gateway the docker network (docker ip address). The metadata service simple rejects the call since the IP address of docker container is not associated with the nova instance.
Note: The metadata service itself can be accessed (http://169.254.169.254) but it is not possible to access the actual data (http://169.254.169.254/2009-04-04 - Status 404)
I was able to work around the issue by simply changing the route inside the container:
# Hack: In order to receive data from the metadata service we must make sure we resolve the data via our nova network.
#
# A docker container in openstack has two NICs.
# - eth0 has a IP address on the docker0 bridge which is usually an e.g. 172.0.0.0 IP address.
# - pvnetXXXX is a IP address assigned by nova.
#
# Extract the NIC name of the nova network.
#
NOVA_NIC=$(ip a | grep pvnet | head -n 1 | cut -d: -f2)
while [ "$NOVA_NIC" == "" ] ; do
echo "Find nova NIC..."
sleep 1
NOVA_NIC=$(ip a | grep pvnet | head -n 1 | cut -d: -f2)
done
echo "Device $NOVA_NIC found. Wait until ready."
sleep 3
# Setup a network route to insure we use the nova network.
#
echo "[INFO] Create default route for $NOVA_NIC. Gateway 10.0.0.1"
ip r r default via 10.0.0.1 dev $NOVA_NIC
# Shutdown eth0 since icps will fetch enabled enterface for streaming.
ip l set down dev eth0
This approach is obviously a poor solution since it has certain expectation of the network.
Another solution might be extend the docker driver to add a firewall rule which will masquerade requests on 169.254.169.254 with the actual nova network IP address
I third solution would need improvements in docker. If docker would have a network mode which allows to assign the IP from outside this issue would be solved. That of course is a just which must be accepted by the docker community.
Network Threads:
- https://groups.google.com/forum/#!topic/docker-dev/YfCeX8TBweA
Simple script to test metadata inside the container:
#!/bin/bash
status=$(curl -I -s -o /dev/null -w "%{http_code}" http://169.254.169.254/2009-04-04)
while [ $status != '200' ]; do
echo "Cannot access metadata, status: '$status', try again..."
date # easier to see in docker logs that loop is still running.
status=$(curl -I -s -o /dev/null -w "%{http_code}" http://169.254.169.254/2009-04-04)
sleep 1
done
echo "Yes we got some user data:"
curl http://169.254.169.254/2009-04-04 |
|
| 2013-12-11 14:02:06 |
Daniel Kuffner |
summary |
Docker: Metadata service doesn't work |
Nova Docker: Metadata service doesn't work |
|
| 2013-12-15 12:22:23 |
Daniel Kuffner |
nova: status |
New |
In Progress |
|
| 2013-12-15 12:22:31 |
Daniel Kuffner |
nova: assignee |
|
Daniel Kuffner (daniel-kuffner) |
|
| 2014-02-07 20:03:46 |
OpenStack Infra |
nova: status |
In Progress |
Fix Committed |
|
| 2014-02-21 14:45:49 |
Russell Bryant |
nova: milestone |
|
icehouse-3 |
|
| 2014-03-05 13:19:59 |
Thierry Carrez |
nova: status |
Fix Committed |
Fix Released |
|
| 2014-04-17 09:11:33 |
Thierry Carrez |
nova: milestone |
icehouse-3 |
2014.1 |
|
