Nova ignores the --vlan_interface config option on compute nodes

[Philippe Berthault]

Hello,

I've installed OpenStack Nova Essex-3 on two hosts with Ubuntu 11.10. The first host runs all nova services (nova-all), so it acts as a controller + compute and the second host runs only nova-compute.

The controller has two NIC: eth1 for public network and eth2 for private network. So in the nova.conf of the controller, I've declared:
  --public_interface=eth1
  --vlan_interface=eth2

The compute node has also two NIC: eth0 for public network and eth1 for private network. So in the nova.conf of the compute node, I've declared:
  --public_interface=eth0
  --vlan_interface=eth1

When the scheduler starts a VM on the first host, all is OK. But when the scheduler starts a VM on the compute node, there is an error because OpenStack tries to use the eth2 interface instead of eth1. Nova ignores the --vlan_interface option of the nova.conf file on the compute node.

Before using Essex-3, I've tried with the Diablo release and there was the same problem. I've tried also with the --multi_host configuration option and the problem always exists.

Here the log of the compute node:

2012-02-17 13:45:51,645 DEBUG nova.compute.manager [-] instance network_info: |[[{u'bridge': u'br100', u'multi_host': False, u'bridge_interface': u'eth2', u'vlan': 100, u'id': 1, u'injected': False, u'cidr': u'192.168.0.0/24', u'cidr_v6': None}, {u'dns': [], u'should_create_bridge': True, u'net_uuid': u'f85f30c2-82d8-43e5-8b40-9fe33fdebdcf', u'vif_uuid': u'f75b076e-966d-42f9-b70d-589fe160e5f6', u'label': u'private', u'broadcast': u'192.168.0.255', u'ips': [{u'ip': u'192.168.0.7', u'netmask': u'255.255.255.0', u'enabled': u'1'}], u'mac': u'02:16:3e:49:df:61', u'rxtx_cap': 0, u'should_create_vlan': True, u'dhcp_server': u'192.168.0.1', u'gateway': u'192.168.0.1'}]]| from (pid=5363) debug /usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/log.py:175

2012-02-17 13:45:51,793 ERROR nova.compute.manager [-] Instance 49a3bccc-13f1-4f5e-9733-b552e578001c failed to spawn
(nova.compute.manager): TRACE: Traceback (most recent call last):
(nova.compute.manager): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/compute/manager.py", line 516, in _spawn
(nova.compute.manager): TRACE: network_info, block_device_info)
(nova.compute.manager): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/exception.py", line 126, in wrapped
(nova.compute.manager): TRACE: return f(*args, **kw)
(nova.compute.manager): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/virt/libvirt/connection.py", line 671, in spawn
(nova.compute.manager): TRACE: block_device_info=block_device_info)
(nova.compute.manager): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/virt/libvirt/connection.py", line 1206, in to_xml
(nova.compute.manager): TRACE: rescue, block_device_info)
(nova.compute.manager): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/virt/libvirt/connection.py", line 1093, in _prepare_xml_info
(nova.compute.manager): TRACE: nics.append(self.vif_driver.plug(instance, network, mapping))
(nova.compute.manager): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/virt/libvirt/vif.py", line 87, in plug
(nova.compute.manager): TRACE: network['bridge_interface'])
(nova.compute.manager): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/network/linux_net.py", line 953, in ensure_vlan_bridge
(nova.compute.manager): TRACE: bridge_interface, mac_address)
(nova.compute.manager): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/utils.py", line 844, in inner
(nova.compute.manager): TRACE: retval = f(*args, **kwargs)
(nova.compute.manager): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/network/linux_net.py", line 966, in ensure_vlan
(nova.compute.manager): TRACE: 'id', vlan_num, run_as_root=True)
(nova.compute.manager): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/network/linux_net.py", line 818, in _execute
(nova.compute.manager): TRACE: return utils.execute(*cmd, **kwargs)
(nova.compute.manager): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/utils.py", line 225, in execute
(nova.compute.manager): TRACE: cmd=' '.join(cmd))
(nova.compute.manager): TRACE: ProcessExecutionError: Unexpected error while running command.
(nova.compute.manager): TRACE: Command: sudo ip link add link eth2 name vlan100 type vlan id 100
(nova.compute.manager): TRACE: Exit code: 255
(nova.compute.manager): TRACE: Stdout: ''
(nova.compute.manager): TRACE: Stderr: 'Cannot find device "eth2"\n'

[Philippe Berthault (philippe-berthault-deactivatedaccount)]

To bypass the ignored vlan_interface option problem, I have doing the following actions:

  1) Add multi_host option on my configuration.
  2) Associate the compute node to my project with the "nova-manage network modify ..." command.
  3) Modify the networks table of nova database by setting the bridge_interface to "eth1" on this compute node.
  4) Start nova-network on the compute nodes.

Now, I'am able to start an instance on the compute node but there is another error.

On the controller, the euca-describe-instances command gives the status "running" on the instance and the "nova list" command gives the status "ACTIVE". But on the compute node, the instance has failed with the following error in the compute-node.log file:

Command: sudo qemu-nbd -c /dev/nbd15 /var/lib/nova/instances/instance-0000000b/disk
Exit code: 1
Stdout: ''
Stderr: "qemu-nbd: Could not access '/dev/nbd15': No such file or directory\n"

I've trying after stopping all nova services and libvirt-bin but this "nbd15" problem persists.

[Philippe Berthault (philippe-berthault-deactivatedaccount)]

I can ping an instance on the compute node but not ssh it. It seems that the problem does'nt come from "nbd15" error as reported in the previous comment but rather on a network problem because with the euca-get-console-output command, I've the following messages:

DataSourceEc2.py[WARNING]: waiting for metadata service at http://169.254.169.254/2009-04-04/meta-data/instance-id
DataSourceEc2.py[WARNING]: 16:46:45 [ 1/100]: url error [[Errno 111] Connection refused]
DataSourceEc2.py[WARNING]: 16:46:46 [ 2/100]: url error [[Errno 111] Connection refused]
etc..

From the compute node where is running the instance, a ping to 169.254.169.254 address is OK
and a nmap to the controller reports the following:
# nmap 172.31.70.89
  PORT STATE SERVICE
  22/tcp open ssh
  80/tcp open http
  3260/tcp open iscsi
  3306/tcp open mysql
  3333/tcp open dec-notes

The nova.conf file of the compute node is the following:

--dhcpbridge_flagfile=/etc/nova/nova.conf
--dhcpbridge=/usr/bin/nova-dhcpbridge
--logdir=/var/log/nova
--lock_path=/var/lock/nova
--state_path=/var/lib/nova
--verbose
--use_deprecated_auth
--s3_host=172.31.70.89
--rabbit_host=172.31.70.89
--cc_host=172.31.70.89
--ec2_host=172.31.70.89
--fixed_range=192.168.0.0/16
--network_size=8
--routing_source_ip=172.31.70.89
--sql_connection=mysql://novadbadmin:novasecret@172.31.70.89/nova
--glance_api_servers=172.31.70.89:9292
--image_service=nova.image.glance.GlanceImageService
--public_interface=eth0
--vlan_interface=eth1
--connection_type=libvirt
--multi_host

172.31.70.89 is the address of the controller where all services are running.
The nova.conf file of this controller is identical to the one of the compute node excepted
--public_interface=eth1 and --vlan_interface=eth2.

Have you an idea of what is the problem ?
Thanks.