On quantum gate VM cannot get dhcp lease

[Andrea Frittoli]

I'm working on enabling ssh tests back in tempest https://review.openstack.org/#/c/22415/

The test fails in the quantum gate. I did some digging, and it seems to me that the quantum setup in the devstack gate is broken.
If I boot a VM I can see from the console log that the VM is failing to get a DHCP lease, which causes in turn metadata to fail, and VM networking to be broken.

Starting network...
udhcpc (v1.20.1) started
Sending discover...
Sending discover...
Sending discover...
No lease, failing
WARN: /etc/rc3.d/S40network failed
cirrosds 'net' up at 195.26
checking http://169.254.169.254/20090404/instanceid
failed 1/20: up 196.16. request failed
failed 2/20: up 198.79. request failed

jenkins@wip-devstack-1362643791:~/workspace$ nova list
+--------------------------------------+---------------+--------+-------------------+
| ID | Name | Status | Networks |
+--------------------------------------+---------------+--------+-------------------+
| 4741860b-afa8-4e70-935b-d3b4624796e1 | quantumvm0001 | ACTIVE | nova=172.24.4.228 |
+--------------------------------------+---------------+--------+-------------------+

This is the localrc generated by gate scripts:
$ cat /opt/stack/new/devstack/localrc
Q_USE_DEBUG_COMMAND=True
NETWORK_GATEWAY=10.1.0.1
Q_USE_DEBUG_COMMAND=True
NETWORK_GATEWAY=10.1.0.1
DEST=/opt/stack/new
ACTIVE_TIMEOUT=90
BOOT_TIMEOUT=90
ASSOCIATE_TIMEOUT=60
TERMINATE_TIMEOUT=60
MYSQL_PASSWORD=secret
DATABASE_PASSWORD=secret
RABBIT_PASSWORD=secret
ADMIN_PASSWORD=secret
SERVICE_PASSWORD=secret
SERVICE_TOKEN=111222333444
SWIFT_HASH=1234123412341234
ROOTSLEEP=0
ERROR_ON_CLONE=True
ENABLED_SERVICES=g-api,g-reg,key,n-api,n-crt,n-obj,n-cpu,n-sch,horizon,mysql,rabbit,sysstat,swift,cinder,c-api,c-vol,c-sch,n-cond,quantum,q-svc,q-agt,q-dhcp,q-l3,q-meta
SKIP_EXERCISES=boot_from_volume,client-env
SERVICE_HOST=127.0.0.1
# Screen console logs will capture service logs.
SYSLOG=False
SCREEN_LOGDIR=/opt/stack/new/screen-logs
LOGFILE=/opt/stack/new/devstacklog.txt
VERBOSE=True
FIXED_RANGE=10.1.0.0/24
FIXED_NETWORK_SIZE=256
VIRT_DRIVER=libvirt
SWIFT_REPLICAS=1
SCREEN_DEV=False
LOG_COLOR=False
export OS_NO_CACHE=True
CINDER_SECURE_DELETE=False
API_RATE_LIMIT=False
VOLUME_BACKING_FILE_SIZE=10G

[Andrea Frittoli (andrea-frittoli)]

I checked the various quantum services / resources.
Everything looks ok except for the two DHCP agent which report XXX

jenkins@wip-devstack-1362643791:~/workspace$ quantum net-list
+--------------------------------------+---------+------------------------------------------------------+
| id | name | subnets |
+--------------------------------------+---------+------------------------------------------------------+
| bd3653b3-e032-4caf-a9b6-3ac1ceb167ef | private | 37bf52d2-2854-49b9-903d-ccbe187d2174 10.1.0.0/24 |
| fe6ba3ff-f527-47b7-b4fa-75212b2f8b6f | nova | 0adef22a-97a4-4fd0-a6b0-4fa0d59aa3a8 172.24.4.224/28 |
+--------------------------------------+---------+------------------------------------------------------+

jenkins@wip-devstack-1362643791:~/workspace$ quantum subnet-list
+--------------------------------------+------+-----------------+--------------------------------------------------+
| id | name | cidr | allocation_pools |
+--------------------------------------+------+-----------------+--------------------------------------------------+
| 0adef22a-97a4-4fd0-a6b0-4fa0d59aa3a8 | | 172.24.4.224/28 | {"start": "172.24.4.226", "end": "172.24.4.238"} |
| 37bf52d2-2854-49b9-903d-ccbe187d2174 | | 10.1.0.0/24 | {"start": "10.1.0.2", "end": "10.1.0.254"} |
+--------------------------------------+------+-----------------+--------------------------------------------------+

(quantum) net-external-list
+--------------------------------------+------+------------------------------------------------------+
| id | name | subnets |
+--------------------------------------+------+------------------------------------------------------+
| fe6ba3ff-f527-47b7-b4fa-75212b2f8b6f | nova | 0adef22a-97a4-4fd0-a6b0-4fa0d59aa3a8 172.24.4.224/28 |
+--------------------------------------+------+------------------------------------------------------+

(quantum) router-list
+--------------------------------------+---------+--------------------------------------------------------+
| id | name | external_gateway_info |
+--------------------------------------+---------+--------------------------------------------------------+
| 81aece45-0919-40a3-9de2-8dccd566b9fb | router1 | {"network_id": "fe6ba3ff-f527-47b7-b4fa-75212b2f8b6f"} |
+--------------------------------------+---------+--------------------------------------------------------+

(quantum) router-port-list router1
+--------------------------------------+------+-------------------+-------------------------------------------------------------------------------------+
| id | name | mac_address | fixed_ips |
+--------------------------------------+------+-------------------+-------------------------------------------------------------------------------------+
| 5aaeb9...

[Andrea Frittoli (andrea-frittoli)]

It seems unusual that the VM is attached to a port on the external network directly.
When I create a new VM is see two new ports in the quantum port-list, one on private and one on external (nova) network.

I tried booting a VM specifying --nic net-id=[net id], but nova complains with 404 network not found, which is odd, because I can do nova net-list, and it that case the proxying to the quantum API is working fine.

[Andrea Frittoli (andrea-frittoli)]

jenkins@wip-devstack-1362643791:~/workspace$ virsh dumpxml instance-00000005 | grep tap
      <target dev='tap20281990-f1'/>

jenkins@wip-devstack-1362643791:~/workspace$ ip a | grep tap20281990-f1
30: tap20281990-f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master qbr20281990-f1 state UNKNOWN qlen 500

[Andrea Frittoli (andrea-frittoli)]

I see the following error messages in q-dhcp log:
http://logs.openstack.org/22415/16/check/gate-tempest-devstack-vm-quantum/12557/logs/screen-q-dhcp.txt.gz

2013-03-06 22:31:47.287 20495 DEBUG quantum.agent.linux.utils [-] Running command: ['sudo', '/usr/local/bin/quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ip', 'netns', 'exec', 'qdhcp-1353ae8e-9f40-4555-a389-ca2495493250', 'ip', '-o', 'link', 'show', 'tapc4349aa6-21'] execute /opt/stack/new/quantum/quantum/agent/linux/utils.py:42
2013-03-06 22:31:47.510 20495 DEBUG quantum.agent.linux.utils [-]
Command: ['sudo', '/usr/local/bin/quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ip', 'netns', 'exec', 'qdhcp-1353ae8e-9f40-4555-a389-ca2495493250', 'ip', '-o', 'link', 'show', 'tapc4349aa6-21']
Exit code: 1

I wonder whether this is related also to these two:
https://bugs.launchpad.net/quantum/+bug/1080846
https://bugs.launchpad.net/nova/+bug/1152893

[Nachi Ueno (nati-ueno)]

Hi Andrea

excise.sh looks success, so this means each VM can get ip from DHCP.
http://logs.openstack.org/22415/17/check/gate-tempest-devstack-vm-quantum/13199/console.html.gz

Note that, dhcp is disabled for external network, so VM on the external network won't get ip from dhcp,
so your log looks correct.

Unlike, nova-network, quantum provides truly isolated network. So we can't ping from the host.
We need some way to get in to the quantum network.
One way is quantum-debug command (see https://github.com/openstack-dev/devstack/blob/master/exercises/quantum-adv-test.sh#L238)

[Clark Boylan (cboylan)]

Moving to Quantum's bug tracker as this appears to be an issue that will need coordination between the QA and Quantum teams (based on Nachi's comment). The Infra team does not have the expertise to solve this problem. Ping us if we need to make changes to the infrastructure to enable things though.