CI failure "unable to enable dhcp for"

[Robert Collins]

http://logstash.openstack.org/#eyJmaWVsZHMiOltdLCJzZWFyY2giOiJcIlVuYWJsZSB0byBlbmFibGUgZGhjcCBmb3JcIiIsInRpbWVmcmFtZSI6ImN1c3RvbSIsImdyYXBobW9kZSI6ImNvdW50Iiwib2Zmc2V0IjowLCJ0aW1lIjp7ImZyb20iOiIyMDE0LTA1LTI1VDA0OjE0OjEzKzAwOjAwIiwidG8iOiIyMDE0LTA1LTI2VDA0OjE0OjEzKzAwOjAwIiwidXNlcl9pbnRlcnZhbCI6IjM2MDAwMDAifSwic3RhbXAiOjE0MDEwNzc3OTU4MTJ9

http://logs.openstack.org/90/90890/12/check/check-tempest-dsvm-virtual-ironic/6bae7fc/console.html

2014-05-26 00:27:32.990 | Log File Has Errors: q-dhcp
2014-05-26 00:27:32.991 | *** Not Whitelisted *** 2014-05-26 00:21:58.575 12112 ERROR neutron.agent.dhcp_agent [req-8d469c27-df65-402e-8f1f-b459380e6d18 None] Unable to enable dhcp for 26379fb2-52f6-4ccb-8d74-d284a2969aa0.
2014-05-26 00:27:32.992 | *** Not Whitelisted *** 2014-05-26 00:21:58.575 12112 TRACE neutron.agent.dhcp_agent Command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qdhcp-26379fb2-52f6-4ccb-8d74-d284a2969aa0', 'env', 'NEUTRON_NETWORK_ID=26379fb2-52f6-4ccb-8d74-d284a2969aa0', 'dnsmasq', '--no-hosts', '--no-resolv', '--strict-order', '--bind-interfaces', '--interface=tape2fb3516-9d', '--except-interface=lo', '--pid-file=/opt/stack/data/neutron/dhcp/26379fb2-52f6-4ccb-8d74-d284a2969aa0/pid', '--dhcp-hostsfile=/opt/stack/data/neutron/dhcp/26379fb2-52f6-4ccb-8d74-d284a2969aa0/host', '--addn-hosts=/opt/stack/data/neutron/dhcp/26379fb2-52f6-4ccb-8d74-d284a2969aa0/addn_hosts', '--dhcp-optsfile=/opt/stack/data/neutron/dhcp/26379fb2-52f6-4ccb-8d74-d284a2969aa0/opts', '--leasefile-ro', '--dhcp-range=set:tag0,2003::,static,86400s', '--dhcp-lease-max=16777216', '--conf-file=', '--domain=openstacklocal']
2014-05-26 00:27:32.993 | *** Not Whitelisted *** 2014-05-26 00:22:02.276 12112 ERROR neutron.agent.dhcp_agent [req-813bbfd7-bd5c-4926-aad8-5d748032dbaa None] Unable to enable dhcp for 8315194f-b044-4238-a545-e80c9482daed.
2014-05-26 00:27:32.994 | *** Not Whitelisted *** 2014-05-26 00:22:02.276 12112 TRACE neutron.agent.dhcp_agent Command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qdhcp-8315194f-b044-4238-a545-e80c9482daed', 'env', 'NEUTRON_NETWORK_ID=8315194f-b044-4238-a545-e80c9482daed', 'dnsmasq', '--no-hosts', '--no-resolv', '--strict-order', '--bind-interfaces', '--interface=tap1c14cc1e-0b', '--except-interface=lo', '--pid-file=/opt/stack/data/neutron/dhcp/8315194f-b044-4238-a545-e80c9482daed/pid', '--dhcp-hostsfile=/opt/stack/data/neutron/dhcp/8315194f-b044-4238-a545-e80c9482daed/host', '--addn-hosts=/opt/stack/data/neutron/dhcp/8315194f-b044-4238-a545-e80c9482daed/addn_hosts', '--dhcp-optsfile=/opt/stack/data/neutron/dhcp/8315194f-b044-4238-a545-e80c9482daed/opts', '--leasefile-ro', '--dhcp-range=set:tag0,2003::,static,86400s', '--dhcp-lease-max=16777216', '--conf-file=', '--domain=openstacklocal']

[Kevin Benton (kevinbenton)]

I think this might be related to this bug: https://bugs.launchpad.net/neutron/+bug/1233339

[aeva black (tenbrae)]

Deeper and more specific logstash search:

http://logstash.openstack.org/#eyJmaWVsZHMiOlsiYnVpbGRfc3RhdHVzIiwiYnVpbGRfdXVpZCJdLCJzZWFyY2giOiJidWlsZF9uYW1lOlwiY2hlY2stdGVtcGVzdC1kc3ZtLXZpcnR1YWwtaXJvbmljXCIgQU5EIGZpbGVuYW1lOlwiY29uc29sZS5odG1sXCIgQU5EIFwidW5hYmxlIHRvIGVuYWJsZSBkaGNwIGZvclwiIiwidGltZWZyYW1lIjoiY3VzdG9tIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJvZmZzZXQiOjAsInRpbWUiOnsiZnJvbSI6IjIwMTQtMDUtMjJUMDQ6MTQ6MTMrMDA6MDAiLCJ0byI6IjIwMTQtMDUtMjZUMDQ6MTQ6MTMrMDA6MDAiLCJ1c2VyX2ludGVydmFsIjoiMzYwMDAwMCJ9LCJzdGFtcCI6MTQwMTEyMTk3NjIwN30=

(search for... build_name:"check-tempest-dsvm-virtual-ironic" AND filename:"console.html" AND "unable to enable dhcp for")

There are significantly more SUCCESS than FAIL over the last week which match that search. I don't believe this is causing the test to fail.

Looking at the linked console.html file for the failed test, the actual cause is pointed out by this line:

2014-05-26 00:27:32.994 | Logs have errors
2014-05-26 00:27:32.995 | Log: ir-api not allowed to have ERRORS or TRACES

We have a bug open for that, here: https://bugs.launchpad.net/ironic/+bug/1321494