instack: Failed to set DHCP BOOT options for any port

[Dan Prince]

When trying to deploy full disk images with instack (to real baremetal instances) I'm hitting the following exception:

Sep 12 13:05:22 instack.localdomain ironic-conductor[13407]: 2015-09-12 13:05:22.935 13407 DEBUG ironic.drivers.modules.pxe [-] Fetching necessary kernel and ramdisk for node 8743d750-7de0-4163-867f-b667d5eb31f0 _cache_ramdisk_kernel /usr/lib/python2.7/site-packages/ironic/drivers/modules/pxe.py:365
Sep 12 13:05:22 instack.localdomain ironic-conductor[13407]: 2015-09-12 13:05:22.964 13407 ERROR ironic.dhcp.neutron [-] Failed to update Neutron port 6589fdb9-2570-4968-a252-ba0454362373.
Sep 12 13:05:22 instack.localdomain ironic-conductor[13407]: 2015-09-12 13:05:22.964 13407 ERROR ironic.dhcp.neutron Traceback (most recent call last):
Sep 12 13:05:22 instack.localdomain ironic-conductor[13407]: 2015-09-12 13:05:22.964 13407 ERROR ironic.dhcp.neutron File "/usr/lib/python2.7/site-packages/ironic/dhcp/neutron.py", line 126, in update_port_dhcp_opts
Sep 12 13:05:22 instack.localdomain ironic-conductor[13407]: 2015-09-12 13:05:22.964 13407 ERROR ironic.dhcp.neutron _build_client(token).update_port(port_id, port_req_body)
Sep 12 13:05:22 instack.localdomain ironic-conductor[13407]: 2015-09-12 13:05:22.964 13407 ERROR ironic.dhcp.neutron File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 102, in with_params
Sep 12 13:05:22 instack.localdomain ironic-conductor[13407]: 2015-09-12 13:05:22.964 13407 ERROR ironic.dhcp.neutron ret = self.function(instance, *args, **kwargs)
Sep 12 13:05:22 instack.localdomain ironic-conductor[13407]: 2015-09-12 13:05:22.964 13407 ERROR ironic.dhcp.neutron File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 554, in update_port
Sep 12 13:05:22 instack.localdomain ironic-conductor[13407]: 2015-09-12 13:05:22.964 13407 ERROR ironic.dhcp.neutron return self.put(self.port_path % (port), body=body)
Sep 12 13:05:22 instack.localdomain ironic-conductor[13407]: 2015-09-12 13:05:22.964 13407 ERROR ironic.dhcp.neutron File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 302, in put
Sep 12 13:05:22 instack.localdomain ironic-conductor[13407]: 2015-09-12 13:05:22.964 13407 ERROR ironic.dhcp.neutron headers=headers, params=params)
Sep 12 13:05:22 instack.localdomain ironic-conductor[13407]: 2015-09-12 13:05:22.964 13407 ERROR ironic.dhcp.neutron File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 270, in retry_request
Sep 12 13:05:22 instack.localdomain ironic-conductor[13407]: 2015-09-12 13:05:22.964 13407 ERROR ironic.dhcp.neutron headers=headers, params=params)
Sep 12 13:05:22 instack.localdomain ironic-conductor[13407]: 2015-09-12 13:05:22.964 13407 ERROR ironic.dhcp.neutron File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 200, in do_request
Sep 12 13:05:22 instack.localdomain ironic-conductor[13407]: 2015-09-12 13:05:22.964 13407 ERROR ironic.dhcp.neutron content_type=self.content_type())
Sep 12 13:05:22 instack.localdomain ironic-conductor[13407]: 2015-09-12 13:05:22.964 13407 ERROR ironic.dhcp.neutron File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 158, in do_request
Sep 12 13:05:22 instack.localdomain ironic-conductor[13407]: 2015-09-12 13:05:22.964 13407 ERROR ironic.dhcp.neutron self.authenticate_and_fetch_endpoint_url()
Sep 12 13:05:22 instack.localdomain ironic-conductor[13407]: 2015-09-12 13:05:22.964 13407 ERROR ironic.dhcp.neutron File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 123, in authenticate_and_fetch_endpoint_url
Sep 12 13:05:22 instack.localdomain ironic-conductor[13407]: 2015-09-12 13:05:22.964 13407 ERROR ironic.dhcp.neutron self.authenticate()
Sep 12 13:05:22 instack.localdomain ironic-conductor[13407]: 2015-09-12 13:05:22.964 13407 ERROR ironic.dhcp.neutron File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 237, in authenticate
Sep 12 13:05:22 instack.localdomain ironic-conductor[13407]: 2015-09-12 13:05:22.964 13407 ERROR ironic.dhcp.neutron self._authenticate_keystone()
Sep 12 13:05:22 instack.localdomain ironic-conductor[13407]: 2015-09-12 13:05:22.964 13407 ERROR ironic.dhcp.neutron File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 218, in _authenticate_keystone
Sep 12 13:05:22 instack.localdomain ironic-conductor[13407]: 2015-09-12 13:05:22.964 13407 ERROR ironic.dhcp.neutron raise exceptions.Unauthorized(message=resp_body)
Sep 12 13:05:22 instack.localdomain ironic-conductor[13407]: 2015-09-12 13:05:22.964 13407 ERROR ironic.dhcp.neutron Unauthorized: {"error": {"message": "The resource could not be found.", "code": 404, "title": "Not Found"}}
Sep 12 13:05:22 instack.localdomain ironic-conductor[13407]: 2015-09-12 13:05:22.964 13407 ERROR ironic.dhcp.neutron
Sep 12 13:05:22 instack.localdomain ironic-conductor[13407]: 2015-09-12 13:05:22.977 13407 ERROR ironic.drivers.modules.iscsi_deploy [-] Deploy failed for instance b4dde407-4678-4fec-913b-04752699b9d8. Error: Failed to set DHCP BOOT options for any port on node 8743d750-7de0-4163-867f-b667d5eb31f0.

[Dan Prince (dan-prince)]

Looks like (one) of the issues here is related to a keystone auth failure when Ironic tries to setup DHCP options.

The default Ironic auth_uri looks like this:

auth_uri=http://172.19.0.3:5000/

----

The current TripleO environment however doesn't yet configure keystone V3 so it should actually be this:

auth_uri=http://172.19.0.3:5000/v2.0/
identity_uri=http://172.19.0.3:35357/

We should first push a new puppet-ironic patch which supports the new identity_uri setting, and then update instack to use it.

----

After making these changes locally I was able to successfully boot a full disk image w/ Instack.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/223126

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/223126
Committed: https://git.openstack.org/cgit/openstack/instack-undercloud/commit/?id=d7190272c0c356a7c59233e2c8592887de5958eb
Submitter: Jenkins
Branch: master

commit d7190272c0c356a7c59233e2c8592887de5958eb
Author: Dan Prince <email address hidden>
Date: Mon Sep 14 10:31:00 2015 -0400

    Correct ironic auth settings

    The existing ironic.conf uses an unversioned keystone auth_url
    setting. As the keystone API now defaults to v3 this can
    cause provisioning failures if Ironic needs to communicate
    with other services (neutron). This is because we haven't yet
    updated TripleO to properly support configuration with
    keystone V3.

    We also need to set the ironic neutron_url because
    if we don't puppet-ironic will use the auth_protocol and
    auth_host parameters to construct it incorrectly if
    we don't (since we no longer set those parameters).

    Change-Id: Ief7fc3dbbd8b8c33ea3ec7bd127231698af27a7f
    Depends-On: Ibcc63d82b6e7b9de6ef12209a43f39a4cdecb5a2
    Closes-bug: #1495090