Cleaning failed when setting ironic provision state to provide

Bug #1624972 reported by Tao Li
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ironic
Expired
Undecided
Unassigned

Bug Description

when i set node provision to provide, the provision_state is changed to clean failed.
check the log as follows. I am using mitaka stable release。

2016-09-19 11:06:49.353 31136 DEBUG ironic.common.pxe_utils [req-f27d3efe-1512-4ec6-ba61-8cdf71f4027b - - - - -] Cleaning up PXE config for node 942c8cb4-5a90-4116-9595-6ef0baed3b5b clean_up_pxe_config /usr/lib/python2.7/site-packages/ironic/common/pxe_utils.py:259
2016-09-19 11:06:49.354 31136 DEBUG ironic.drivers.modules.deploy_utils [req-f27d3efe-1512-4ec6-ba61-8cdf71f4027b - - - - -] Deploy boot mode is None for 942c8cb4-5a90-4116-9595-6ef0baed3b5b. get_boot_mode_for_deploy /usr/lib/python2.7/site-packages/ironic/drivers/modules/deploy_utils.py:796
2016-09-19 11:06:49.355 31136 DEBUG oslo_concurrency.lockutils [req-f27d3efe-1512-4ec6-ba61-8cdf71f4027b - - - - -] Lock "master_image" acquired by "ironic.drivers.modules.image_cache.clean_up" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2016-09-19 11:06:49.355 31136 DEBUG ironic.drivers.modules.image_cache [req-f27d3efe-1512-4ec6-ba61-8cdf71f4027b - - - - -] Starting clean up for master image cache /tftpboot/master_images clean_up /usr/lib/python2.7/site-packages/ironic/drivers/modules/image_cache.py:196
2016-09-19 11:06:49.356 31136 DEBUG oslo_concurrency.lockutils [req-f27d3efe-1512-4ec6-ba61-8cdf71f4027b - - - - -] Lock "master_image" released by "ironic.drivers.modules.image_cache.clean_up" :: held 0.001s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2016-09-19 11:06:49.365 31136 ERROR ironic.conductor.utils [req-f27d3efe-1512-4ec6-ba61-8cdf71f4027b - - - - -] (u'Failed to tear down cleaning on node %(uuid)s, reason: %(err)s', {'uuid': '942c8cb4-5a90-4116-9595-6ef0baed3b5b', 'err': Unauthorized()})
2016-09-19 11:06:49.365 31136 ERROR ironic.conductor.utils Traceback (most recent call last):
2016-09-19 11:06:49.365 31136 ERROR ironic.conductor.utils File "/usr/lib/python2.7/site-packages/ironic/conductor/utils.py", line 223, in cleaning_error_handler
2016-09-19 11:06:49.365 31136 ERROR ironic.conductor.utils task.driver.deploy.tear_down_cleaning(task)
2016-09-19 11:06:49.365 31136 ERROR ironic.conductor.utils File "/usr/lib/python2.7/site-packages/ironic/drivers/modules/iscsi_deploy.py", line 698, in tear_down_cleaning
2016-09-19 11:06:49.365 31136 ERROR ironic.conductor.utils task, manage_boot=True)
2016-09-19 11:06:49.365 31136 ERROR ironic.conductor.utils File "/usr/lib/python2.7/site-packages/ironic/drivers/modules/deploy_utils.py", line 1059, in tear_down_inband_cleaning
2016-09-19 11:06:49.365 31136 ERROR ironic.conductor.utils tear_down_cleaning_ports(task)
2016-09-19 11:06:49.365 31136 ERROR ironic.conductor.utils File "/usr/lib/python2.7/site-packages/ironic/drivers/modules/deploy_utils.py", line 959, in tear_down_cleaning_ports
2016-09-19 11:06:49.365 31136 ERROR ironic.conductor.utils provider.provider.delete_cleaning_ports(task)
2016-09-19 11:06:49.365 31136 ERROR ironic.conductor.utils File "/usr/lib/python2.7/site-packages/ironic/dhcp/neutron.py", line 368, in delete_cleaning_ports
2016-09-19 11:06:49.365 31136 ERROR ironic.conductor.utils ports = neutron_client.list_ports(**params)
2016-09-19 11:06:49.365 31136 ERROR ironic.conductor.utils File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 97, in with_params
2016-09-19 11:06:49.365 31136 ERROR ironic.conductor.utils ret = self.function(instance, *args, **kwargs)
2016-09-19 11:06:49.365 31136 ERROR ironic.conductor.utils File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 644, in list_ports
2016-09-19 11:06:49.365 31136 ERROR ironic.conductor.utils **_params)
2016-09-19 11:06:49.365 31136 ERROR ironic.conductor.utils File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 373, in list
2016-09-19 11:06:49.365 31136 ERROR ironic.conductor.utils for r in self._pagination(collection, path, **params):
2016-09-19 11:06:49.365 31136 ERROR ironic.conductor.utils File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 388, in _pagination
2016-09-19 11:06:49.365 31136 ERROR ironic.conductor.utils res = self.get(path, params=params)
2016-09-19 11:06:49.365 31136 ERROR ironic.conductor.utils File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 358, in get
2016-09-19 11:06:49.365 31136 ERROR ironic.conductor.utils headers=headers, params=params)
2016-09-19 11:06:49.365 31136 ERROR ironic.conductor.utils File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 335, in retry_request
2016-09-19 11:06:49.365 31136 ERROR ironic.conductor.utils headers=headers, params=params)
2016-09-19 11:06:49.365 31136 ERROR ironic.conductor.utils File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 286, in do_request
2016-09-19 11:06:49.365 31136 ERROR ironic.conductor.utils resp, replybody = self.httpclient.do_request(action, method, body=body)
2016-09-19 11:06:49.365 31136 ERROR ironic.conductor.utils File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 177, in do_request
2016-09-19 11:06:49.365 31136 ERROR ironic.conductor.utils self.endpoint_url + url, method, **kwargs)
2016-09-19 11:06:49.365 31136 ERROR ironic.conductor.utils File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 109, in _cs_request
2016-09-19 11:06:49.365 31136 ERROR ironic.conductor.utils raise exceptions.Unauthorized(message=body)
2016-09-19 11:06:49.365 31136 ERROR ironic.conductor.utils Unauthorized: Authentication required
2016-09-19 11:06:49.365 31136 ERROR ironic.conductor.utils
2016-09-19 11:06:49.367 31136 DEBUG ironic.common.states [req-f27d3efe-1512-4ec6-ba61-8cdf71f4027b - - - - -] Exiting old state 'cleaning' in response to event 'fail' on_exit /usr/lib/python2.7/site-packages/ironic/common/states.py:203
2016-09-19 11:06:49.367 31136 DEBUG ironic.common.states [req-f27d3efe-1512-4ec6-ba61-8cdf71f4027b - - - - -] Entering new state 'clean failed' in response to event 'fail' on_enter /usr/lib/python2.7/site-packages/ironic/common/states.py:209

The ironic configuration is as follows.

[neutron]

#
# Options defined in ironic.dhcp.neutron
#

# URL for connecting to neutron. (string value)
url=http://$my_ip:9696

# Timeout value for connecting to neutron in seconds. (integer
# value)
#url_timeout=30

# Client retries in the case of a failed request. (integer
# value)
#retries=3

# Default authentication strategy to use when connecting to
# neutron. Running neutron in noauth mode (related to but not
# affected by this setting) is insecure and should only be
# used for testing. (string value)
# Possible values: keystone, noauth
auth_strategy=noauth

# UUID of the network to create Neutron ports on, when booting
# to a ramdisk for cleaning using Neutron DHCP. (string value)
cleaning_network_uuid=6b65062f-534f-4131-8693-3a719997fe92

[keystone_authtoken]

#
# Options defined in keystonemiddleware.auth_token
#

# Complete public Identity API endpoint. (string value)
auth_uri=http://192.168.2.203:5000/v3

# API version of the admin Identity API endpoint. (string
# value)
auth_version=3

When I change to neutron auth_strategy=keystone, then also raise an exception:

2016-09-19 10:19:03.943 20637 ERROR ironic.conductor.utils task.driver.deploy.tear_down_cleaning(task)
2016-09-19 10:19:03.943 20637 ERROR ironic.conductor.utils File "/usr/lib/python2.7/site-packages/ironic/drivers/modules/iscsi_deploy.py", line 698, in tear_down_cleaning
2016-09-19 10:19:03.943 20637 ERROR ironic.conductor.utils task, manage_boot=True)
2016-09-19 10:19:03.943 20637 ERROR ironic.conductor.utils File "/usr/lib/python2.7/site-packages/ironic/drivers/modules/deploy_utils.py", line 1059, in tear_down_inband_cleaning
2016-09-19 10:19:03.943 20637 ERROR ironic.conductor.utils tear_down_cleaning_ports(task)
2016-09-19 10:19:03.943 20637 ERROR ironic.conductor.utils File "/usr/lib/python2.7/site-packages/ironic/drivers/modules/deploy_utils.py", line 959, in tear_down_cleaning_ports
2016-09-19 10:19:03.943 20637 ERROR ironic.conductor.utils provider.provider.delete_cleaning_ports(task)
2016-09-19 10:19:03.943 20637 ERROR ironic.conductor.utils File "/usr/lib/python2.7/site-packages/ironic/dhcp/neutron.py", line 368, in delete_cleaning_ports
2016-09-19 10:19:03.943 20637 ERROR ironic.conductor.utils ports = neutron_client.list_ports(**params)
2016-09-19 10:19:03.943 20637 ERROR ironic.conductor.utils File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 97, in with_params
2016-09-19 10:19:03.943 20637 ERROR ironic.conductor.utils ret = self.function(instance, *args, **kwargs)
2016-09-19 10:19:03.943 20637 ERROR ironic.conductor.utils File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 644, in list_ports
2016-09-19 10:19:03.943 20637 ERROR ironic.conductor.utils **_params)
2016-09-19 10:19:03.943 20637 ERROR ironic.conductor.utils File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 373, in list
2016-09-19 10:19:03.943 20637 ERROR ironic.conductor.utils for r in self._pagination(collection, path, **params):
2016-09-19 10:19:03.943 20637 ERROR ironic.conductor.utils File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 388, in _pagination
2016-09-19 10:19:03.943 20637 ERROR ironic.conductor.utils res = self.get(path, params=params)
2016-09-19 10:19:03.943 20637 ERROR ironic.conductor.utils File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 358, in get
2016-09-19 10:19:03.943 20637 ERROR ironic.conductor.utils headers=headers, params=params)
2016-09-19 10:19:03.943 20637 ERROR ironic.conductor.utils File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 335, in retry_request
2016-09-19 10:19:03.943 20637 ERROR ironic.conductor.utils headers=headers, params=params)
2016-09-19 10:19:03.943 20637 ERROR ironic.conductor.utils File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 286, in do_request
2016-09-19 10:19:03.943 20637 ERROR ironic.conductor.utils resp, replybody = self.httpclient.do_request(action, method, body=body)
2016-09-19 10:19:03.943 20637 ERROR ironic.conductor.utils File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 158, in do_request
2016-09-19 10:19:03.943 20637 ERROR ironic.conductor.utils self.authenticate_and_fetch_endpoint_url()
2016-09-19 10:19:03.943 20637 ERROR ironic.conductor.utils File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 123, in authenticate_and_fetch_endpoint_url
2016-09-19 10:19:03.943 20637 ERROR ironic.conductor.utils self.authenticate()
2016-09-19 10:19:03.943 20637 ERROR ironic.conductor.utils File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 239, in authenticate
2016-09-19 10:19:03.943 20637 ERROR ironic.conductor.utils self._authenticate_keystone()
2016-09-19 10:19:03.943 20637 ERROR ironic.conductor.utils File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 220, in _authenticate_keystone
2016-09-19 10:19:03.943 20637 ERROR ironic.conductor.utils raise exceptions.Unauthorized(message=resp_body)
2016-09-19 10:19:03.943 20637 ERROR ironic.conductor.utils Unauthorized: {"error": {"message": "The resource could not be found.", "code": 404, "title": "Not Found"}}
2016-09-19 10:19:03.943 20637 ERROR ironic.conductor.utils

So, is this a bug?

Revision history for this message
Jordan Pittier (jordan-pittier) wrote :

As Tempest is going to remove support for Ironic testing in its source code (in favor of a tempest-plugin maintained by the Ironic team), I am retargeting this bug to Ironic.

no longer affects: tempest
Revision history for this message
Jay Faulkner (jason-oldos) wrote :

It appears your local ironic could not authenticate to Neutron or find the endpoint, causing cleaning to fail. Can you confirm that your configuration is accurate?

We'll need more information about how your environment is configured in order to determine if this is a systematic bug.

Changed in ironic:
status: New → Incomplete
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for Ironic because there has been no activity for 60 days.]

Changed in ironic:
status: Incomplete → Expired
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.