partprobe sometime fails with Device or resource busy

Bug #1626540 reported by Martin André
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ironic
Incomplete
Undecided
octopuszhang

Bug Description

When deploying a whole disk image, I've been facing issue with partprobe failing due to partition being is use:

2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor [req-6ae28454-4dca-4a21-8c2d-843ac6b28bab - - - - -] Asynchronous exception for node 14510c8a-3bc6-4a07-9f5a-b4fd20461fe2: Node failed to get image for deploy. Exception: Deploy failed for instance c97944f7-b688-45cc-8807-c6708f4a3c37. Error: Failed to retrieve partition labels on disk /dev/disk/by-path/ip-192.0.2.12:3260-iscsi-iqn.2008-10.org.openstack:14510c8a-3bc6-4a07-9f5a-b4fd20461fe2-lun-1 for node 14510c8a-3bc6-4a07-9f5a-b4fd20461fe2. Error: Unexpected error while running command.
Command: sudo ironic-rootwrap /etc/ironic/rootwrap.conf partprobe /dev/disk/by-path/ip-192.0.2.12:3260-iscsi-iqn.2008-10.org.openstack:14510c8a-3bc6-4a07-9f5a-b4fd20461fe2-lun-1
Exit code: 1
Stdout: u''
Stderr: u"Error: Error informing the kernel about modifications to partition /dev/sda1 -- Device or resource busy. This means Linux won't know about any changes you made to /dev/sda1 until you reboot -- so you shouldn't mount it or use it in any way before rebooting.\nError: Failed to add partition 1 (Device or resource busy)\n"
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor Traceback (most recent call last):
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/site-packages/ironic/drivers/modules/agent_base_vendor.py", line 482, in heartbeat
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor self.continue_deploy(task)
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/site-packages/ironic_lib/metrics.py", line 61, in wrapped
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor result = f(*args, **kwargs)
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/site-packages/ironic/conductor/task_manager.py", line 138, in wrapper
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor return f(*args, **kwargs)
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/site-packages/ironic/drivers/modules/iscsi_deploy.py", line 381, in continue_deploy
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor uuid_dict_returned = do_agent_iscsi_deploy(task, self._client)
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/site-packages/ironic_lib/metrics.py", line 61, in wrapped
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor result = f(*args, **kwargs)
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/site-packages/ironic/drivers/modules/iscsi_deploy.py", line 313, in do_agent_iscsi_deploy
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor uuid_dict_returned = continue_deploy(task, iqn=iqn, address=address)
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/site-packages/ironic_lib/metrics.py", line 61, in wrapped
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor result = f(*args, **kwargs)
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/site-packages/ironic/drivers/modules/iscsi_deploy.py", line 250, in continue_deploy
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor _fail_deploy(task, msg)
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/site-packages/ironic/drivers/modules/iscsi_deploy.py", line 227, in _fail_deploy
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor raise exception.InstanceDeployFailure(msg)
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor InstanceDeployFailure: Deploy failed for instance c97944f7-b688-45cc-8807-c6708f4a3c37. Error: Failed to retrieve partition labels on disk /dev/disk/by-path/ip-192.0.2.12:3260-iscsi-iqn.2008-10.org.openstack:14510c8a-3bc6-4a07-9f5a-b4fd20461fe2-lun-1 for node 14510c8a-3bc6-4a07-9f5a-b4fd20461fe2. Error: Unexpected error while running command.
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor Command: sudo ironic-rootwrap /etc/ironic/rootwrap.conf partprobe /dev/disk/by-path/ip-192.0.2.12:3260-iscsi-iqn.2008-10.org.openstack:14510c8a-3bc6-4a07-9f5a-b4fd20461fe2-lun-1
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor Exit code: 1
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor Stdout: u''
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor Stderr: u"Error: Error informing the kernel about modifications to partition /dev/sda1 -- Device or resource busy. This means Linux won't know about any changes you made to /dev/sda1 until you reboot -- so you shouldn't mount it or use it in any way before rebooting.\nError: Failed to add partition 1 (Device or resource busy)\n"
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor

Retrying the command when if fails by specifying an 'attempts=10' argument in https://github.com/openstack/ironic-lib/blob/c724cbf368398277071f5b2c735eca32cca22646/ironic_lib/disk_utils.py#L588 may help.

Revision history for this message
Martin André (mandre) wrote :

Better formatted traceback at http://paste.openstack.org/show/582540/

Revision history for this message
Łukasz Leszczuk (lleszczuk) wrote :

What was installed on the disk previously, was it anything using device mapper(lvm?)?

Dmitry Tantsur (divius)
Changed in ironic:
status: New → Incomplete
Revision history for this message
Anup (anup-d-navare) wrote :

Does this bug still exist?

Revision history for this message
Martin André (mandre) wrote :

Yes, I'm still seeing the error in the logs, although ironic seems to be able to recover from it and goes on with my deployment.

Revision history for this message
Martin André (mandre) wrote :

This happens on a fresh tripleo-quickstart installed undercloud.

Revision history for this message
octopuszhang (1004988384-n) wrote :

I have reproduced this bug, If someone is interested in it, the solution is at https://review.openstack.org/#/c/414573/. Please ignore the gate error. The gate failed because it still use ironic-lib.filter from ironic project, not related to the code.

Changed in ironic:
assignee: nobody → octopuszhang (1004988384-n)
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers