ironic-lib

Node failed at deploy. Exception: int() argument must be a string or a number, not 'tuple'

Bug #1676422 reported by George Shuklin on 2017-03-27

This bug affects 1 person

	Status	Importance	Assigned to
Ironic	Invalid	Undecided	Unassigned
Ubuntu Cloud Archive	Fix Released	Undecided	Unassigned
Newton	Triaged	Medium	Unassigned
ironic-lib	Fix Released	Medium	Julia Kreger
python-ironic-lib (Ubuntu)	Fix Released	Medium	Unassigned

Bug Description

Affected version: newton
configuration: pxe_drac, neutron networking

Problem: at last stage of deployment it fails with message: Node failed to get image for deploy. Exception: int() argument must be a string or a number, not 'tuple'

I changed code and added 'raise' in corresponding `except Exception as e` block.

It shows following trace:

ironic.common.utils[18788]: DEBUG Command stderr is: "" [req-e7e7224a-c198-48c1-bf4d-d157ea91e1a1 - - - - -] execute /usr/lib/python2.7/dist-packages/ironic/common/utils.py:76
oslo_concurrency.processutils[18788]: DEBUG Running cmd (subprocess): sudo ironic-rootwrap /etc/ironic/rootwrap.conf iscsiadm -m node -p 204.74.228.37:3260 -T iqn.2008-10.org.openstack:dbf66326-3def-47a8-b061-0b409d3d544c -o delete [req-e7e7224a-c198-48c1-bf4d-d157ea91e1a1 - - - - -] execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:344
oslo_concurrency.processutils[18788]: DEBUG CMD "sudo ironic-rootwrap /etc/ironic/rootwrap.conf iscsiadm -m node -p 204.74.228.37:3260 -T iqn.2008-10.org.openstack:dbf66326-3def-47a8-b061-0b409d3d544c -o delete" returned: 0 in 0.473s [req-e7e7224a-c198-48c1-bf4d-d157ea91e1a1 - - - - -] execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:374
ironic.common.utils[18788]: DEBUG Execution completed, command line is "iscsiadm -m node -p 204.74.228.37:3260 -T iqn.2008-10.org.openstack:dbf66326-3def-47a8-b061-0b409d3d544c -o delete" [req-e7e7224a-c198-48c1-bf4d-d157ea91e1a1 - - - - -] execute /usr/lib/python2.7/dist-packages/ironic/common/utils.py:74
ironic.common.utils[18788]: DEBUG Command stdout is: "" [req-e7e7224a-c198-48c1-bf4d-d157ea91e1a1 - - - - -] execute /usr/lib/python2.7/dist-packages/ironic/common/utils.py:75
ironic.common.utils[18788]: DEBUG Command stderr is: "" [req-e7e7224a-c198-48c1-bf4d-d157ea91e1a1 - - - - -] execute /usr/lib/python2.7/dist-packages/ironic/common/utils.py:76
ironic.drivers.modules.agent_base_vendor[18788]: ERROR Asynchronous exception for node dbf66326-3def-47a8-b061-0b409d3d544c: Node failed to get image for deploy. Exception: int() argument must be a string or a number, not 'tuple' [req-e7e7224a-c198-48c1-bf4d-d157ea91e1a1 - - - - -]
ironic.drivers.modules.agent_base_vendor[18788]: TRACE Traceback (most recent call last):
ironic.drivers.modules.agent_base_vendor[18788]: TRACE File "/usr/lib/python2.7/dist-packages/ironic/drivers/modules/agent_base_vendor.py", line 482, in heartbeat
ironic.drivers.modules.agent_base_vendor[18788]: TRACE self.continue_deploy(task)
ironic.drivers.modules.agent_base_vendor[18788]: TRACE File "/usr/lib/python2.7/dist-packages/ironic_lib/metrics.py", line 61, in wrapped
ironic.drivers.modules.agent_base_vendor[18788]: TRACE result = f(*args, **kwargs)
ironic.drivers.modules.agent_base_vendor[18788]: TRACE File "/usr/lib/python2.7/dist-packages/ironic/conductor/task_manager.py", line 138, in wrapper
ironic.drivers.modules.agent_base_vendor[18788]: TRACE return f(*args, **kwargs)
ironic.drivers.modules.agent_base_vendor[18788]: TRACE File "/usr/lib/python2.7/dist-packages/ironic/drivers/modules/iscsi_deploy.py", line 382, in continue_deploy
ironic.drivers.modules.agent_base_vendor[18788]: TRACE uuid_dict_returned = do_agent_iscsi_deploy(task, self._client)
ironic.drivers.modules.agent_base_vendor[18788]: TRACE File "/usr/lib/python2.7/dist-packages/ironic_lib/metrics.py", line 61, in wrapped
ironic.drivers.modules.agent_base_vendor[18788]: TRACE result = f(*args, **kwargs)
ironic.drivers.modules.agent_base_vendor[18788]: TRACE File "/usr/lib/python2.7/dist-packages/ironic/drivers/modules/iscsi_deploy.py", line 314, in do_agent_iscsi_deploy
ironic.drivers.modules.agent_base_vendor[18788]: TRACE uuid_dict_returned = continue_deploy(task, iqn=iqn, address=address)
ironic.drivers.modules.agent_base_vendor[18788]: TRACE File "/usr/lib/python2.7/dist-packages/ironic_lib/metrics.py", line 61, in wrapped
ironic.drivers.modules.agent_base_vendor[18788]: TRACE result = f(*args, **kwargs)
ironic.drivers.modules.agent_base_vendor[18788]: TRACE File "/usr/lib/python2.7/dist-packages/ironic/drivers/modules/iscsi_deploy.py", line 243, in continue_deploy
ironic.drivers.modules.agent_base_vendor[18788]: TRACE uuid_dict_returned = deploy_utils.deploy_disk_image(**params)
ironic.drivers.modules.agent_base_vendor[18788]: TRACE File "/usr/lib/python2.7/dist-packages/ironic/drivers/modules/deploy_utils.py", line 388, in deploy_disk_image
ironic.drivers.modules.agent_base_vendor[18788]: TRACE configdrive)
ironic.drivers.modules.agent_base_vendor[18788]: TRACE File "/usr/lib/python2.7/dist-packages/ironic_lib/disk_utils.py", line 722, in create_config_drive_partition
ironic.drivers.modules.agent_base_vendor[18788]: TRACE if _is_disk_larger_than_max_size(device, node_uuid):
ironic.drivers.modules.agent_base_vendor[18788]: TRACE File "/usr/lib/python2.7/dist-packages/ironic_lib/disk_utils.py", line 571, in _is_disk_larger_than_max_size
ironic.drivers.modules.agent_base_vendor[18788]: TRACE disksize_mb = int(disksize_bytes) // 1024 // 1024
ironic.drivers.modules.agent_base_vendor[18788]: TRACE TypeError: int() argument must be a string or a number, not 'tuple'
ironic.drivers.modules.agent_base_vendor[18788]: TRACE

Revision history for this message

George Shuklin (george-shuklin) wrote on 2017-03-27:

This trace is ugly. Better version:

File "/usr/lib/python2.7/dist-packages/ironic/drivers/modules/agent_base_vendor.py", line 482, in heartbeat
  self.continue_deploy(task)
File "/usr/lib/python2.7/dist-packages/ironic_lib/metrics.py", line 61, in wrapped
  result = f(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/ironic/conductor/task_manager.py", line 138, in wrapper
  return f(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/ironic/drivers/modules/iscsi_deploy.py", line 382, in continue_deploy
  uuid_dict_returned = do_agent_iscsi_deploy(task, self._client)
File "/usr/lib/python2.7/dist-packages/ironic_lib/metrics.py", line 61, in wrapped
  result = f(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/ironic/drivers/modules/iscsi_deploy.py", line 314, in do_agent_iscsi_deploy
  uuid_dict_returned = continue_deploy(task, iqn=iqn, address=address)
File "/usr/lib/python2.7/dist-packages/ironic_lib/metrics.py", line 61, in wrapped
  result = f(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/ironic/drivers/modules/iscsi_deploy.py", line 243, in continue_deploy
  uuid_dict_returned = deploy_utils.deploy_disk_image(**params)
File "/usr/lib/python2.7/dist-packages/ironic/drivers/modules/deploy_utils.py", line 388, in deploy_disk_image
  configdrive)
File "/usr/lib/python2.7/dist-packages/ironic_lib/disk_utils.py", line 722, in create_config_drive_partition
  if _is_disk_larger_than_max_size(device, node_uuid):
File "/usr/lib/python2.7/dist-packages/ironic_lib/disk_utils.py", line 571, in _is_disk_larger_than_max_size
  disksize_mb = int(disksize_bytes) // 1024 // 1024

summary:

- Node failed to get image for deploy. Exception: int() argument must be
- a string or a number, not 'tuple'
+ Node failed at deploy. Exception: int() argument must be a string or a
+ number, not 'tuple'

Revision history for this message

George Shuklin (george-shuklin) wrote on 2017-03-27:

Update: I set up instance creation with --config-drive=True option.

Revision history for this message

Michael Turek (mjturek) wrote on 2017-03-27:

Thanks for the report George!

Can you provide some more information? Some useful starting information would be:

What OS/distro are you using?

How was ironic set up? (installation guide, devstack, packstack?)

Thanks!

Changed in ironic:
status:	New → Incomplete

Revision history for this message

George Shuklin (george-shuklin) wrote on 2017-03-28:

Hello.

I done this with Xenial/newton distribution, but I think this bug should appears on any installation of Ironic/Newton.

Conditions:

0. integration with nova/neutron.
1. pxe_drac driver is in use for node.
2. instance is booted with --config-drive True

As far as I can see by the trace, issue happens somewhere near the code which writes config drive into node in pxe (iscsi) mode.

I'm not sure, but may be it can be reproduced in other pxe_* drivers (they should use iscsi to mount node drive to ironic-conductor node).

Changed in ironic:
status:	Incomplete → New

Julia Kreger (juliaashleykreger) on 2017-03-28

Changed in ironic-lib:
assignee:	nobody → Julia Kreger (juliaashleykreger)
status:	New → Confirmed
importance:	Undecided → Critical

Revision history for this message

Julia Kreger (juliaashleykreger) wrote on 2017-03-28:

This looks like a bug in ironic-lib as it has to do with handling command output. That being said, the call could still be failing with the bug in the command output being fixed. You may wish to check your kernel log to see if you see any errors there. In the mean time, I'm working on a patch for the basic issue in ironic-lib.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-03-28: Fix proposed to ironic-lib (master)

Fix proposed to branch: master
Review: https://review.openstack.org/450754

Changed in ironic-lib:
status:	Confirmed → In Progress

Revision history for this message

Julia Kreger (juliaashleykreger) wrote on 2017-03-28:

Please give https://review.openstack.org/450754 a try in your ironic-lib, and see if the clears this issue up for you.

Changed in ironic:
status:	New → Triaged

Revision history for this message

George Shuklin (george-shuklin) wrote on 2017-03-28:

There is no related dmesg errors, only messages from iscsi:

[2752572.185425] sd 4:0:0:1: [sda] Synchronizing SCSI cache
[2754877.255269] scsi host5: iSCSI Initiator over TCP/IP
[2754877.517408] scsi 5:0:0:0: RAID IET Controller 0001 PQ: 0 ANSI: 5
[2754877.518327] scsi 5:0:0:0: Attached scsi generic sg0 type 12
[2754877.519606] scsi 5:0:0:1: Direct-Access IET VIRTUAL-DISK 0001 PQ: 0 ANSI: 5
[2754877.520364] sd 5:0:0:1: Attached scsi generic sg1 type 0
[2754877.520956] sd 5:0:0:1: [sda] 937703088 512-byte logical blocks: (480 GB/447 GiB)
[2754877.521549] sd 5:0:0:1: [sda] Write Protect is off
[2754877.521556] sd 5:0:0:1: [sda] Mode Sense: 69 00 00 08
[2754877.521864] sd 5:0:0:1: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[2754877.527445] sd 5:0:0:1: [sda] Attached SCSI disk
[2754897.817887] sda: sda1
[2754898.789711] sda: sda1
[2754901.652210] sd 5:0:0:1: [sda] Synchronizing SCSI cache
[2756295.626738] scsi host6: iSCSI Initiator over TCP/IP
[2756295.886870] scsi 6:0:0:0: RAID IET Controller 0001 PQ: 0 ANSI: 5
[2756295.887637] scsi 6:0:0:0: Attached scsi generic sg0 type 12
[2756295.888337] scsi 6:0:0:1: Direct-Access IET VIRTUAL-DISK 0001 PQ: 0 ANSI: 5
[2756295.889156] sd 6:0:0:1: Attached scsi generic sg1 type 0
[2756295.892813] sd 6:0:0:1: [sda] 937703088 512-byte logical blocks: (480 GB/447 GiB)
[2756295.893590] sd 6:0:0:1: [sda] Write Protect is off
[2756295.893608] sd 6:0:0:1: [sda] Mode Sense: 69 00 00 08
[2756295.893944] sd 6:0:0:1: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[2756295.899332] sd 6:0:0:1: [sda] Attached SCSI disk
[2756316.290751] sda: sda1
[2756317.296678] sda: sda1
[2756320.197870] sd 6:0:0:1: [sda] Synchronizing SCSI cache
[2757572.317804] scsi host7: iSCSI Initiator over TCP/IP
[2757572.576966] scsi 7:0:0:0: RAID IET Controller 0001 PQ: 0 ANSI: 5
[2757572.577919] scsi 7:0:0:0: Attached scsi generic sg0 type 12
[2757572.580114] scsi 7:0:0:1: Direct-Access IET VIRTUAL-DISK 0001 PQ: 0 ANSI: 5
[2757572.580889] sd 7:0:0:1: Attached scsi generic sg1 type 0
[2757572.582723] sd 7:0:0:1: [sda] 937703088 512-byte logical blocks: (480 GB/447 GiB)
[2757572.585768] sd 7:0:0:1: [sda] Write Protect is off
[2757572.585773] sd 7:0:0:1: [sda] Mode Sense: 69 00 00 08
[2757572.588952] sd 7:0:0:1: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[2757572.594710] sd 7:0:0:1: [sda] Attached SCSI disk
[2757592.882001] sda: sda1

There is no related dmesg errors, only messages from iscsi:

[2752572.185425] sd 4:0:0:1: [sda] Synchronizing SCSI cache
[2754877.255269] scsi host5: iSCSI Initiator over TCP/IP
[2754877.517408] scsi 5:0:0:0: RAID              IET      Controller       0001 PQ: 0 ANSI: 5
[2754877.518327] scsi 5:0:0:0: Attached scsi generic sg0 type 12
[2754877.519606] scsi 5:0:0:1: Direct-Access     IET      VIRTUAL-DISK     0001 PQ: 0 ANSI: 5
[2754877.520364] sd 5:0:0:1: Attached scsi generic sg1 type 0
[2754877.520956] sd 5:0:0:1: [sda] 937703088 512-byte logical blocks: (480 GB/447 GiB)
[2754877.521549] sd 5:0:0:1: [sda] Write Protect is off
[2754877.521556] sd 5:0:0:1: [sda] Mode Sense: 69 00 00 08
[2754877.521864] sd 5:0:0:1: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[2754877.527445] sd 5:0:0:1: [sda] Attached SCSI disk
[2754897.817887]  sda: sda1
[2754898.789711]  sda: sda1
[2754901.652210] sd 5:0:0:1: [sda] Synchronizing SCSI cache
[2756295.626738] scsi host6: iSCSI Initiator over TCP/IP
[2756295.886870] scsi 6:0:0:0: RAID              IET      Controller       0001 PQ: 0 ANSI: 5
[2756295.887637] scsi 6:0:0:0: Attached scsi generic sg0 type 12
[2756295.888337] scsi 6:0:0:1: Direct-Access     IET      VIRTUAL-DISK     0001 PQ: 0 ANSI: 5
[2756295.889156] sd 6:0:0:1: Attached scsi generic sg1 type 0
[2756295.892813] sd 6:0:0:1: [sda] 937703088 512-byte logical blocks: (480 GB/447 GiB)
[2756295.893590] sd 6:0:0:1: [sda] Write Protect is off
[2756295.893608] sd 6:0:0:1: [sda] Mode Sense: 69 00 00 08
[2756295.893944] sd 6:0:0:1: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[2756295.899332] sd 6:0:0:1: [sda] Attached SCSI disk
[2756316.290751]  sda: sda1
[2756317.296678]  sda: sda1
[2756320.197870] sd 6:0:0:1: [sda] Synchronizing SCSI cache
[2757572.317804] scsi host7: iSCSI Initiator over TCP/IP
[2757572.576966] scsi 7:0:0:0: RAID              IET      Controller       0001 PQ: 0 ANSI: 5
[2757572.577919] scsi 7:0:0:0: Attached scsi generic sg0 type 12
[2757572.580114] scsi 7:0:0:1: Direct-Access     IET      VIRTUAL-DISK     0001 PQ: 0 ANSI: 5
[2757572.580889] sd 7:0:0:1: Attached scsi generic sg1 type 0
[2757572.582723] sd 7:0:0:1: [sda] 937703088 512-byte logical blocks: (480 GB/447 GiB)
[2757572.585768] sd 7:0:0:1: [sda] Write Protect is off
[2757572.585773] sd 7:0:0:1: [sda] Mode Sense: 69 00 00 08
[2757572.588952] sd 7:0:0:1: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[2757572.594710] sd 7:0:0:1: [sda] Attached SCSI disk
[2757592.882001]  sda: sda1

Revision history for this message

Vasyl Saienko (vsaienko) wrote on 2017-03-29:

From the provided trace I'm wondered what version of ironic-lib you are using? In newton the code looks like [1]

File "/usr/lib/python2.7/dist-packages/ironic_lib/disk_utils.py", line 571, in _is_disk_larger_than_max_size
disksize_mb = int(disksize_bytes) // 1024 // 1024

[1] https://github.com/openstack/ironic-lib/blob/stable/newton/ironic_lib/disk_utils.py#L599
disksize_mb = int(disksize_bytes[0].strip()) // 1024 // 1024

Revision history for this message

George Shuklin (george-shuklin) wrote on 2017-03-29:

#10

My version:

disksize_mb = int(disksize_bytes) // 1024 // 1024
return disksize_mb > MAX_DISK_SIZE_MB_SUPPORTED_BY_MBR

ironic-lib version: python-ironic-lib 2.1.0-0ubuntu1~cloud0

Revision history for this message

Julia Kreger (juliaashleykreger) wrote on 2017-03-29:

#11

Looks like you should ideally have 2.1.3, which is where this issue should have been fixed, and would have been the latest tag for newton deployments.

https://github.com/openstack/ironic-lib/blob/2.1.3/ironic_lib/disk_utils.py#L599

Changed in ironic:
status:	Triaged → Invalid
Changed in ironic-lib:
importance:	Critical → Undecided
status:	In Progress → Incomplete

Revision history for this message

Vasyl Saienko (vsaienko) wrote on 2017-03-29:

#12

so indeed in version 2.1.0 there is a buggy code that was fixed by:
https://github.com/openstack/ironic-lib/commit/33af42c96556ec66087954f115697d79967efe91
and it was fixed in ironic-lib 2.2.0

Changed in ironic-lib:
importance:	Undecided → Low
importance:	Low → Medium

Revision history for this message

George Shuklin (george-shuklin) wrote on 2017-03-29:

#13

Hello Ubuntu guys. Can you please upload proper version of ironic-lib into cloud-archive?

Revision history for this message

James Page (james-page) wrote on 2017-10-23:

#14

Relates to newton UCA.

affects:	ironic (Ubuntu) → python-ironic-lib (Ubuntu)
Changed in python-ironic-lib (Ubuntu):
status:	New → Triaged
importance:	Undecided → Medium
status:	Triaged → Fix Released
Changed in cloud-archive:
status:	New → Fix Released

Revision history for this message

Ruby Loo (rloo) wrote on 2018-01-10:

#15

looks like this has been fixed :)

Changed in ironic-lib:
status:	Incomplete → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.