ironic-python-agent

IPA fails to deploy partition image with CoreOS based ramdisk

Bug #1590493 reported by Pavlo Shchelokovskyy on 2016-06-08

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	ironic-python-agent	Fix Released	High	Unassigned

Bug Description

Using current DevStack master and virtual nodes with agent_ssh driver, I can not deploy user's partition images.

Image is build with "disk-image-create ubuntu baremetal grub2" and 3 parts of it uploaded to Glance (kernel, initrd and qcow2 root partition).

Flavor extra_specs and node properties are set to capability boot_option local.

Virtual Ironic Node has 2GB RAM and single virtio disk of 10GB (11GiB)

Failure is specifically with CoreOS-based ramdisk image, tinyipa does not have this problem

$ tail -F ir-cond.log | grep ERROR

2016-06-08 17:35:08.638 ERROR ironic.drivers.modules.agent [req-1d9c2bca-1aac-45fc-b08c-665528845ff0 None None] node b968f163-6d3e-4846-97ac-5e2f5af129fb command status errored: {u'message': u'Error writing configdrive to device: Writing configdrive to device /dev/vda failed with exit code 1. stdout: copy_configdrive_to_disk.sh: Working on MBR only device /dev/vda\ncopy_configdrive_to_disk.sh: Adding configdrive partition to /dev/vda\ncopy_configdrive_to_disk.sh: Error creating configdrive on /dev/vda\n. stderr: partx: /dev/vda: failed to read partition table\nparted: invalid token: primary\nError: Expecting a file system type.\n.', u'code': 500, u'type': u'ConfigDriveWriteError', u'details': u'Writing configdrive to device /dev/vda failed with exit code 1. stdout: copy_configdrive_to_disk.sh: Working on MBR only device /dev/vda\ncopy_configdrive_to_disk.sh: Adding configdrive partition to /dev/vda\ncopy_configdrive_to_disk.sh: Error creating configdrive on /dev/vda\n. stderr: partx: /dev/vda: failed to read partition table\nparted: invalid token: primary\nError: Expecting a file system type.\n.'}

relevant VM log:

[ 960.210378] chroot[440]: 2016-06-08 14:35:06.427 440 INFO ironic_python_agent.extensions.standby [-] Image /tmp/3a3ea0b0-46cc-4497-98b4-f5a9dd00ca65 written to device /dev/vda in 863.150555849 seconds
[ 960.250977] chroot[440]: 2016-06-08 14:35:06.468 440 INFO ironic_python_agent.extensions.standby [-] copying configdrive to disk with command /bin/bash /usr/local/lib/python2.7/dist-packages/ironic_python_agent/extensions/../shell/copy_configdrive_to_disk.sh /tmp/configdrive /dev/vda
[ 961.826376] chroot[440]: 2016-06-08 14:35:08.030 440 ERROR root [-] Command failed: prepare_image, error: Error writing configdrive to device: Writing configdrive to device /dev/vda failed with exit code 1. stdout: copy_configdrive_to_disk.sh: Working on MBR only device /dev/vda
[ 961.832737] chroot[440]: copy_configdrive_to_disk.sh: Adding configdrive partition to /dev/vda
[ 961.842312] chroot[440]: copy_configdrive_to_disk.sh: Error creating configdrive on /dev/vda
[ 961.849416] chroot[440]: . stderr: partx: /dev/vda: failed to read partition table
[ 961.857572] chroot[440]: parted: invalid token: primary
[ 961.865729] chroot[440]: Error: Expecting a file system type.
[ 961.873891] chroot[440]: .
[ 961.881924] chroot[440]: 2016-06-08 14:35:08.030 440 ERROR root Traceback (most recent call last):
[ 961.890403] chroot[440]: 2016-06-08 14:35:08.030 440 ERROR root File "/usr/local/lib/python2.7/dist-packages/ironic_python_agent/extensions/base.py", line 151, in run
[ 961.898718] chroot[440]: 2016-06-08 14:35:08.030 440 ERROR root result = self.execute_method(**self.command_params)
[ 961.907284] chroot[440]: 2016-06-08 14:35:08.030 440 ERROR root File "/usr/local/lib/python2.7/dist-packages/ironic_python_agent/extensions/standby.py", line 286, in prepare_image
[ 961.911709] chroot[440]: 2016-06-08 14:35:08.030 440 ERROR root _write_configdrive_to_partition(configdrive, device)
[ 961.916391] chroot[440]: 2016-06-08 14:35:08.030 440 ERROR root File "/usr/local/lib/python2.7/dist-packages/ironic_python_agent/extensions/standby.py", line 109, in _write_configdrive_to_partition
[ 961.921386] chroot[440]: 2016-06-08 14:35:08.030 440 ERROR root e.stderr)
[ 961.925551] chroot[440]: 2016-06-08 14:35:08.030 440 ERROR root ConfigDriveWriteError: Error writing configdrive to device: Writing configdrive to device /dev/vda failed with exit code 1. stdout: copy_configdrive_to_disk.sh: Working on MBR only device /dev/vda
[ 961.930405] chroot[440]: 2016-06-08 14:35:08.030 440 ERROR root copy_configdrive_to_disk.sh: Adding configdrive partition to /dev/vda
[ 961.934873] chroot[440]: 2016-06-08 14:35:08.030 440 ERROR root copy_configdrive_to_disk.sh: Error creating configdrive on /dev/vda
[ 961.940466] chroot[440]: 2016-06-08 14:35:08.030 440 ERROR root . stderr: partx: /dev/vda: failed to read partition table
[ 961.944886] chroot[440]: 2016-06-08 14:35:08.030 440 ERROR root parted: invalid token: primary
[ 961.950484] chroot[440]: 2016-06-08 14:35:08.030 440 ERROR root Error: Expecting a file system type.

I suspect "parted" might have been updated, and now its output is parsed incorrectly - "primary" definitely refers to partition type, not filesystem type expected.

See original description

Pavlo Shchelokovskyy (pshchelo) on 2016-06-09

summary:	- IPA fails to deploy partition image + IPA fails to deploy partition image with CoreOS based ramdisk
description:	updated

Pavlo Shchelokovskyy (pshchelo) on 2016-06-09

description:

updated

Revision history for this message

Vitaliy Nogin (vnogin) wrote on 2016-07-25:

Download full text (12.0 KiB)

Hi,

I faced the same issue during baremetal node deployment using bifrost.

The below you can find errors from the ironic log file:

Hi,

I faced the same issue during baremetal node deployment using bifrost.

The below you can find errors from the ironic log file:

2016-07-25 13:26:40.638 22176 DEBUG ironic.drivers.modules.agent_client [req-a743c1dd-3be2-4b0b-a629-bf235ab5a9ad - - - - -] Status of agent commands for node 00000000-0000-0000-0000-000000000002: prepare_image: result "None", error "{u'message': u'Error writing configdrive to device: Writing configdrive to device /dev/sda failed with exit code 1. stdout: copy_configdrive_to_disk.sh: Working on MBR only device /dev/sda\ncopy_configdrive_to_disk.sh: Adding configdrive partition to /dev/sda\ncopy_configdrive_to_disk.sh: Error creating configdrive on /dev/sda\n. stderr: partx: /dev/sda: failed to read partition table\nparted: invalid token: primary\nError: Expecting a file system type.\n.', u'code': 500, u'type': u'ConfigDriveWriteError', u'details': u'Writing configdrive to device /dev/sda failed with exit code 1. stdout: copy_configdrive_to_disk.sh: Working on MBR only device /dev/sda\ncopy_configdrive_to_disk.sh: Adding configdrive partition to /dev/sda\ncopy_configdrive_to_disk.sh: Error creating configdrive on /dev/sda\n. stderr: partx: /dev/sda: failed to read partition table\nparted: invalid token: primary\nError: Expecting a file system type.\n.'}" get_commands_status /usr/local/lib/python2.7/dist-packages/ironic/drivers/modules/agent_client.py:113
2016-07-25 13:26:40.646 22176 DEBUG ironic.drivers.modules.agent_client [req-a743c1dd-3be2-4b0b-a629-bf235ab5a9ad - - - - -] Status of agent commands for node 00000000-0000-0000-0000-000000000002: prepare_image: result "None", error "{u'message': u'Error writing configdrive to device: Writing configdrive to device /dev/sda failed with exit code 1. stdout: copy_configdrive_to_disk.sh: Working on MBR only device /dev/sda\ncopy_configdrive_to_disk.sh: Adding configdrive partition to /dev/sda\ncopy_configdrive_to_disk.sh: Error creating configdrive on /dev/sda\n. stderr: partx: /dev/sda: failed to read partition table\nparted: invalid token: primary\nError: Expecting a file system type.\n.', u'code': 500, u'type': u'ConfigDriveWriteError', u'details': u'Writing configdrive to device /dev/sda failed with exit code 1. stdout: copy_configdrive_to_disk.sh: Working on MBR only device /dev/sda\ncopy_configdrive_to_disk.sh: Adding configdrive partition to /dev/sda\ncopy_configdrive_to_disk.sh: Error creating configdrive on /dev/sda\n. stderr: partx: /dev/sda: failed to read partition table\nparted: invalid token: primary\nError: Expecting a file system type.\n.'}" get_commands_status /usr/local/lib/python2.7/dist-packages/ironic/drivers/modules/agent_client.py:113
2016-07-25 13:26:40.679 22176 DEBUG ironic.drivers.modules.agent_client [req-a743c1dd-3be2-4b0b-a629-bf235ab5a9ad - - - - -] Status of agent commands for node 00000000-0000-0000-0000-000000000002: prepare_image: result "None", error "{u'message': u'Error writing configdrive to device: Writing configdrive to device /dev/sda failed with exit code 1. stdout: copy_configdrive_to_disk.sh: Working on MBR only device /dev/sda\ncopy_configdrive_to_disk.sh: Adding configdrive partition to /dev/sda\ncopy_configdrive_to_disk.sh: Error creating configdrive on /dev/sda\n. stderr: partx: /dev/sda: failed to read partition table\nparted: invalid token: primary\nError: Expecting a file system type.\n.', u'code': 500, u'type': u'ConfigDriveWriteError', u'details': u'Writing configdrive to device /dev/sda failed with exit code 1. stdout: copy_configdrive_to_disk.sh: Working on MBR only device /dev/sda\ncopy_configdrive_to_disk.sh: Adding configdrive partition to /dev/sda\ncopy_configdrive_to_disk.sh: Error creating configdrive on /dev/sda\n. stderr: partx: /dev/sda: failed to read partition table\nparted: invalid token: primary\nError: Expecting a file system type.\n.'}" get_commands_status /usr/local/lib/python2.7/dist-packages/ironic/drivers/modules/agent_client.py:113
2016-07-25 13:26:40.679 22176 ERROR ironic.drivers.modules.agent [req-a743c1dd-3be2-4b0b-a629-bf235ab5a9ad - - - - -] node 00000000-0000-0000-0000-000000000002 command status errored: {u'message': u'Error writing configdrive to device: Writing configdrive to device /dev/sda failed with exit code 1. stdout: copy_configdrive_to_disk.sh: Working on MBR only device /dev/sda\ncopy_configdrive_to_disk.sh: Adding configdrive partition to /dev/sda\ncopy_configdrive_to_disk.sh: Error creating configdrive on /dev/sda\n. stderr: partx: /dev/sda: failed to read partition table\nparted: invalid token: primary\nError: Expecting a file system type.\n.', u'code': 500, u'type': u'ConfigDriveWriteError', u'details': u'Writing configdrive to device /dev/sda failed with exit code 1. stdout: copy_configdrive_to_disk.sh: Working on MBR only device /dev/sda\ncopy_configdrive_to_disk.sh: Adding configdrive partition to /dev/sda\ncopy_configdrive_to_disk.sh: Error creating configdrive on /dev/sda\n. stderr: partx: /dev/sda: failed to read partition table\nparted: invalid token: primary\nError: Expecting a file system type.\n.'}

It seems that copy_configdrive_to_disk.sh script needs to be fixed:

root@cz7828:/opt/stack/bifrost/playbooks# ironic node-show 00000000-0000-0000-0000-000000000002
+------------------------+--------------------------------------------------------------------------+
| Property               | Value                                                                    |
+------------------------+--------------------------------------------------------------------------+
| chassis_uuid           |                                                                          |
| clean_step             | {}                                                                       |
| console_enabled        | False                                                                    |
| created_at             | 2016-07-25T11:18:54+00:00                                                |
| driver                 | agent_ipmitool                                                           |
| driver_info            | {u'ipmi_password': u'******', u'ipmi_address': u'5.43.227.182',          |
|                        | u'ipmi_username': u'engineer', u'deploy_kernel':                         |
|                        | u'http://172.16.166.15:8080/coreos_pxe.vmlinuz', u'deploy_ramdisk':      |
|                        | u'http://172.16.166.15:8080/coreos_pxe_image-oem.cpio.gz'}               |
| driver_internal_info   | {u'agent_url': u'http://172.16.166.24:9999', u'is_whole_disk_image':     |
|                        | True, u'agent_last_heartbeat': 1469446000}                               |
| extra                  | {}                                                                       |
| inspection_finished_at | None                                                                     |
| inspection_started_at  | None                                                                     |
| instance_info          | {u'root_gb': 10, u'image_source':                                        |
|                        | u'http://172.16.166.15:8080/shadoff.qcow2', u'image_type': u'whole-disk- |
|                        | image', u'configdrive': u'http://172.16.166.15:8080/configdrive-00000000 |
|                        | -0000-0000-0000-000000000002.iso.gz', u'image_url':                      |
|                        | u'http://172.16.166.15:8080/shadoff.qcow2', u'image_disk_format':        |
|                        | u'qcow2', u'image_checksum': u'376ca535e3c752315dbf77df3f11839a'}        |
| instance_uuid          | None                                                                     |
| last_error             | node 00000000-0000-0000-0000-000000000002 command status errored:        |
|                        | {u'message': u'Error writing configdrive to device: Writing configdrive  |
|                        | to device /dev/sda failed with exit code 1. stdout:                      |
|                        | copy_configdrive_to_disk.sh: Working on MBR only device                  |
|                        | /dev/sda                                                                 |
|                        | copy_configdrive_to_disk.sh: Adding configdrive partition to             |
|                        | /dev/sda                                                                 |
|                        | copy_configdrive_to_disk.sh: Error creating configdrive on               |
|                        | /dev/sda                                                                 |
|                        | . stderr: partx: /dev/sda: failed to read partition                      |
|                        | table                                                                    |
|                        | parted: invalid token: primary                                           |
|                        | Error: Expecting a file system                                           |
|                        | type.                                                                    |
|                        | .', u'code': 500, u'type': u'ConfigDriveWriteError', u'details':         |
|                        | u'Writing configdrive to device /dev/sda failed with exit code 1.        |
|                        | stdout: copy_configdrive_to_disk.sh: Working on MBR only device          |
|                        | /dev/sda                                                                 |
|                        | copy_configdrive_to_disk.sh: Adding configdrive partition to             |
|                        | /dev/sda                                                                 |
|                        | copy_configdrive_to_disk.sh: Error creating configdrive on               |
|                        | /dev/sda                                                                 |
|                        | . stderr: partx: /dev/sda: failed to read partition                      |
|                        | table                                                                    |
|                        | parted: invalid token: primary                                           |
|                        | Error: Expecting a file system                                           |
|                        | type.                                                                    |
|                        | .'}                                                                      |
| maintenance            | False                                                                    |
| maintenance_reason     | None                                                                     |
| name                   | testlab22                                                                |
| network_interface      |                                                                          |
| power_state            | power off                                                                |
| properties             | {u'memory_mb': u'16000', u'cpu_arch': u'x86_64', u'local_gb': u'60',     |
|                        | u'cpus': u'8'}                                                           |
| provision_state        | deploy failed                                                            |
| provision_updated_at   | 2016-07-25T11:26:40+00:00                                                |
| raid_config            |                                                                          |
| reservation            | None                                                                     |
| target_power_state     | None                                                                     |
| target_provision_state | active                                                                   |
| target_raid_config     |                                                                          |
| updated_at             | 2016-07-25T11:26:55+00:00                                                |
| uuid                   | 00000000-0000-0000-0000-000000000002                                     |
+------------------------+--------------------------------------------------------------------------+

Regards,
Vitaliy

Jay Faulkner (jason-oldos) on 2016-09-14

Changed in ironic-python-agent:
status:	New → Confirmed
importance:	Undecided → High

Jay Faulkner (jason-oldos) on 2016-12-05

Changed in ironic-python-agent:
status:	Confirmed → Triaged

Revision history for this message

Anup (anup-d-navare) wrote on 2017-02-23:

Hi,
Does this bug still exist? I see similar fix https://review.openstack.org/#/c/296466/

Anup (anup-d-navare) on 2017-02-23

Changed in ironic-python-agent:
assignee:	nobody → Anup (anup-d-navare)

Revision history for this message

Anup (anup-d-navare) wrote on 2017-02-23:

I was able to deploy partition image with agent_ssh driver. I created image with disk-imagebuilder as specified in the description and uploaded it in glance. Here are the commands I used to build the image and upload in glance. http://paste.openstack.org/show/600277/
Used "nova boot" to create an instance. Feel free to let me know if there are other steps to reproduce the bug.

Revision history for this message

Pavlo Shchelokovskyy (pshchelo) wrote on 2017-02-24:

The bug was fixed in ironic_lib, and currently the gates on partition images from ipa source are passing for coreos deploy ramdisk, so I mark the bug as fix released

Changed in ironic-python-agent:
status:	Triaged → Fix Released

Anup (anup-d-navare) on 2017-02-24

Changed in ironic-python-agent:
assignee:	Anup (anup-d-navare) → nobody

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.