curtin miscalculates partition offsets when logical_block_size is 4k

[Ryan Beisner]

We experience a very high rate of failure in deploying ppc64el via the MAAS web ui, for any Ubuntu daily image release (Trusty, Wily, Xenial).

In 15 attempts, I was able to get 3 machines to a 'deployed' state. We have 4 Power8 machines and they all exhibit this behavior.

With Trusty and Wily, when the unit fails to deploy, the machines are accessible.

With Xenial, when the unit fails to deploy, the machine is not reachable via network.

In all cases, the oobm serial console shows the machine at the login prompt.

We are also observing that the ipmi version, although initially set to version 2.0, is reverting to 1.5 after a unit fails deployment and is released. (https://bugs.launchpad.net/maas/+bug/1556219) This means that all subsequent power commands will fail until the user manually sets the ipmi back to 2.0 for the machines.

What is interesting/odd in the machine output, is that it's trying very hard to do operations on /dev/sde when /dev/sda is what I would expect.

# Storage looks like this:

File systems
Name Size Mountpoint File system
sda-part2 283.8 GB / ext4

Available disks and partitions
Name Model Serial BootSizeDevice TypeFile systemTags

sdb
 283.8 GB Physical rotary

sdc
 283.8 GB Physical rotary

sdd
 283.8 GB Physical rotary

sde
 283.8 GB Physical rotary

Used disks and partitions
Name Model Serial BootDevice typeUsed for
sda PhysicalGPT partitioned with 1 partition
sda-part2 Partitionext4 formatted filesystem mounted at /

# Network looks like this:
eth0
 Physical 10.X.X.X (Auto assign)

eth1
Physical (Unconfigured)

eth2
Physical (Unconfigured)

eth3
Physical (Unconfigured)

eth4
Physical (Unconfigured)

eth5
Physical (Unconfigured)

# Machine output

Running command ['partprobe', '/dev/sde'] with allowed return codes [0, 1] (shell=False, capture=False)
Running command ['udevadm', 'settle'] with allowed return codes [0] (shell=False, capture=False)
Running command ['mdadm', '--assemble', '--scan'] with allowed return codes [0, 1, 2] (shell=False, capture=True)
Running command ['udevadm', 'settle'] with allowed return codes [0] (shell=False, capture=False)
clear_holders running on '/sys/block/sde/sde1', with holders '[]'
Running command ['sgdisk', '--zap-all', '/dev/sde1'] with allowed return codes [0, 1, 2, 5] (shell=False, capture=True)
clear_holders running on '/sys/block/sde', with holders '[]'
Running command ['sgdisk', '--zap-all', '/dev/sde'] with allowed return codes [0, 1, 2, 5] (shell=False, capture=True)
labeling device: '/dev/sde' with 'gpt' partition table
Running command ['sgdisk', '--clear', '/dev/sde'] with allowed return codes [0] (shell=False, capture=False)
Creating new GPT entries.
The operation has completed successfully.
Running command ['partprobe', '/dev/sde'] with allowed return codes [0, 1] (shell=False, capture=False)
Running command ['udevadm', 'settle'] with allowed return codes [0] (shell=False, capture=False)
Running command ['blkid', '-o', 'export', '/dev/sde'] with allowed return codes [0, 2] (shell=False, capture=True)
Running command ['partprobe', '/dev/sdb'] with allowed return codes [0, 1] (shell=False, capture=False)
Running command ['udevadm', 'settle'] with allowed return codes [0] (shell=False, capture=False)
Running command ['mdadm', '--assemble', '--scan'] with allowed return codes [0, 1, 2] (shell=False, capture=True)
Running command ['udevadm', 'settle'] with allowed return codes [0] (shell=False, capture=False)
clear_holders running on '/sys/block/sdb', with holders '[]'
Running command ['sgdisk', '--zap-all', '/dev/sdb'] with allowed return codes [0, 1, 2, 5] (shell=False, capture=True)
Running command ['partprobe', '/dev/sdb'] with allowed return codes [0, 1] (shell=False, capture=False)
Running command ['udevadm', 'settle'] with allowed return codes [0] (shell=False, capture=False)
Running command ['blkid', '-o', 'export', '/dev/sdb'] with allowed return codes [0, 2] (shell=False, capture=True)
Can't find a uuid for volume: sdb. Skipping dname.
Running command ['partprobe', '/dev/sda'] with allowed return codes [0, 1] (shell=False, capture=False)
Running command ['udevadm', 'settle'] with allowed return codes [0] (shell=False, capture=False)
Running command ['mdadm', '--assemble', '--scan'] with allowed return codes [0, 1, 2] (shell=False, capture=True)
Running command ['udevadm', 'settle'] with allowed return codes [0] (shell=False, capture=False)
clear_holders running on '/sys/block/sda', with holders '[]'
Running command ['sgdisk', '--zap-all', '/dev/sda'] with allowed return codes [0, 1, 2, 5] (shell=False, capture=True)
Running command ['partprobe', '/dev/sda'] with allowed return codes [0, 1] (shell=False, capture=False)
Running command ['udevadm', 'settle'] with allowed return codes [0] (shell=False, capture=False)
Running command ['blkid', '-o', 'export', '/dev/sda'] with allowed return codes [0, 2] (shell=False, capture=True)
Can't find a uuid for volume: sdc. Skipping dname.
Running command ['partprobe', '/dev/sdd'] with allowed return codes [0, 1] (shell=False, capture=False)
Running command ['udevadm', 'settle'] with allowed return codes [0] (shell=False, capture=False)
Running command ['mdadm', '--assemble', '--scan'] with allowed return codes [0, 1, 2] (shell=False, capture=True)
Running command ['udevadm', 'settle'] with allowed return codes [0] (shell=False, capture=False)
clear_holders running on '/sys/block/sdd', with holders '[]'
Running command ['sgdisk', '--zap-all', '/dev/sdd'] with allowed return codes [0, 1, 2, 5] (shell=False, capture=True)
Running command ['partprobe', '/dev/sdd'] with allowed return codes [0, 1] (shell=False, capture=False)
Running command ['udevadm', 'settle'] with allowed return codes [0] (shell=False, capture=False)
Running command ['blkid', '-o', 'export', '/dev/sdd'] with allowed return codes [0, 2] (shell=False, capture=True)
Can't find a uuid for volume: sdd. Skipping dname.
Running command ['partprobe', '/dev/sdc'] with allowed return codes [0, 1] (shell=False, capture=False)
Running command ['udevadm', 'settle'] with allowed return codes [0] (shell=False, capture=False)
Running command ['mdadm', '--assemble', '--scan'] with allowed return codes [0, 1, 2] (shell=False, capture=True)
Running command ['udevadm', 'settle'] with allowed return codes [0] (shell=False, capture=False)
clear_holders running on '/sys/block/sdc', with holders '[]'
Running command ['sgdisk', '--zap-all', '/dev/sdc'] with allowed return codes [0, 1, 2, 5] (shell=False, capture=True)
Running command ['partprobe', '/dev/sdc'] with allowed return codes [0, 1] (shell=False, capture=False)
Running command ['udevadm', 'settle'] with allowed return codes [0] (shell=False, capture=False)
Running command ['blkid', '-o', 'export', '/dev/sdc'] with allowed return codes [0, 2] (shell=False, capture=True)
Can't find a uuid for volume: sde. Skipping dname.
Running command ['partprobe', '/dev/sde'] with allowed return codes [0, 1] (shell=False, capture=False)
Running command ['udevadm', 'settle'] with allowed return codes [0] (shell=False, capture=False)
Running command ['partprobe', '/dev/sde'] with allowed return codes [0, 1] (shell=False, capture=False)
Running command ['udevadm', 'settle'] with allowed return codes [0] (shell=False, capture=False)
adding partition 'sda-part1' to disk 'sda'
Running command ['sgdisk', '--new', '1:256.0:2303.0', '--typecode=1:4100', '/dev/sde'] with allowed return codes [0] (shell=False, capture=False)
The operation has completed successfully.
Running command ['partprobe', '/dev/sde'] with allowed return codes [0, 1] (shell=False, capture=False)
Running command ['udevadm', 'settle'] with allowed return codes [0] (shell=False, capture=False)
Running command ['partprobe', '/dev/sde'] with allowed return codes [0, 1] (shell=False, capture=False)
Running command ['udevadm', 'settle'] with allowed return codes [0] (shell=False, capture=False)
Running command ['dd', 'bs=512', 'if=/dev/zero', 'of=/dev/sde1'] with allowed return codes [0, 1, 2, 5] (shell=False, capture=True)
Running command ['partprobe', '/dev/sde'] with allowed return codes [0, 1] (shell=False, capture=False)
Running command ['udevadm', 'settle'] with allowed return codes [0] (shell=False, capture=False)
Running command ['partprobe', '/dev/sde'] with allowed return codes [0, 1] (shell=False, capture=False)
Running command ['udevadm', 'settle'] with allowed return codes [0] (shell=False, capture=False)
Running command ['blkid', '-o', 'export', '/dev/sde1'] with allowed return codes [0, 2] (shell=False, capture=True)
Running command ['partprobe', '/dev/sde'] with allowed return codes [0, 1] (shell=False, capture=False)
Running command ['udevadm', 'settle'] with allowed return codes [0] (shell=False, capture=False)
Running command ['partprobe', '/dev/sde'] with allowed return codes [0, 1] (shell=False, capture=False)
Running command ['udevadm', 'settle'] with allowed return codes [0] (shell=False, capture=False)
previous partition number for 'sda-part2' found to be '1'
adding partition 'sda-part2' to disk 'sda'
Running command ['sgdisk', '--new', '2:18432:69300223', '--typecode=2:8300', '/dev/sde'] with allowed return codes [0] (shell=False, capture=False)
Could not create partition 2 from 18432 to 69300223
Could not change partition 2's type code to 8300!
Error encountered; not saving changes.
An error occured handling 'sda-part2': ProcessExecutionError - Unexpected error while running command.
Command: ['sgdisk', '--new', '2:18432:69300223', '--typecode=2:8300', '/dev/sde']
Exit code: 4
Reason: -
Stdout: ''
Stderr: ''
Traceback (most recent call last):
  File "/curtin/curtin/commands/main.py", line 209, in main
    ret = args.func(args)
  File "/curtin/curtin/commands/block_meta.py", line 62, in block_meta
    meta_custom(args)
  File "/curtin/curtin/commands/block_meta.py", line 1083, in meta_custom
    handler(command, storage_config_dict)
  File "/curtin/curtin/commands/block_meta.py", line 643, in partition_handler
    util.subp(cmd)
  File "/curtin/curtin/util.py", line 140, in subp
    return _subp(*args, **kwargs)
  File "/curtin/curtin/util.py", line 90, in _subp
    cmd=args)
curtin.util.ProcessExecutionError: Unexpected error while running command.
Command: ['sgdisk', '--new', '2:18432:69300223', '--typecode=2:8300', '/dev/sde']
Exit code: 4
Reason: -
Stdout: ''
Stderr: ''
Unexpected error while running command.
Command: ['sgdisk', '--new', '2:18432:69300223', '--typecode=2:8300', '/dev/sde']
Exit code: 4
Reason: -
Stdout: ''
Stderr: ''
Installation failed with exception: Unexpected error while running command.
Command: ['curtin', 'block-meta', 'custom']
Exit code: 3
Reason: -
Stdout: b'Running command [\'partprobe\', \'/dev/sde\'] with allowed return codes [0, 1] (shell=False, capture=False)\nRunning command [\'udevadm\', \'settle\'] with allowed return codes [0] (shell=False, capture=False)\nRunning command [\'mdadm\', \'--assemble\', \'--scan\'] with allowed return codes [0, 1, 2] (shell=False, capture=True)\nRunning command [\'udevadm\', \'settle\'] with allowed return codes [0] (shell=False, capture=False)\nclear_holders running on \'/sys/block/sde/sde1\', with holders \'[]\'\nRunning command [\'sgdisk\', \'--zap-all\', \'/dev/sde1\'] with allowed return codes [0, 1, 2, 5] (shell=False, capture=True)\nclear_holders running on \'/sys/block/sde\', with holders \'[]\'\nRunning command [\'sgdisk\', \'--zap-all\', \'/dev/sde\'] with allowed return codes [0, 1, 2, 5] (shell=False, capture=True)\nlabeling device: \'/dev/sde\' with \'gpt\' partition table\nRunning command [\'sgdisk\', \'--clear\', \'/dev/sde\'] with allowed return codes [0] (shell=False, capture=False)\nCreating new GPT entries.\nThe operation has completed successfully.\nRunning command [\'partprobe\', \'/dev/sde\'] with allowed return codes [0, 1] (shell=False, capture=False)\nRunning command [\'udevadm\', \'settle\'] with allowed return codes [0] (shell=False, capture=False)\nRunning command [\'blkid\', \'-o\', \'export\', \'/dev/sde\'] with allowed return codes [0, 2] (shell=False, capture=True)\nRunning command [\'partprobe\', \'/dev/sdb\'] with allowed return codes [0, 1] (shell=False, capture=False)\nRunning command [\'udevadm\', \'settle\'] with allowed return codes [0] (shell=False, capture=False)\nRunning command [\'mdadm\', \'--assemble\', \'--scan\'] with allowed return codes [0, 1, 2] (shell=False, capture=True)\nRunning command [\'udevadm\', \'settle\'] with allowed return codes [0] (shell=False, capture=False)\nclear_holders running on \'/sys/block/sdb\', with holders \'[]\'\nRunning command [\'sgdisk\', \'--zap-all\', \'/dev/sdb\'] with allowed return codes [0, 1, 2, 5] (shell=False, capture=True)\nRunning command [\'partprobe\', \'/dev/sdb\'] with allowed return codes [0, 1] (shell=False, capture=False)\nRunning command [\'udevadm\', \'settle\'] with allowed return codes [0] (shell=False, capture=False)\nRunning command [\'blkid\', \'-o\', \'export\', \'/dev/sdb\'] with allowed return codes [0, 2] (shell=False, capture=True)\nCan\'t find a uuid for volume: sdb. Skipping dname.\nRunning command [\'partprobe\', \'/dev/sda\'] with allowed return codes [0, 1] (shell=False, capture=False)\nRunning command [\'udevadm\', \'settle\'] with allowed return codes [0] (shell=False, capture=False)\nRunning command [\'mdadm\', \'--assemble\', \'--scan\'] with allowed return codes [0, 1, 2] (shell=False, capture=True)\nRunning command [\'udevadm\', \'settle\'] with allowed return codes [0] (shell=False, capture=False)\nclear_holders running on \'/sys/block/sda\', with holders \'[]\'\nRunning command [\'sgdisk\', \'--zap-all\', \'/dev/sda\'] with allowed return codes [0, 1, 2, 5] (shell=False, capture=True)\nRunning command [\'partprobe\', \'/dev/sda\'] with allowed return codes [0, 1] (shell=False, capture=False)\nRunning command [\'udevadm\', \'settle\'] with allowed return codes [0] (shell=False, capture=False)\nRunning command [\'blkid\', \'-o\', \'export\', \'/dev/sda\'] with allowed return codes [0, 2] (shell=False, capture=True)\nCan\'t find a uuid for volume: sdc. Skipping dname.\nRunning command [\'partprobe\', \'/dev/sdd\'] with allowed return codes [0, 1] (shell=False, capture=False)\nRunning command [\'udevadm\', \'settle\'] with allowed return codes [0] (shell=False, capture=False)\nRunning command [\'mdadm\', \'--assemble\', \'--scan\'] with allowed return codes [0, 1, 2] (shell=False, capture=True)\nRunning command [\'udevadm\', \'settle\'] with allowed return codes [0] (shell=False, capture=False)\nclear_holders running on \'/sys/block/sdd\', with holders \'[]\'\nRunning command [\'sgdisk\', \'--zap-all\', \'/dev/sdd\'] with allowed return codes [0, 1, 2, 5] (shell=False, capture=True)\nRunning command [\'partprobe\', \'/dev/sdd\'] with allowed return codes [0, 1] (shell=False, capture=False)\nRunning command [\'udevadm\', \'settle\'] with allowed return codes [0] (shell=False, capture=False)\nRunning command [\'blkid\', \'-o\', \'export\', \'/dev/sdd\'] with allowed return codes [0, 2] (shell=False, capture=True)\nCan\'t find a uuid for volume: sdd. Skipping dname.\nRunning command [\'partprobe\', \'/dev/sdc\'] with allowed return codes [0, 1] (shell=False, capture=False)\nRunning command [\'udevadm\', \'settle\'] with allowed return codes [0] (shell=False, capture=False)\nRunning command [\'mdadm\', \'--assemble\', \'--scan\'] with allowed return codes [0, 1, 2] (shell=False, capture=True)\nRunning command [\'udevadm\', \'settle\'] with allowed return codes [0] (shell=False, capture=False)\nclear_holders running on \'/sys/block/sdc\', with holders \'[]\'\nRunning command [\'sgdisk\', \'--zap-all\', \'/dev/sdc\'] with allowed return codes [0, 1, 2, 5] (shell=False, capture=True)\nRunning command [\'partprobe\', \'/dev/sdc\'] with allowed return codes [0, 1] (shell=False, capture=False)\nRunning command [\'udevadm\', \'settle\'] with allowed return codes [0] (shell=False, capture=False)\nRunning command [\'blkid\', \'-o\', \'export\', \'/dev/sdc\'] with allowed return codes [0, 2] (shell=False, capture=True)\nCan\'t find a uuid for volume: sde. Skipping dname.\nRunning command [\'partprobe\', \'/dev/sde\'] with allowed return codes [0, 1] (shell=False, capture=False)\nRunning command [\'udevadm\', \'settle\'] with allowed return codes [0] (shell=False, capture=False)\nRunning command [\'partprobe\', \'/dev/sde\'] with allowed return codes [0, 1] (shell=False, capture=False)\nRunning command [\'udevadm\', \'settle\'] with allowed return codes [0] (shell=False, capture=False)\nadding partition \'sda-part1\' to disk \'sda\'\nRunning command [\'sgdisk\', \'--new\', \'1:256.0:2303.0\', \'--typecode=1:4100\', \'/dev/sde\'] with allowed return codes [0] (shell=False, capture=False)\nThe operation has completed successfully.\nRunning command [\'partprobe\', \'/dev/sde\'] with allowed return codes [0, 1] (shell=False, capture=False)\nRunning command [\'udevadm\', \'settle\'] with allowed return codes [0] (shell=False, capture=False)\nRunning command [\'partprobe\', \'/dev/sde\'] with allowed return codes [0, 1] (shell=False, capture=False)\nRunning command [\'udevadm\', \'settle\'] with allowed return codes [0] (shell=False, capture=False)\nRunning command [\'dd\', \'bs=512\', \'if=/dev/zero\', \'of=/dev/sde1\'] with allowed return codes [0, 1, 2, 5] (shell=False, capture=True)\nRunning command [\'partprobe\', \'/dev/sde\'] with allowed return codes [0, 1] (shell=False, capture=False)\nRunning command [\'udevadm\', \'settle\'] with allowed return codes [0] (shell=False, capture=False)\nRunning command [\'partprobe\', \'/dev/sde\'] with allowed return codes [0, 1] (shell=False, capture=False)\nRunning command [\'udevadm\', \'settle\'] with allowed return codes [0] (shell=False, capture=False)\nRunning command [\'blkid\', \'-o\', \'export\', \'/dev/sde1\'] with allowed return codes [0, 2] (shell=False, capture=True)\nRunning command [\'partprobe\', \'/dev/sde\'] with allowed return codes [0, 1] (shell=False, capture=False)\nRunning command [\'udevadm\', \'settle\'] with allowed return codes [0] (shell=False, capture=False)\nRunning command [\'partprobe\', \'/dev/sde\'] with allowed return codes [0, 1] (shell=False, capture=False)\nRunning command [\'udevadm\', \'settle\'] with allowed return codes [0] (shell=False, capture=False)\nprevious partition number for \'sda-part2\' found to be \'1\'\nadding partition \'sda-part2\' to disk \'sda\'\nRunning command [\'sgdisk\', \'--new\', \'2:18432:69300223\', \'--typecode=2:8300\', \'/dev/sde\'] with allowed return codes [0] (shell=False, capture=False)\nCould not create partition 2 from 18432 to 69300223\nCould not change partition 2\'s type code to 8300!\nError encountered; not saving changes.\nAn error occured handling \'sda-part2\': ProcessExecutionError - Unexpected error while running command.\nCommand: [\'sgdisk\', \'--new\', \'2:18432:69300223\', \'--typecode=2:8300\', \'/dev/sde\']\nExit code: 4\nReason: -\nStdout: \'\'\nStderr: \'\'\nTraceback (most recent call last):\n File "/curtin/curtin/commands/main.py", line 209, in main\n ret = args.func(args)\n File "/curtin/curtin/commands/block_meta.py", line 62, in block_meta\n meta_custom(args)\n File "/curtin/curtin/commands/block_meta.py", line 1083, in meta_custom\n handler(command, storage_config_dict)\n File "/curtin/curtin/commands/block_meta.py", line 643, in partition_handler\n util.subp(cmd)\n File "/curtin/curtin/util.py", line 140, in subp\n return _subp(*args, **kwargs)\n File "/curtin/curtin/util.py", line 90, in _subp\n cmd=args)\ncurtin.util.ProcessExecutionError: Unexpected error while running command.\nCommand: [\'sgdisk\', \'--new\', \'2:18432:69300223\', \'--typecode=2:8300\', \'/dev/sde\']\nExit code: 4\nReason: -\nStdout: \'\'\nStderr: \'\'\nUnexpected error while running command.\nCommand: [\'sgdisk\', \'--new\', \'2:18432:69300223\', \'--typecode=2:8300\', \'/dev/sde\']\nExit code: 4\nReason: -\nStdout: \'\'\nStderr: \'\'\n'
Stderr: ''

[Andres Rodriguez (andreserl)]

Also, if the machine is in Failed Deployment, can we please get the curtin config of that node:

maas <user> node get-curtin-config <system-id>

[Ryan Harper (raharper)]

Isn't this related to multipathing?

[Blake Rouse (blake-rouse)]

Also:

maas <user> block-devices read <system-id>

[Ryan Beisner (1chb1n)]

maas <user> node get-curtin-config <system-id>:
http://paste.ubuntu.com/15578365/

maas <user> block-devices read <system-id>:
http://paste.ubuntu.com/15578362/

[Ryan Harper (raharper)]

The sda vs sde can and will happen; it's likely that /dev/disk/by-id/<serial of "sda"> is symlinked to sde; What matters it the 'serial' or 'path' key in the structure.

I don;t think that's the issue here.

Adding the first partition (prep) is successful. Then when adding the second partition we fail, sgdisk says:

Could not create partition 2 from 18432 to 69300223

That's *usually* a miscalculation on sizes and offsets and total disk size.
Given that this is prep partitions, maybe we can recheck the values.

Total disk size:

283794997248B

The ending sector of the second partition, 69300223 * 4k is > than the total disk size.
This is what failed.

The offset 18432 looks odd; prep partition is 8MiB, the offset sent is 4MiB, though curtin only applied 1M offset (256.0). None-the less, the next start point should be
1M + 8M, or at most 4M + 8M , we can see the end sector of the prep partition is 2303 from the sgdisk command, so we should see:

2:2304.0:69284096.0

69284096.0 * 4096 = 283787657216.0

283787657216.0 > 283794997248
False

so, it fits.

Now, how did the second partition offset get so far off. curtin reads size and offset from /sys/class/block/<disk>/partition/{start, size}.

So, the questions now are:

1. did we read the correct path?

Yes, we've wiped all drives prior to creating new partitions, so no other disk will have partitions.
Then we see the message: previous partition number for 'sda-part2' found to be '1';
curtin has found /sys/class/block/sde/sde1, and will now read
/sys/class/block/sde/sde1/{size, start}

2. What values were in those fields.
curtin calculates the start of the new partition with:

offset_sectors = previous_start + previous_size

where previous_start is the value in /sys/class/block/sde/sde1/start and
previous_size is the value in /sys/class/block/sde/sde1/size

[Scott Moser (smoser)]

It's worth trying to recommission the node. Sorry if this is noise. But axew in that same maas send to do fine.

[Ryan Harper (raharper)]

In particular, it's at least worth seeing if the block devices are also 64k
block_size as well.

Looking at axew, I wonder what the curtin storage config looks like, I see
this for the inital partition creation:

['sgdisk', '--new', '1:2048:18431', '--typecode=1:4100', '/dev/sdd'] with
allowed return codes [0] (shell=False, capture=False)

which is a prep partition (type 4100) using 512b sectors (1M offset / 512)
== 2048
the end is 2048 + (8 * 1024 * 1024) / 512 - 1 sector)
2048 + (8388608 - 512) / 512
2048 + 16383
18431

And the second partition command:

previous partition number for 'sda-part2' found to be '1'
adding partition 'sda-part2' to disk 'sda'
Running command ['sgdisk', '--new', '2:18432:554272767',
'--typecode=2:8300', '/dev/sdd'] with allowed return codes [0]
(shell=False, capture=False)

This runs the same code which finds the previous partition, then reads size
and start from sysfs. It finds
the same 18431 ending sector, adds one and we start the new section with
18432.

On the failing node,
the default sector size is 4k, we take our 1M offset / 4096, and we get 256
on the offset.

But when calculating the second partition, somehow the logical_size is 512,
otherwise we'd
get a different starting sector.

18432 is correct for bs=512, but should be 2304 for 4k.

I confirmed the python math, /sys/block/sde/queue/logical_block_size is
returning 4096
then our initial offset is:

alignment_offset = (1 << 20) / logical_block_size_bytes
logical_block_size_bytes = 4096
(1 << 20) / logical_block_size_bytes
256.0

However, when we read /sys/block/sde/sde1/start

we're getting start=2048 size=16384, 18432

That's bizarre, and I'd think some sort of kernel bug unless there's some
setting that controlling that value.

And it's really odd to have it set that high but the other values in sysfs
returning based on 512 instead of 4096

On Fri, Apr 1, 2016 at 2:38 PM, Scott Moser <email address hidden> wrote:

> It's worth trying to recommission the node. Sorry if this is noise. But
> axew in that same maas send to do fine.
>
> --
> You received this bug notification because you are subscribed to curtin.
> Matching subscriptions: curtin-bugs-all
> https://bugs.launchpad.net/bugs/1564977
>
> Title:
> ppc64el 1.9.1 tries and fails sgdisk on the wrong device
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/curtin/+bug/1564977/+subscriptions
>

[Ryan Beisner (1chb1n)]

FYI - I iterated axew last night while it was released and it too failed to deploy a few times. We eventually got it into a deployed state.

I've recommissioned and completely deleted and re-added the other 3 machines but still the failure persists (seemingly randomly).

> On Apr 1, 2016, at 2:38 PM, Scott Moser <email address hidden> wrote:
>
> It's worth trying to recommission the node. Sorry if this is noise. But
> axew in that same maas send to do fine.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1564977
>
> Title:
> ppc64el 1.9.1 tries and fails sgdisk on the wrong device
>
> Status in curtin:
> New
> Status in MAAS:
> Incomplete
>
> Bug description:
> We experience a very high rate of failure in deploying ppc64el via the
> MAAS web ui, for any Ubuntu daily image release (Trusty, Wily,
> Xenial).
>
> In 15 attempts, I was able to get 3 machines to a 'deployed' state.
> We have 4 Power8 machines and they all exhibit this behavior.
>
> With Trusty and Wily, when the unit fails to deploy, the machines are
> accessible.
>
> With Xenial, when the unit fails to deploy, the machine is not
> reachable via network.
>
> In all cases, the oobm serial console shows the machine at the login
> prompt.
>
> We are also observing that the ipmi version, although initially set to
> version 2.0, is reverting to 1.5 after a unit fails deployment and is
> released. (https://bugs.launchpad.net/maas/+bug/1556219) This means
> that all subsequent power commands will fail until the user manually
> sets the ipmi back to 2.0 for the machines.
>
> What is interesting/odd in the machine output, is that it's trying
> very hard to do operations on /dev/sde when /dev/sda is what I would
> expect.
>
>
> # Storage looks like this:
>
> File systems
> Name Size Mountpoint File system
> sda-part2 283.8 GB / ext4
>
> Available disks and partitions
> Name Model Serial BootSizeDevice TypeFile systemTags
>
> sdb
> 283.8 GB Physical rotary
>
> sdc
> 283.8 GB Physical rotary
>
> sdd
> 283.8 GB Physical rotary
>
> sde
> 283.8 GB Physical rotary
>
> Used disks and partitions
> Name Model Serial BootDevice typeUsed for
> sda PhysicalGPT partitioned with 1 partition
> sda-part2 Partitionext4 formatted filesystem mounted at /
>
>
> # Network looks like this:
> eth0
> Physical 10.X.X.X (Auto assign)
>
> eth1
> Physical (Unconfigured)
>
> eth2
> Physical (Unconfigured)
>
> eth3
> Physical (Unconfigured)
>
> eth4
> Physical (Unconfigured)
>
> eth5
> Physical (Unconfigured)
>
>
>
> # Machine output
>
> Running command ['partprobe', '/dev/sde'] with allowed return codes [0, 1] (shell=False, capture=False)
> Running command ['udevadm', 'settle'] with allowed return codes [0] (shell=False, capture=False)
> Running command ['mdadm', '--assemble', '--scan'] with allowed return codes [0, 1, 2] (shell=False, capture=True)
> Running command ['udevadm', 'settle'] with allowed return codes [0] (shell=False, capture=False)
> clear_holders running on '/sys/block/sde/sde1', with holders '[]'
> Running comm...

[Ryan Harper (raharper)]

It's not random; one or more of the drives on the various machines is set to use 4k sectors.

I've reproduced the failure by forcing qemu to run the target disks in 4k mode (logical_block_size=4k,physical_block_size=4k).

Then we fail the same way. The root of the issue is that values from /sys/class/block/XXX/size are *always* reported in 512b sizes, no matter if the devices is in 4k or some other size.

This exposed a bug in the calcuation of where to start and end for any partition past the first one.

The fix is to convert the size/start to bytes and divide by the logical_block_size_bytes, to return to drive native sector count.

[Ryan Harper (raharper)]

What a right pain to track this one down. Note, you'll need to disable any 4k devices, or ensure they support 512b logical sector size (most do); otherwise, those disks (if selected as your boot drive) will only function if you're booting via UEFI.

Now, let's say you want to do that. Here's what's needed.

1) You EFI partition must be at least 512MB, any smaller and FAT32 with 4k sector-size will eat itself and not function; but only warn you (non-fatal error as far as mkfs.vfat is concerned)
2) curtin needs to learn how to pass in -S 4096 (to set the sector size for the filesystem)
3) curtin needs to learn from partman-efi that when one passes -S 4096 to mkfs.vfat, it doesn't calculate it's sectors per cluster correctly, and must also pass in '-s 1' otherwise UEFI bootloader can't read the filesystem
4) curtin needs to allow test-case to specify the logical and physical blocksize and apply that on a per-device basis.

Will link a branch soon to resolve this issue. In the mean time, see about disabling those 4k Advanced Format drives (or enabling legacy compat mode logical=512b/physical=4096b.

[Ryan Beisner (1chb1n)]

Thanks for all your work on this. We've worked around this by configuring the affected nodes in the MAAS UI to ignore the 4K drives (boot to /dev/sdb...).

[Andres Rodriguez (andreserl)]

Just to clarify on what would MAAS need.

13:42 < rharper> if you know it's 4k, you need to ensure that the partition is at least 512MB, no smaller; it can only be uefi
13:42 < rharper> curtin doesn't need to know the block-size, we;ll figure it out
13:43 < rharper> the EFI partition, and GPT only; that should be sufficient

[Blake Rouse (blake-rouse)]

After talking with rharper this is more of a curtin issue than a MAAS. Marking as Invalid for MAAS at the moment after the curtin fix if MAAS is causing any issues will re-open.

[Ryan Harper (raharper)]

Blake,

I included some checks of values when running in 512b and 4k mode of blockdev:

""" validate maas setting use of blockdev
        --getsz get size in 512-byte sectors
        --getss get logical block (sector) size
        --getpbsz get physical block (sector) size
        --getbsz get blocksize
"""

MAAS should be using --getss (logical); that is what curtin uses to convert disk/partition sizes to sectors.

On a 512b logical_sector , 512b physical_sector VM: we get the following output against a 10G virtio disk:

vda_blockdev_getbsz: 4096
vda_blockdev_getpbsz: 512
vda_blockdev_getss: 512
vda_blockdev_getsz: 20971520

The same disk with 4k logical / 4k physical is:

vda_blockdev_getbsz: 4096
vda_blockdev_getpbsz: 4096
vda_blockdev_getss: 4096
vda_blockdev_getsz: 20971520

[Scott Moser (smoser)]

fixed in curtin trunk at revno 383.

[Scott Moser (smoser)]

fix-released in yakkety and on its way to xenial, wily, trusty under bug 1577872

[Scott Moser (smoser)]

This bug is believed to be fixed in curtin in 17.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.