| 2016-09-21 19:36:23 |
Matt Bearup |
bug |
|
|
added bug |
| 2016-10-12 15:11:02 |
Scott Moser |
description |
The symptom is similar to 1611074 but the cause is different. In this case it seems there is an error accessing /dev/sdb1 when lsblk is run, possibly because sgdisk isn't done creating the partition. The specific error message is "/dev/sdb1: not a block device." A simple wait and retry here may resolve the issue.
util.py[DEBUG]: Running command ['/sbin/sgdisk', '-p', '/dev/sdb'] with allowed return codes [0] (shell=False, capture=True)
cc_disk_setup.py[DEBUG]: Device partitioning layout matches
util.py[DEBUG]: Creating partition on /dev/disk/cloud/azure_resource took 0.056 seconds
cc_disk_setup.py[DEBUG]: setting up filesystems: [{'filesystem': 'ext4', 'device': 'ephemeral0.1', 'replace_fs': 'ntfs'}]
cc_disk_setup.py[DEBUG]: ephemeral0.1 is mapped to disk=/dev/disk/cloud/azure_resource part=1
cc_disk_setup.py[DEBUG]: Creating new filesystem.
cc_disk_setup.py[DEBUG]: Checking /dev/sdb against default devices
cc_disk_setup.py[DEBUG]: Manual request of partition 1 for /dev/sdb1
cc_disk_setup.py[DEBUG]: Checking device /dev/sdb1
util.py[DEBUG]: Running command ['/sbin/blkid', '-c', '/dev/null', '/dev/sdb1'] with allowed return codes [0, 2] (shell=False, capture=True)
cc_disk_setup.py[DEBUG]: Device /dev/sdb1 has None None
cc_disk_setup.py[DEBUG]: Device /dev/sdb1 is cleared for formating
cc_disk_setup.py[DEBUG]: File system None will be created on /dev/sdb1
util.py[DEBUG]: Running command ['/bin/lsblk', '--pairs', '--output', 'NAME,TYPE,FSTYPE,LABEL', '/dev/sdb1', '--nodeps'] with allowed return codes [0] (shell=False, capture=True)
util.py[DEBUG]: Creating fs for /dev/disk/cloud/azure_resource took 0.008 seconds
util.py[WARNING]: Failed during filesystem operation#012Failed during disk check for /dev/sdb1#012Unexpected error while running command.#012Command: ['/bin/lsblk', '--pairs', '--output', 'NAME,TYPE,FSTYPE,LABEL', '/dev/sdb1', '--nodeps']#012Exit code: 32#012Reason: -#012Stdout: ''#012Stderr: 'lsblk: /dev/sdb1: not a block device\n' |
The symptom is similar to bug 1611074 but the cause is different. In this case it seems there is an error accessing /dev/sdb1 when lsblk is run, possibly because sgdisk isn't done creating the partition. The specific error message is "/dev/sdb1: not a block device." A simple wait and retry here may resolve the issue.
util.py[DEBUG]: Running command ['/sbin/sgdisk', '-p', '/dev/sdb'] with allowed return codes [0] (shell=False, capture=True)
cc_disk_setup.py[DEBUG]: Device partitioning layout matches
util.py[DEBUG]: Creating partition on /dev/disk/cloud/azure_resource took 0.056 seconds
cc_disk_setup.py[DEBUG]: setting up filesystems: [{'filesystem': 'ext4', 'device': 'ephemeral0.1', 'replace_fs': 'ntfs'}]
cc_disk_setup.py[DEBUG]: ephemeral0.1 is mapped to disk=/dev/disk/cloud/azure_resource part=1
cc_disk_setup.py[DEBUG]: Creating new filesystem.
cc_disk_setup.py[DEBUG]: Checking /dev/sdb against default devices
cc_disk_setup.py[DEBUG]: Manual request of partition 1 for /dev/sdb1
cc_disk_setup.py[DEBUG]: Checking device /dev/sdb1
util.py[DEBUG]: Running command ['/sbin/blkid', '-c', '/dev/null', '/dev/sdb1'] with allowed return codes [0, 2] (shell=False, capture=True)
cc_disk_setup.py[DEBUG]: Device /dev/sdb1 has None None
cc_disk_setup.py[DEBUG]: Device /dev/sdb1 is cleared for formating
cc_disk_setup.py[DEBUG]: File system None will be created on /dev/sdb1
util.py[DEBUG]: Running command ['/bin/lsblk', '--pairs', '--output', 'NAME,TYPE,FSTYPE,LABEL', '/dev/sdb1', '--nodeps'] with allowed return codes [0] (shell=False, capture=True)
util.py[DEBUG]: Creating fs for /dev/disk/cloud/azure_resource took 0.008 seconds
util.py[WARNING]: Failed during filesystem operation#012Failed during disk check for /dev/sdb1#012Unexpected error while running command.#012Command: ['/bin/lsblk', '--pairs', '--output', 'NAME,TYPE,FSTYPE,LABEL', '/dev/sdb1', '--nodeps']#012Exit code: 32#012Reason: -#012Stdout: ''#012Stderr: 'lsblk: /dev/sdb1: not a block device\n' |
|
| 2016-10-21 03:09:05 |
Scott Moser |
bug task added |
|
cloud-init (Ubuntu) |
|
| 2016-10-21 03:09:12 |
Scott Moser |
cloud-init: status |
New |
Confirmed |
|
| 2016-10-21 03:09:14 |
Scott Moser |
cloud-init (Ubuntu): status |
New |
Confirmed |
|
| 2016-10-21 03:09:17 |
Scott Moser |
cloud-init: importance |
Undecided |
Medium |
|
| 2016-10-21 03:09:19 |
Scott Moser |
cloud-init (Ubuntu): importance |
Undecided |
Medium |
|
| 2016-10-21 16:50:28 |
Scott Moser |
attachment added |
|
script should reproduce failure https://bugs.launchpad.net/cloud-init/+bug/1626243/+attachment/4764911/+files/go-short.sh |
|
| 2016-10-25 21:02:12 |
Scott Moser |
cloud-init: status |
Confirmed |
Fix Committed |
|
| 2016-10-25 22:17:38 |
Launchpad Janitor |
cloud-init (Ubuntu): status |
Confirmed |
Fix Released |
|
| 2016-11-07 19:59:20 |
Scott Moser |
nominated for series |
|
Ubuntu Yakkety |
|
| 2016-11-07 19:59:20 |
Scott Moser |
bug task added |
|
cloud-init (Ubuntu Yakkety) |
|
| 2016-11-07 19:59:20 |
Scott Moser |
nominated for series |
|
Ubuntu Zesty |
|
| 2016-11-07 19:59:20 |
Scott Moser |
bug task added |
|
cloud-init (Ubuntu Zesty) |
|
| 2016-11-07 19:59:20 |
Scott Moser |
nominated for series |
|
Ubuntu Xenial |
|
| 2016-11-07 19:59:20 |
Scott Moser |
bug task added |
|
cloud-init (Ubuntu Xenial) |
|
| 2016-11-07 19:59:28 |
Scott Moser |
cloud-init (Ubuntu Xenial): status |
New |
Confirmed |
|
| 2016-11-07 19:59:31 |
Scott Moser |
cloud-init (Ubuntu Yakkety): status |
New |
Confirmed |
|
| 2016-11-07 19:59:34 |
Scott Moser |
cloud-init (Ubuntu Xenial): importance |
Undecided |
Medium |
|
| 2016-11-07 19:59:36 |
Scott Moser |
cloud-init (Ubuntu Yakkety): importance |
Undecided |
Medium |
|
| 2016-11-07 20:50:02 |
Scott Moser |
description |
The symptom is similar to bug 1611074 but the cause is different. In this case it seems there is an error accessing /dev/sdb1 when lsblk is run, possibly because sgdisk isn't done creating the partition. The specific error message is "/dev/sdb1: not a block device." A simple wait and retry here may resolve the issue.
util.py[DEBUG]: Running command ['/sbin/sgdisk', '-p', '/dev/sdb'] with allowed return codes [0] (shell=False, capture=True)
cc_disk_setup.py[DEBUG]: Device partitioning layout matches
util.py[DEBUG]: Creating partition on /dev/disk/cloud/azure_resource took 0.056 seconds
cc_disk_setup.py[DEBUG]: setting up filesystems: [{'filesystem': 'ext4', 'device': 'ephemeral0.1', 'replace_fs': 'ntfs'}]
cc_disk_setup.py[DEBUG]: ephemeral0.1 is mapped to disk=/dev/disk/cloud/azure_resource part=1
cc_disk_setup.py[DEBUG]: Creating new filesystem.
cc_disk_setup.py[DEBUG]: Checking /dev/sdb against default devices
cc_disk_setup.py[DEBUG]: Manual request of partition 1 for /dev/sdb1
cc_disk_setup.py[DEBUG]: Checking device /dev/sdb1
util.py[DEBUG]: Running command ['/sbin/blkid', '-c', '/dev/null', '/dev/sdb1'] with allowed return codes [0, 2] (shell=False, capture=True)
cc_disk_setup.py[DEBUG]: Device /dev/sdb1 has None None
cc_disk_setup.py[DEBUG]: Device /dev/sdb1 is cleared for formating
cc_disk_setup.py[DEBUG]: File system None will be created on /dev/sdb1
util.py[DEBUG]: Running command ['/bin/lsblk', '--pairs', '--output', 'NAME,TYPE,FSTYPE,LABEL', '/dev/sdb1', '--nodeps'] with allowed return codes [0] (shell=False, capture=True)
util.py[DEBUG]: Creating fs for /dev/disk/cloud/azure_resource took 0.008 seconds
util.py[WARNING]: Failed during filesystem operation#012Failed during disk check for /dev/sdb1#012Unexpected error while running command.#012Command: ['/bin/lsblk', '--pairs', '--output', 'NAME,TYPE,FSTYPE,LABEL', '/dev/sdb1', '--nodeps']#012Exit code: 32#012Reason: -#012Stdout: ''#012Stderr: 'lsblk: /dev/sdb1: not a block device\n' |
=== Begin SRU Template ===
[Impact]
There is a race condition that occurs when cloud-init tries to partition a
block device (/dev/sdb) and then put a filesystem on a partition on it. It is
possible that cloud-init tries to run mkfs on /dev/sdb1 after partitioning the
device /dev/sdb but before the partition device node '/dev/sdb1' exists.
When this race condition occurs, cloud-init will fail to make the "ephemeral"
device available to the user on Azure.
[Test Case]
A reliable reproduce test case is hard to come by here. The failure case
is believed to be well understood.
[Regression Potential]
There should be very little chance for regression, as essentially all the change
does is change:
1. sgdisk -n 1:0:0 /dev/sdb
2. mkfs.ext4 /dev/sdb1
to
1. sgdisk -n 1:0:0 /dev/sdb
1a udevadm settle
1b blockdev --rereadpt
1c usdevadm settle
2. mkfs.ext4 /dev/sdb1
The steps '1b' and '1c' above are not necessary, but were present already in
the method. They serve here as additional wait.
[Other Info]
The change that fixes this is viewable at [1]. For context, viewin all of
cc_disk_setup.py [2]. Basically we just add a call to read_parttbl [3] to
exec_mkpart_gpt after invoking a sgdisk command that partitions a disk.
read_partbl basically does a udevadm settle which fixes the race condition that
was seen.
[1] https://git.launchpad.net/cloud-init/commit/?id=29348af1c889931e8973f8fc8cb090c063316f7a
[2] https://git.launchpad.net/cloud-init/tree/cloudinit/config/cc_disk_setup.py?id=29348af1c889931e8973f8fc8cb090c063316f7a
[3] https://git.launchpad.net/cloud-init/tree/cloudinit/config/cc_disk_setup.py?id=29348af1c889931e8973f8fc8cb090c063316f7a#n674
=== End SRU Template ===
The symptom is similar to bug 1611074 but the cause is different. In this case it seems there is an error accessing /dev/sdb1 when lsblk is run, possibly because sgdisk isn't done creating the partition. The specific error message is "/dev/sdb1: not a block device." A simple wait and retry here may resolve the issue.
util.py[DEBUG]: Running command ['/sbin/sgdisk', '-p', '/dev/sdb'] with allowed return codes [0] (shell=False, capture=True)
cc_disk_setup.py[DEBUG]: Device partitioning layout matches
util.py[DEBUG]: Creating partition on /dev/disk/cloud/azure_resource took 0.056 seconds
cc_disk_setup.py[DEBUG]: setting up filesystems: [{'filesystem': 'ext4', 'device': 'ephemeral0.1', 'replace_fs': 'ntfs'}]
cc_disk_setup.py[DEBUG]: ephemeral0.1 is mapped to disk=/dev/disk/cloud/azure_resource part=1
cc_disk_setup.py[DEBUG]: Creating new filesystem.
cc_disk_setup.py[DEBUG]: Checking /dev/sdb against default devices
cc_disk_setup.py[DEBUG]: Manual request of partition 1 for /dev/sdb1
cc_disk_setup.py[DEBUG]: Checking device /dev/sdb1
util.py[DEBUG]: Running command ['/sbin/blkid', '-c', '/dev/null', '/dev/sdb1'] with allowed return codes [0, 2] (shell=False, capture=True)
cc_disk_setup.py[DEBUG]: Device /dev/sdb1 has None None
cc_disk_setup.py[DEBUG]: Device /dev/sdb1 is cleared for formating
cc_disk_setup.py[DEBUG]: File system None will be created on /dev/sdb1
util.py[DEBUG]: Running command ['/bin/lsblk', '--pairs', '--output', 'NAME,TYPE,FSTYPE,LABEL', '/dev/sdb1', '--nodeps'] with allowed return codes [0] (shell=False, capture=True)
util.py[DEBUG]: Creating fs for /dev/disk/cloud/azure_resource took 0.008 seconds
util.py[WARNING]: Failed during filesystem operation#012Failed during disk check for /dev/sdb1#012Unexpected error while running command.#012Command: ['/bin/lsblk', '--pairs', '--output', 'NAME,TYPE,FSTYPE,LABEL', '/dev/sdb1', '--nodeps']#012Exit code: 32#012Reason: -#012Stdout: ''#012Stderr: 'lsblk: /dev/sdb1: not a block device\n' |
|
| 2016-11-15 23:31:11 |
Steve Langasek |
cloud-init (Ubuntu Xenial): status |
Confirmed |
Fix Committed |
|
| 2016-11-15 23:31:14 |
Steve Langasek |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2016-11-15 23:31:16 |
Steve Langasek |
bug |
|
|
added subscriber SRU Verification |
| 2016-11-15 23:31:22 |
Steve Langasek |
tags |
|
verification-needed |
|
| 2016-11-16 15:28:57 |
Scott Moser |
description |
=== Begin SRU Template ===
[Impact]
There is a race condition that occurs when cloud-init tries to partition a
block device (/dev/sdb) and then put a filesystem on a partition on it. It is
possible that cloud-init tries to run mkfs on /dev/sdb1 after partitioning the
device /dev/sdb but before the partition device node '/dev/sdb1' exists.
When this race condition occurs, cloud-init will fail to make the "ephemeral"
device available to the user on Azure.
[Test Case]
A reliable reproduce test case is hard to come by here. The failure case
is believed to be well understood.
[Regression Potential]
There should be very little chance for regression, as essentially all the change
does is change:
1. sgdisk -n 1:0:0 /dev/sdb
2. mkfs.ext4 /dev/sdb1
to
1. sgdisk -n 1:0:0 /dev/sdb
1a udevadm settle
1b blockdev --rereadpt
1c usdevadm settle
2. mkfs.ext4 /dev/sdb1
The steps '1b' and '1c' above are not necessary, but were present already in
the method. They serve here as additional wait.
[Other Info]
The change that fixes this is viewable at [1]. For context, viewin all of
cc_disk_setup.py [2]. Basically we just add a call to read_parttbl [3] to
exec_mkpart_gpt after invoking a sgdisk command that partitions a disk.
read_partbl basically does a udevadm settle which fixes the race condition that
was seen.
[1] https://git.launchpad.net/cloud-init/commit/?id=29348af1c889931e8973f8fc8cb090c063316f7a
[2] https://git.launchpad.net/cloud-init/tree/cloudinit/config/cc_disk_setup.py?id=29348af1c889931e8973f8fc8cb090c063316f7a
[3] https://git.launchpad.net/cloud-init/tree/cloudinit/config/cc_disk_setup.py?id=29348af1c889931e8973f8fc8cb090c063316f7a#n674
=== End SRU Template ===
The symptom is similar to bug 1611074 but the cause is different. In this case it seems there is an error accessing /dev/sdb1 when lsblk is run, possibly because sgdisk isn't done creating the partition. The specific error message is "/dev/sdb1: not a block device." A simple wait and retry here may resolve the issue.
util.py[DEBUG]: Running command ['/sbin/sgdisk', '-p', '/dev/sdb'] with allowed return codes [0] (shell=False, capture=True)
cc_disk_setup.py[DEBUG]: Device partitioning layout matches
util.py[DEBUG]: Creating partition on /dev/disk/cloud/azure_resource took 0.056 seconds
cc_disk_setup.py[DEBUG]: setting up filesystems: [{'filesystem': 'ext4', 'device': 'ephemeral0.1', 'replace_fs': 'ntfs'}]
cc_disk_setup.py[DEBUG]: ephemeral0.1 is mapped to disk=/dev/disk/cloud/azure_resource part=1
cc_disk_setup.py[DEBUG]: Creating new filesystem.
cc_disk_setup.py[DEBUG]: Checking /dev/sdb against default devices
cc_disk_setup.py[DEBUG]: Manual request of partition 1 for /dev/sdb1
cc_disk_setup.py[DEBUG]: Checking device /dev/sdb1
util.py[DEBUG]: Running command ['/sbin/blkid', '-c', '/dev/null', '/dev/sdb1'] with allowed return codes [0, 2] (shell=False, capture=True)
cc_disk_setup.py[DEBUG]: Device /dev/sdb1 has None None
cc_disk_setup.py[DEBUG]: Device /dev/sdb1 is cleared for formating
cc_disk_setup.py[DEBUG]: File system None will be created on /dev/sdb1
util.py[DEBUG]: Running command ['/bin/lsblk', '--pairs', '--output', 'NAME,TYPE,FSTYPE,LABEL', '/dev/sdb1', '--nodeps'] with allowed return codes [0] (shell=False, capture=True)
util.py[DEBUG]: Creating fs for /dev/disk/cloud/azure_resource took 0.008 seconds
util.py[WARNING]: Failed during filesystem operation#012Failed during disk check for /dev/sdb1#012Unexpected error while running command.#012Command: ['/bin/lsblk', '--pairs', '--output', 'NAME,TYPE,FSTYPE,LABEL', '/dev/sdb1', '--nodeps']#012Exit code: 32#012Reason: -#012Stdout: ''#012Stderr: 'lsblk: /dev/sdb1: not a block device\n' |
=== Begin SRU Template ===
[Impact]
There is a race condition that occurs when cloud-init tries to partition a
block device (/dev/sdb) and then put a filesystem on a partition on it. It is
possible that cloud-init tries to run mkfs on /dev/sdb1 after partitioning the
device /dev/sdb but before the partition device node '/dev/sdb1' exists.
When this race condition occurs, cloud-init will fail to make the "ephemeral"
device available to the user on Azure.
[Test Case]
A reliable reproduce test case is hard to come by here. The failure case
is believed to be well understood.
[Regression Potential]
There should be very little chance for regression, as essentially all the change
does is change:
1. sgdisk -n 1:0:0 /dev/sdb
2. mkfs.ext4 /dev/sdb1
to
1. sgdisk -n 1:0:0 /dev/sdb
1a udevadm settle
1b blockdev --rereadpt
1c udevadm settle
2. mkfs.ext4 /dev/sdb1
The steps '1b' and '1c' above are not necessary, but were present already in
the method. They serve here as additional wait.
[Other Info]
The change that fixes this is viewable at [1]. For context, viewin all of
cc_disk_setup.py [2]. Basically we just add a call to read_parttbl [3] to
exec_mkpart_gpt after invoking a sgdisk command that partitions a disk.
read_partbl basically does a udevadm settle which fixes the race condition that
was seen.
[1] https://git.launchpad.net/cloud-init/commit/?id=29348af1c889931e8973f8fc8cb090c063316f7a
[2] https://git.launchpad.net/cloud-init/tree/cloudinit/config/cc_disk_setup.py?id=29348af1c889931e8973f8fc8cb090c063316f7a
[3] https://git.launchpad.net/cloud-init/tree/cloudinit/config/cc_disk_setup.py?id=29348af1c889931e8973f8fc8cb090c063316f7a#n674
=== End SRU Template ===
The symptom is similar to bug 1611074 but the cause is different. In this case it seems there is an error accessing /dev/sdb1 when lsblk is run, possibly because sgdisk isn't done creating the partition. The specific error message is "/dev/sdb1: not a block device." A simple wait and retry here may resolve the issue.
util.py[DEBUG]: Running command ['/sbin/sgdisk', '-p', '/dev/sdb'] with allowed return codes [0] (shell=False, capture=True)
cc_disk_setup.py[DEBUG]: Device partitioning layout matches
util.py[DEBUG]: Creating partition on /dev/disk/cloud/azure_resource took 0.056 seconds
cc_disk_setup.py[DEBUG]: setting up filesystems: [{'filesystem': 'ext4', 'device': 'ephemeral0.1', 'replace_fs': 'ntfs'}]
cc_disk_setup.py[DEBUG]: ephemeral0.1 is mapped to disk=/dev/disk/cloud/azure_resource part=1
cc_disk_setup.py[DEBUG]: Creating new filesystem.
cc_disk_setup.py[DEBUG]: Checking /dev/sdb against default devices
cc_disk_setup.py[DEBUG]: Manual request of partition 1 for /dev/sdb1
cc_disk_setup.py[DEBUG]: Checking device /dev/sdb1
util.py[DEBUG]: Running command ['/sbin/blkid', '-c', '/dev/null', '/dev/sdb1'] with allowed return codes [0, 2] (shell=False, capture=True)
cc_disk_setup.py[DEBUG]: Device /dev/sdb1 has None None
cc_disk_setup.py[DEBUG]: Device /dev/sdb1 is cleared for formating
cc_disk_setup.py[DEBUG]: File system None will be created on /dev/sdb1
util.py[DEBUG]: Running command ['/bin/lsblk', '--pairs', '--output', 'NAME,TYPE,FSTYPE,LABEL', '/dev/sdb1', '--nodeps'] with allowed return codes [0] (shell=False, capture=True)
util.py[DEBUG]: Creating fs for /dev/disk/cloud/azure_resource took 0.008 seconds
util.py[WARNING]: Failed during filesystem operation#012Failed during disk check for /dev/sdb1#012Unexpected error while running command.#012Command: ['/bin/lsblk', '--pairs', '--output', 'NAME,TYPE,FSTYPE,LABEL', '/dev/sdb1', '--nodeps']#012Exit code: 32#012Reason: -#012Stdout: ''#012Stderr: 'lsblk: /dev/sdb1: not a block device\n' |
|
| 2016-11-22 01:58:08 |
Scott Moser |
tags |
verification-needed |
verification-done |
|
| 2016-11-24 20:00:36 |
Launchpad Janitor |
cloud-init (Ubuntu Xenial): status |
Fix Committed |
Fix Released |
|
| 2016-11-24 20:01:03 |
Adam Conrad |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2016-11-29 23:42:06 |
Chris Halse Rogers |
cloud-init (Ubuntu Yakkety): status |
Confirmed |
Fix Committed |
|
| 2016-11-29 23:42:09 |
Chris Halse Rogers |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2016-11-29 23:42:21 |
Chris Halse Rogers |
tags |
verification-done |
|
|
| 2016-11-29 23:42:22 |
Chris Halse Rogers |
tags |
|
verification-needed |
|
| 2016-12-19 17:41:33 |
Scott Moser |
tags |
verification-needed |
verification-done |
|
| 2016-12-21 01:05:10 |
Launchpad Janitor |
cloud-init (Ubuntu Yakkety): status |
Fix Committed |
Fix Released |
|
| 2016-12-23 17:39:35 |
Scott Moser |
cloud-init: status |
Fix Committed |
Fix Released |
|
| 2023-05-10 17:03:15 |
James Falcon |
bug watch added |
|
https://github.com/canonical/cloud-init/issues/2733 |
|
