Failed deployment: FileNotFoundError: [Errno 2] No such file or directory: '/sys/class/block/bcache0/bcache0p1/slaves'

Bug #1811117 reported by Ashley Lai on 2019-01-09
14
This bug affects 1 person
Affects Status Importance Assigned to Milestone
curtin
Critical
Ryan Harper

Bug Description

Bundle deployment failed with juju status shows the message:
  Failed deployment: 'cloudinit' running modules for final

Jan 9 06:35:17 beartic cloud-init[2993]: Traceback (most recent call last):
Jan 9 06:35:17 beartic cloud-init[2993]: File "/curtin/curtin/commands/main.py", line 201, in main
Jan 9 06:35:17 beartic cloud-init[2993]: ret = args.func(args)
Jan 9 06:35:17 beartic cloud-init[2993]: File "/curtin/curtin/commands/block_meta.py", line 58, in block_meta
Jan 9 06:35:17 beartic cloud-init[2993]: meta_custom(args)
Jan 9 06:35:17 beartic cloud-init[2993]: File "/curtin/curtin/commands/block_meta.py", line 1471, in meta_custom
Jan 9 06:35:17 beartic cloud-init[2993]: clear_holders.clear_holders(disk_paths)
Jan 9 06:35:17 beartic cloud-init[2993]: File "/curtin/curtin/block/clear_holders.py", line 587, in clear_holders
Jan 9 06:35:17 beartic cloud-init[2993]: shutdown_function(dev_info['device'])
Jan 9 06:35:17 beartic cloud-init[2993]: File "/curtin/curtin/block/clear_holders.py", line 134, in shutdown_bcache
Jan 9 06:35:17 beartic cloud-init[2993]: os.listdir(os.path.join(device, 'slaves'))]
Jan 9 06:35:17 beartic cloud-init[2993]: FileNotFoundError: [Errno 2] No such file or directory: '/sys/class/block/bcache0/bcache0p1/slaves'
Jan 9 06:35:17 beartic cloud-init[2993]: [Errno 2] No such file or directory: '/sys/class/block/bcache0/bcache0p1/slaves'
Jan 9 06:35:17 beartic cloud-init[2993]: builtin command failed
Jan 9 06:35:17 beartic cloud-init[2993]: finish: cmd-install/stage-partitioning/builtin: FAIL: running 'curtin block-meta custom'
Jan 9 06:35:17 beartic cloud-init[2993]: builtin took 4.751 seconds
Jan 9 06:35:18 beartic cloud-init[2993]: stage_partitioning took 4.754 seconds
Jan 9 06:35:18 beartic cloud-init[2993]: finish: cmd-install/stage-partitioning: FAIL: configuring storage
Jan 9 06:35:18 beartic cloud-init[2993]: curtin: Installation failed with exception: Unexpected error while running command.
Jan 9 06:35:18 beartic cloud-init[2993]: Command: ['curtin', 'block-meta', 'custom']
Jan 9 06:35:18 beartic cloud-init[2993]: Exit code: 3
Jan 9 06:35:18 beartic cloud-init[2993]: Reason: -

Related branches

Ryan Harper (raharper) wrote :

FileNotFoundError: [Errno 2] No such file or directory: '/sys/class/block/bcache0/bcache0p1/slaves'
Jan 9 06:35:17 beartic cloud-init[2993]: [Errno 2] No such file or directory: '/sys/class/block/bcache0/bcache0p1/slaves'

Can you provide the curtin configuration and the install log (verbose)?

https://discourse.maas.io/t/getting-curtin-debug-logs/169

I didn't think bcache devices could be partitioned directly; so this looks really strange.
None the less, it appears doing a wipe; so curtin needs to handle this.

Changed in curtin:
status: New → Incomplete
Jason Hobbs (jason-hobbs) wrote :

Another repro, this is the rsyslog which I think includes the curtin logs, with verbose on:
http://paste.ubuntu.com/p/NvzCGHJZ9x/

This is the curtin-config
http://paste.ubuntu.com/p/KY89jHJG8S/

summary: - Failed deployment: 'cloudinit' running modules for final
+ Failed deployment: FileNotFoundError: [Errno 2] No such file or
+ directory: '/sys/class/block/bcache0/bcache0p1/slaves'
Changed in curtin:
status: Incomplete → New
Ryan Harper (raharper) wrote :

Hrm, something must be creating a partition on bcachce0 though; after the install. What software runs after an initial install?

Note, curtin will fix this; I'm just curious as partitioning a bcached device is an odd thing to do.

Changed in curtin:
importance: Undecided → High
status: New → Confirmed
Jason Hobbs (jason-hobbs) wrote :

I think that bcache device is getting passed to ceph as an OSD device. ceph partitions OSD devices.

Jason Hobbs (jason-hobbs) wrote :

Subscribed to field critical as this is causing a very high number of solutions QA test failures and we don't have a workaround for it.

Ryan Harper (raharper) on 2019-01-15
Changed in curtin:
status: Confirmed → In Progress
Ryan Harper (raharper) on 2019-01-15
Changed in curtin:
importance: High → Critical
Chad Smith (chad.smith) on 2019-01-15
Changed in curtin:
assignee: nobody → Ryan Harper (raharper)
Ryan Harper (raharper) wrote :

I've got the branch to address this up in a PPA:

ppa:raharper/bugfixes

curtin - 18.2-0ubuntu5~clear-holders-bcache-partitions-lp1811117-ppa5

Available for xenial and bionic.

Please test.

Ryan Harper (raharper) wrote :

Interesting, partitioning of bcache devices only works on newer kernels, Xenial GA (4.4.x) does not support this.

That will make this a bit more tricky.

Jason Hobbs (jason-hobbs) wrote :

I don't think that matters to us, we use hwe 4.15.x to get the bcache consistent naming fixes there.

Jason Hobbs (jason-hobbs) wrote :

We're testing with your ppa now and hitting an error when trying to deploy VMs:

Jan 17 12:08:49 juju-1 cloud-init[1318]: An error occured handling 'vda': RuntimeError - Cannot create disk tag udev rule for /dev/vda [id=vda], missing 'serial' or 'wwn' value

Full log: http://paste.ubuntu.com/p/QdrjHCw8v6/

This seems like an unrelated issue but it's blocking us getting to the point where we reproduce this.

On Thu, Jan 17, 2019 at 6:45 AM Jason Hobbs <email address hidden>
wrote:

> We're testing with your ppa now and hitting an error when trying to
> deploy VMs:
>
> Jan 17 12:08:49 juju-1 cloud-init[1318]: An error occured
> handling 'vda': RuntimeError - Cannot create disk tag udev rule for
> /dev/vda [id=vda], missing 'serial' or 'wwn' value
>
>
Doesn't maas create serials for kvm disks?

I'll apply a workaround and update the package in the ppa.

>
> Full log: http://paste.ubuntu.com/p/QdrjHCw8v6/
>
> This seems like an unrelated issue but it's blocking us getting to the
> point where we reproduce this.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1811117
>
> Title:
> Failed deployment: FileNotFoundError: [Errno 2] No such file or
> directory: '/sys/class/block/bcache0/bcache0p1/slaves'
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/curtin/+bug/1811117/+subscriptions
>

Ryan Harper (raharper) wrote :

I've updated the ppa with a workaround for that issue.

curtin - 18.2-0ubuntu9~clear-holders-bcache-partitions-lp1811117-ppa9

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers