Ubuntu
linux package

Bug #1729145
Comment #20

Comment 20 for bug 1729145

Revision history for this message

Ryan Harper (raharper) wrote on 2017-12-01: Re: [Bug 1729145] Re: /dev/bcache/by-uuid links not created after reboot

#20

Dec 01 05:07:25.679368 ubuntu systemd-udevd[474]: LINK
'disk/by-uuid/0a270acb-56b8-4498-8bad-b3bb149fe869'
/lib/udev/rules.d/60-persistent-storage.rules:79
Dec 01 05:07:25.683008 ubuntu systemd-udevd[474]: WARK:
dev_old_name=/dev/bcache/by-uuid/92d882d8-38cd-4537-847b-6f9c40ba67b4
dev_old_path=/devices/virtual/block/bcache0
Dec 01 05:07:25.683042 ubuntu systemd-udevd[474]: WARK:
dev_name=/dev/disk/by-uuid/0a270acb-56b8-4498-8bad-b3bb149fe869
dev_path=/devices/virtual/block/bcache0
Dec 01 05:07:25.683066 ubuntu systemd-udevd[474]: WARK:
streq(name=/dev/bcache/by-uuid/92d882d8-38cd-4537-847b-6f9c40ba67b4,
name_current=/dev/disk/by-uuid/0a270acb-56b8-4498-8bad-b3bb149fe869)
Dec 01 05:07:25.683076 ubuntu systemd-udevd[474]: update old name,
'/dev/bcache/by-uuid/92d882d8-38cd-4537-847b-6f9c40ba67b4' no longer
belonging to '/devices/virtual/block/bcache0'
Dec 01 05:07:25.683085 ubuntu systemd-udevd[474]: WARK:
link_update(name=/dev/bcache/by-uuid/92d882d8-38cd-4537-847b-6f9c40ba67b4,
false
Dec 01 05:07:25.683094 ubuntu systemd-udevd[474]: no reference left, remove
'/dev/bcache/by-uuid/92d882d8-38cd-4537-847b-6f9c40ba67b4'
Dec 01 05:07:25.683104 ubuntu systemd-udevd[474]: WARK:
dev_old_name=/dev/disk/by-uuid/0a270acb-56b8-4498-8bad-b3bb149fe869
dev_old_path=/devices/virtual/block/bcache0
Dec 01 05:07:25.683113 ubuntu systemd-udevd[474]: WARK:
dev_name=/dev/disk/by-uuid/0a270acb-56b8-4498-8bad-b3bb149fe869
dev_path=/devices/virtual/block/bcache0
Dec 01 05:07:25.683122 ubuntu systemd-udevd[474]: WARK:
streq(name=/dev/disk/by-uuid/0a270acb-56b8-4498-8bad-b3bb149fe869,
name_current=/dev/disk/by-uuid/0a270acb-56b8-4498-8bad-b3bb149fe869)
Dec 01 05:07:25.683131 ubuntu systemd-udevd[474]: WARK: found match!
continue
Dec 01 05:07:25.683146 ubuntu systemd-udevd[474]: handling device node
'/dev/bcache0', devnum=b250:0, mode=0660, uid=0, gid=6
Dec 01 05:07:25.683155 ubuntu systemd-udevd[474]: set permissions
/dev/bcache0, 060660, uid=0, gid=6
Dec 01 05:07:25.683165 ubuntu systemd-udevd[474]: preserve already existing
symlink '/dev/block/250:0' to '../bcache0'
Dec 01 05:07:25.683174 ubuntu systemd-udevd[474]: found 'b250:0' claiming
'/run/udev/links/\x2fdisk\x2fby-uuid\x2f0a270acb-56b8-4498-8bad-b3bb149fe869'
Dec 01 05:07:25.683183 ubuntu systemd-udevd[474]: creating link
'/dev/disk/by-uuid/0a270acb-56b8-4498-8bad-b3bb149fe869' to '/dev/bcache0'
Dec 01 05:07:25.683193 ubuntu systemd-udevd[474]: preserve already existing
symlink '/dev/disk/by-uuid/0a270acb-56b8-4498-8bad-b3bb149fe869' to
'../../bcache0'
Dec 01 05:07:25.683201 ubuntu systemd-udevd[474]: created db file
'/run/udev/data/b250:0' for '/devices/virtual/block/bcache0'
Dec 01 05:07:25.683211 ubuntu systemd[1]:
dev-disk-by\x2duuid-0a270acb\x2d56b8\x2d4498\x2d8bad\x2db3bb149fe869.device:
Changed dead -> plugged
Dec 01 05:07:25.683226 ubuntu systemd[1]: dev-bcache0.device: Changed dead
-> plugged
Dec 01 05:07:25.683234 ubuntu systemd[1]:
sys-devices-virtual-block-bcache0.device: Changed dead -> plugged

The new dev (from /deb/bcache0) only has a by-uuid devlink as the UEVENT
that generates the CACHED_UUID devlink is not recorded/discovered
after the original binding of the cache device and backing store. This
forces udev to remove the link.

I hacked in a change which allows the old link to stay if the old_name, or
old_path match; this
keeps the symlinks around.

This is required (or the kernel fix to emit the CACHED_UUID on cold plug);
but not sufficient.
What else is needed is a change to the bcache rules file such that when a
bcacheN device is
replugged, it will query the backing device's superblock to find out the
dev.uuid, and import that
value as CACHED_UUID so that the dname rule can match the path
/dev/bcache/by-uuid/<CACHED_UUID> and dname
will point to that, which abstracts away whether it points to bcache0, 1 ,
or N.

/dev/disk/by-dname/foo -> ../../../bcache/by-uuid/<dev.uuid> ->
../../bcacheN

On Thu, Nov 30, 2017 at 7:16 PM, Ryan Harper <email address hidden>
wrote:

> It looks like there is some ordering issues:
>
> This is a grep through /run/udev/links ; these are checked by udev-dev
>
> # find . -name 'b250*'
> ./\x2fdisk\x2fby-uuid\x2f0a270acb-56b8-4498-8bad-b3bb149fe869/b250:1
> ./\x2fdisk\x2fby-uuid\x2f92b0868d-7e56-4956-8e55-2c90ebee4a72/b250:0
> ./\x2fbcache\x2fby-uuid\x2f92d882d8-38cd-4537-847b-6f9c40ba67b4/b250:1
> ./\x2fbcache\x2fby-uuid\x2f57e009b1-6bf4-42ea-abe0-334b10941a0b/b250:0
>
> So both /dev/bcache/by-uuid and /dev/disk/by-uuid both point to the bcache
> device (b250:0)
>
> udevd shows this:
>
> Nov 30 23:39:10.738290 ubuntu systemd-udevd[465]: LINK
> 'disk/by-uuid/0a270acb-56b8-4498-8bad-b3bb149fe869' /lib/udev/rules.d/60-
> persistent-storage.rules:79
> Nov 30 23:39:10.738304 ubuntu systemd-udevd[465]: update old name,
> '/dev/bcache/by-uuid/92d882d8-38cd-4537-847b-6f9c40ba67b4' no longer
> belonging to '/devices/virtual/block/bcache1'
> Nov 30 23:39:10.738321 ubuntu systemd-udevd[465]: no reference left,
> remove '/dev/bcache/by-uuid/92d882d8-38cd-4537-847b-6f9c40ba67b4'
> Nov 30 23:39:10.738361 ubuntu systemd-udevd[465]: found 'b250:1' claiming
> '/run/udev/links/\x2fdisk\x2fby-uuid\x2f0a270acb-56b8-
> 4498-8bad-b3bb149fe869'
> Nov 30 23:39:10.738370 ubuntu systemd-udevd[465]: creating link
> '/dev/disk/by-uuid/0a270acb-56b8-4498-8bad-b3bb149fe869' to '/dev/bcache1'
> Nov 30 23:39:10.738380 ubuntu systemd-udevd[465]: preserve already
> existing symlink '/dev/disk/by-uuid/0a270acb-56b8-4498-8bad-b3bb149fe869'
> to '../../bcache1'
> Nov 30 23:39:10.743215 ubuntu systemd-udevd[487]: LINK
> 'disk/by-uuid/92b0868d-7e56-4956-8e55-2c90ebee4a72' /lib/udev/rules.d/60-
> persistent-storage.rules:79
> Nov 30 23:39:10.743228 ubuntu systemd-udevd[487]: update old name,
> '/dev/bcache/by-uuid/57e009b1-6bf4-42ea-abe0-334b10941a0b' no longer
> belonging to '/devices/virtual/block/bcache0'
> Nov 30 23:39:10.743247 ubuntu systemd-udevd[487]: no reference left,
> remove '/dev/bcache/by-uuid/57e009b1-6bf4-42ea-abe0-334b10941a0b'
> Nov 30 23:39:10.743291 ubuntu systemd-udevd[487]: found 'b250:0' claiming
> '/run/udev/links/\x2fdisk\x2fby-uuid\x2f92b0868d-7e56-
> 4956-8e55-2c90ebee4a72'
> Nov 30 23:39:10.743302 ubuntu systemd-udevd[487]: creating link
> '/dev/disk/by-uuid/92b0868d-7e56-4956-8e55-2c90ebee4a72' to '/dev/bcache0'
> Nov 30 23:39:10.743312 ubuntu systemd-udevd[487]: preserve already
> existing symlink '/dev/disk/by-uuid/92b0868d-7e56-4956-8e55-2c90ebee4a72'
> to '../../bcache0'
>
> It would seem that the 'no longer belonging to ...bcache1' is the trigger
> for the removal
> however, 250:1 is indeed bcache1 device.
>
>
> # ls -al /dev/bcache1
> brw-rw---- 1 root disk 250, 1 Nov 30 23:39 /dev/bcache1
>
> Looking at the systemd/udev/udev-event.c it appears that for entries in
> the current udev database that don't have a new event associated with them
> get the call to remove the old link
>
> void udev_event_execute_rules()
> ...
> if (major(udev_device_get_devnum(dev)) > 0) {
> bool apply;
>
> /* remove/update possible left-over symlinks from
> old database entry */
> if (event->dev_db != NULL)
> udev_node_update_old_links(dev,
> event->dev_db);
>
>
> Will need to crawl through the db to see what the the event->dev_db entry
> is and where that get's set.
>
>
>
> On Thu, Nov 30, 2017 at 2:41 PM, Dmitrii Shcherbakov <
> <email address hidden>> wrote:
>
>> Looks like masking systemd-udevd removes this behavior. Need to figure
>> out where it actually clears everything out.
>>
>> ubuntu@maas-xenial4:~$ sudo systemctl mask systemd-udevd
>> 11:32 PM Created symlink from /etc/systemd/system/systemd-udevd.service
>> to /dev/null.
>>
>> # reboot
>>
>> ubuntu@maas-xenial4:~$ ls /dev/bcache/by-uuid/
>> 2963855b-3d2b-4387-abd1-3ae788919de4 727212eb-0f0b-4d20-9191-f0577a38454e
>> 835afacf-6213-48a9-864c-43733822668b aa04ab39-a42c-446f-aff5-addbd9
>> c80e06
>>
>> --
>> You received this bug notification because you are subscribed to the bug
>> report.
>> https://bugs.launchpad.net/bugs/1729145
>>
>> Title:
>> /dev/bcache/by-uuid links not created after reboot
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1729145
>> /+subscriptions
>>
>
>

I hacked in a change which allows the old link to stay if the old_name, or
old_path match; this
keeps the symlinks around.

/dev/disk/by-dname/foo -> ../../../bcache/by-uuid/<dev.uuid>  ->
../../bcacheN

On Thu, Nov 30, 2017 at 7:16 PM, Ryan Harper <ryan.harper@canonical.com>
wrote:

> It looks like there is some ordering issues:
>
> This is a grep through /run/udev/links ; these are checked by udev-dev
>
> # find . -name 'b250*'
> ./\x2fdisk\x2fby-uuid\x2f0a270acb-56b8-4498-8bad-b3bb149fe869/b250:1
> ./\x2fdisk\x2fby-uuid\x2f92b0868d-7e56-4956-8e55-2c90ebee4a72/b250:0
> ./\x2fbcache\x2fby-uuid\x2f92d882d8-38cd-4537-847b-6f9c40ba67b4/b250:1
> ./\x2fbcache\x2fby-uuid\x2f57e009b1-6bf4-42ea-abe0-334b10941a0b/b250:0
>
> So both /dev/bcache/by-uuid and /dev/disk/by-uuid both point to the bcache
> device (b250:0)
>
> udevd shows this:
>
> Nov 30 23:39:10.738290 ubuntu systemd-udevd[465]: LINK
> 'disk/by-uuid/0a270acb-56b8-4498-8bad-b3bb149fe869' /lib/udev/rules.d/60-
> persistent-storage.rules:79
> Nov 30 23:39:10.738304 ubuntu systemd-udevd[465]: update old name,
> '/dev/bcache/by-uuid/92d882d8-38cd-4537-847b-6f9c40ba67b4' no longer
> belonging to '/devices/virtual/block/bcache1'
> Nov 30 23:39:10.738321 ubuntu systemd-udevd[465]: no reference left,
> remove '/dev/bcache/by-uuid/92d882d8-38cd-4537-847b-6f9c40ba67b4'
> Nov 30 23:39:10.738361 ubuntu systemd-udevd[465]: found 'b250:1' claiming
> '/run/udev/links/\x2fdisk\x2fby-uuid\x2f0a270acb-56b8-
> 4498-8bad-b3bb149fe869'
> Nov 30 23:39:10.738370 ubuntu systemd-udevd[465]: creating link
> '/dev/disk/by-uuid/0a270acb-56b8-4498-8bad-b3bb149fe869' to '/dev/bcache1'
> Nov 30 23:39:10.738380 ubuntu systemd-udevd[465]: preserve already
> existing symlink '/dev/disk/by-uuid/0a270acb-56b8-4498-8bad-b3bb149fe869'
> to '../../bcache1'
> Nov 30 23:39:10.743215 ubuntu systemd-udevd[487]: LINK
> 'disk/by-uuid/92b0868d-7e56-4956-8e55-2c90ebee4a72' /lib/udev/rules.d/60-
> persistent-storage.rules:79
> Nov 30 23:39:10.743228 ubuntu systemd-udevd[487]: update old name,
> '/dev/bcache/by-uuid/57e009b1-6bf4-42ea-abe0-334b10941a0b' no longer
> belonging to '/devices/virtual/block/bcache0'
> Nov 30 23:39:10.743247 ubuntu systemd-udevd[487]: no reference left,
> remove '/dev/bcache/by-uuid/57e009b1-6bf4-42ea-abe0-334b10941a0b'
> Nov 30 23:39:10.743291 ubuntu systemd-udevd[487]: found 'b250:0' claiming
> '/run/udev/links/\x2fdisk\x2fby-uuid\x2f92b0868d-7e56-
> 4956-8e55-2c90ebee4a72'
> Nov 30 23:39:10.743302 ubuntu systemd-udevd[487]: creating link
> '/dev/disk/by-uuid/92b0868d-7e56-4956-8e55-2c90ebee4a72' to '/dev/bcache0'
> Nov 30 23:39:10.743312 ubuntu systemd-udevd[487]: preserve already
> existing symlink '/dev/disk/by-uuid/92b0868d-7e56-4956-8e55-2c90ebee4a72'
> to '../../bcache0'
>
> It would seem that the 'no longer belonging to ...bcache1' is the trigger
> for the removal
> however, 250:1 is indeed bcache1 device.
>
>
> # ls -al /dev/bcache1
> brw-rw---- 1 root disk 250, 1 Nov 30 23:39 /dev/bcache1
>
> Looking at the systemd/udev/udev-event.c it appears that for entries in
> the current udev database that don't have a new event associated with them
> get the call to remove the old link
>
> void udev_event_execute_rules()
>   ...
>   if (major(udev_device_get_devnum(dev)) > 0) {
>                         bool apply;
>
>                         /* remove/update possible left-over symlinks from
> old database entry */
>                         if (event->dev_db != NULL)
>                                 udev_node_update_old_links(dev,
> event->dev_db);
>
>
> Will need to crawl through the db to see what the the event->dev_db entry
> is and where that get's set.
>
>
>
> On Thu, Nov 30, 2017 at 2:41 PM, Dmitrii Shcherbakov <
> 1729145@bugs.launchpad.net> wrote:
>
>> Looks like masking systemd-udevd removes this behavior. Need to figure
>> out where it actually clears everything out.
>>
>> ubuntu@maas-xenial4:~$ sudo systemctl mask systemd-udevd
>> 11:32 PM Created symlink from /etc/systemd/system/systemd-udevd.service
>> to /dev/null.
>>
>> # reboot
>>
>> ubuntu@maas-xenial4:~$ ls /dev/bcache/by-uuid/
>> 2963855b-3d2b-4387-abd1-3ae788919de4  727212eb-0f0b-4d20-9191-f0577a38454e
>> 835afacf-6213-48a9-864c-43733822668b  aa04ab39-a42c-446f-aff5-addbd9
>> c80e06
>>
>> --
>> You received this bug notification because you are subscribed to the bug
>> report.
>> https://bugs.launchpad.net/bugs/1729145
>>
>> Title:
>>   /dev/bcache/by-uuid links not created after reboot
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1729145
>> /+subscriptions
>>
>
>

Ubuntulinux package

Comment 20 for bug 1729145

Ubuntu
linux package