[UC22] can't refresh pc-kernel after snapd 2.61.1 release

[Alexander Shevtsov]

Hi,

Our IoT devices running UC22 image, after snapd 2.61.1 release software update logic stopped working due to the following error:

2024-01-05T10:10:51Z ERROR could not map volume pc from gadget.yaml to any physical disk: cannot find physical disk laid out to map with volume pc

I did some tests on the dangerous image installed on my VM and found that `pc-kernel` snap can't be refreshed with latest snapd version, here is the change log of snapd+pc-kernel transactional update:

Status Spawn Ready Summary
Done yesterday at 22:22 UTC today at 10:10 UTC Ensure prerequisites for "snapd" are available
Undone yesterday at 22:22 UTC today at 10:10 UTC Download snap "snapd" (20671) from channel "latest/stable"
Done yesterday at 22:22 UTC today at 10:10 UTC Fetch and check assertions for snap "snapd" (20671)
Undone yesterday at 22:22 UTC today at 10:10 UTC Mount snap "snapd" (20671)
Undone yesterday at 22:22 UTC today at 10:10 UTC Run pre-refresh hook of "snapd" snap if present
Undone yesterday at 22:22 UTC today at 10:10 UTC Stop snap "snapd" services
Undone yesterday at 22:22 UTC today at 10:10 UTC Remove aliases for snap "snapd"
Undone yesterday at 22:22 UTC today at 10:10 UTC Make current revision for snap "snapd" unavailable
Undone yesterday at 22:22 UTC today at 10:10 UTC Copy snap "snapd" data
Undone yesterday at 22:22 UTC today at 10:10 UTC Setup snap "snapd" (20671) security profiles
Undone yesterday at 22:22 UTC today at 10:10 UTC Make snap "snapd" (20671) available to the system
Undone yesterday at 22:22 UTC today at 10:10 UTC Automatically connect eligible plugs and slots of snap "snapd"
Undone yesterday at 22:22 UTC today at 10:10 UTC Set automatic aliases for snap "snapd"
Undone yesterday at 22:22 UTC today at 10:10 UTC Setup snap "snapd" aliases
Done yesterday at 22:22 UTC today at 10:10 UTC Update managed boot config assets from "snapd" (20671)
Undone yesterday at 22:22 UTC today at 10:10 UTC Run post-refresh hook of "snapd" snap if present
Undone yesterday at 22:22 UTC today at 10:10 UTC Start snap "snapd" (20671) services
Done yesterday at 22:22 UTC today at 10:10 UTC Clean up "snapd" (20671) install
Undone yesterday at 22:22 UTC today at 10:10 UTC Run health check of "snapd" snap
Done yesterday at 22:22 UTC today at 10:10 UTC Ensure prerequisites for "pc-kernel" are available
Undone yesterday at 22:22 UTC today at 10:10 UTC Download snap "pc-kernel" (1540) from channel "22/stable"
Done yesterday at 22:22 UTC today at 10:10 UTC Fetch and check assertions for snap "pc-kernel" (1540)
Undone yesterday at 22:22 UTC today at 10:10 UTC Mount snap "pc-kernel" (1540)
Undone yesterday at 22:22 UTC today at 10:10 UTC Run pre-refresh hook of "pc-kernel" snap if present
Undone yesterday at 22:22 UTC today at 10:10 UTC Stop snap "pc-kernel" services
Undone yesterday at 22:22 UTC today at 10:10 UTC Remove aliases for snap "pc-kernel"
Undone yesterday at 22:22 UTC today at 10:10 UTC Make current revision for snap "pc-kernel" unavailable
Error yesterday at 22:22 UTC today at 10:10 UTC Update assets from kernel "pc-kernel" (1540)
Hold yesterday at 22:22 UTC today at 10:10 UTC Copy snap "pc-kernel" data
Hold yesterday at 22:22 UTC today at 10:10 UTC Setup snap "pc-kernel" (1540) security profiles
Hold yesterday at 22:22 UTC today at 10:10 UTC Make snap "pc-kernel" (1540) available to the system
Hold yesterday at 22:22 UTC today at 10:10 UTC Automatically connect eligible plugs and slots of snap "pc-kernel"
Hold yesterday at 22:22 UTC today at 10:10 UTC Set automatic aliases for snap "pc-kernel"
Hold yesterday at 22:22 UTC today at 10:10 UTC Setup snap "pc-kernel" aliases
Hold yesterday at 22:22 UTC today at 10:10 UTC Run post-refresh hook of "pc-kernel" snap if present
Hold yesterday at 22:22 UTC today at 10:10 UTC Start snap "pc-kernel" (1540) services
Hold yesterday at 22:22 UTC today at 10:10 UTC Clean up "pc-kernel" (1540) install
Hold yesterday at 22:22 UTC today at 10:10 UTC Run configure hook of "pc-kernel" snap if present
Hold yesterday at 22:22 UTC today at 10:10 UTC Run health check of "pc-kernel" snap
Done yesterday at 22:22 UTC today at 10:10 UTC Handling re-refresh of "pc-kernel", "snapd" as needed

......................................................................
Make current revision for snap "snapd" unavailable

2024-01-05T10:10:54Z INFO Requested daemon restart (snapd snap).

......................................................................
Make snap "snapd" (20671) available to the system

2024-01-04T22:22:57Z INFO Requested daemon restart (snapd snap).

......................................................................
Automatically connect eligible plugs and slots of snap "snapd"

2024-01-04T22:22:57Z INFO Waiting for automatic snapd restart...
2024-01-04T22:22:57Z INFO Waiting for automatic snapd restart...

......................................................................
Update assets from kernel "pc-kernel" (1540)

2024-01-05T10:10:51Z ERROR could not map volume pc from gadget.yaml to any physical disk: cannot find physical disk laid out to map with volume pc

*same will happen if you refresh snapd first and then try to refresh pc-kernel
*initial versions: snapd 2.57.6(17883), pc-kernel 5.15.0-53.59.1(1136)
*with previous snapd 2.60.4 everything worked fine

Also attaching volumes section from gadget.yaml:

volumes:
  pc:
    # bootloader configuration is shipped and managed by snapd
    bootloader: grub
    structure:
      - name: mbr
        type: mbr
        size: 440
        update:
          edition: 1
        content:
          - image: pc-boot.img
      - name: BIOS Boot
        type: DA,21686148-6449-6E6F-744E-656564454649
        size: 1M
        offset: 1M
        offset-write: mbr+92
        update:
          edition: 2
        content:
          - image: pc-core.img
      - name: ubuntu-seed
        role: system-seed
        filesystem: vfat
        # UEFI will boot the ESP partition by default first
        type: EF,C12A7328-F81F-11D2-BA4B-00A0C93EC93B
        size: 2500M
        update:
          edition: 2
        content:
          - source: grubx64.efi
            target: EFI/boot/grubx64.efi
          - source: shim.efi.signed
            target: EFI/boot/bootx64.efi
      - name: ubuntu-boot
        role: system-boot
        filesystem: ext4
        type: 83,0FC63DAF-8483-4772-8E79-3D69D8477DE4
        size: 750M
        update:
          edition: 1
        content:
          - source: grubx64.efi
            target: EFI/boot/grubx64.efi
          - source: shim.efi.signed
            target: EFI/boot/bootx64.efi
      - name: ubuntu-save
        role: system-save
        filesystem: ext4
        type: 83,0FC63DAF-8483-4772-8E79-3D69D8477DE4
        size: 16M
      - name: ubuntu-data
        role: system-data
        filesystem: ext4
        type: 83,0FC63DAF-8483-4772-8E79-3D69D8477DE4
        size: 1G

[Alfonso Sanchez-Beato (alfonsosanchezbeato)]

The error indicates a possible mismatch between the disk and the gadget specification. To get some more information, please attach output of command

$ sudo fdisk -l

Then stop snapd and run in with debug log enabled:

$ sudo systemctl stop snapd.service snapd.socket
$ sudo SNAPD_DEBUG=1 /usr/lib/snapd/snapd

Try to install pc-kernel from anothre terminal and please attach the resulting traces to the bug.

[Alexander Shevtsov (alshevtsov)]

Attaching all the information you asked for:

sudo fdisk -l

Disk devloop0 72.86 MiB, 76398592 bytes, 149216 sectors
Units sectors of 1 512 = 512 bytes
Sector size (logicalphysical) 512 bytes 512 bytes
IO size (minimumoptimal) 512 bytes 512 bytes

Disk devloop1 1.11 MiB, 1167360 bytes, 2280 sectors
Units sectors of 1 512 = 512 bytes
Sector size (logicalphysical) 512 bytes 512 bytes
IO size (minimumoptimal) 512 bytes 512 bytes

Disk devloop2 286.6 MiB, 300519424 bytes, 586952 sectors
Units sectors of 1 512 = 512 bytes
Sector size (logicalphysical) 512 bytes 512 bytes
IO size (minimumoptimal) 512 bytes 512 bytes

Disk devloop3 63.23 MiB, 66301952 bytes, 129496 sectors
Units sectors of 1 512 = 512 bytes
Sector size (logicalphysical) 512 bytes 512 bytes
IO size (minimumoptimal) 512 bytes 512 bytes

Disk devloop4 55.32 MiB, 58011648 bytes, 113304 sectors
Units sectors of 1 512 = 512 bytes
Sector size (logicalphysical) 512 bytes 512 bytes
IO size (minimumoptimal) 512 bytes 512 bytes

Disk devloop5 4.62 MiB, 4841472 bytes, 9456 sectors
Units sectors of 1 512 = 512 bytes
Sector size (logicalphysical) 512 bytes 512 bytes
IO size (minimumoptimal) 512 bytes 512 bytes

Disk devloop6 858.55 MiB, 900259840 bytes, 1758320 sectors
Units sectors of 1 512 = 512 bytes
Sector size (logicalphysical) 512 bytes 512 bytes
IO size (minimumoptimal) 512 bytes 512 bytes

Disk devloop7 365.38 MiB, 383131648 bytes, 748304 sectors
Units sectors of 1 512 = 512 bytes
Sector size (logicalphysical) 512 bytes 512 bytes
IO size (minimumoptimal) 512 bytes 512 bytes

Disk devsda 14.65 GiB, 15728640000 bytes, 30720000 sectors
Disk model VBOX HARDDISK
Units sectors of 1 512 = 512 bytes
Sector size (logicalphysical) 512 bytes 512 bytes
IO size (minimumoptimal) 512 bytes 512 bytes
Disklabel type gpt
Disk identifier 31333E55-E3A7-4740-84FD-4FAF442F6A0E

Device Start End Sectors Size Type
devsda1 2048 4095 2048 1M BIOS boot
devsda2 4096 4198399 4194304 2G EFI System
devsda3 4198400 5734399 1536000 750M Linux filesystem
devsda4 5734400 5767167 32768 16M Linux filesystem
devsda5 5767168 30719966 24952799 11.9G Linux filesystem

Disk devloop8 1.11 MiB, 1163264 bytes, 2272 sectors
Units sectors of 1 512 = 512 bytes
Sector size (logicalphysical) 512 bytes 512 bytes
IO size (minimumoptimal) 512 bytes 512 bytes

Disk devloop9 28.36 MiB, 29741056 bytes, 58088 sectors
Units sectors of 1 512 = 512 bytes
Sector size (logicalphysical) 512 bytes 512 bytes
IO size (minimumoptimal) 512 bytes 512 bytes

Disk devloop10 28.03 MiB, 29388800 bytes, 57400 sectors
Units sectors of 1 512 = 512 bytes
Sector size (logicalphysical) 512 bytes 512 bytes
IO size (minimumoptimal) 512 bytes 512 bytes

Disk devloop11 49.62 MiB, 52031488 bytes, 101624 sectors
Units sectors of 1 512 = 512 bytes
Sector size (logicalphysical) 512 bytes 512 bytes
IO size (minimumoptimal) 512 bytes 512 bytes

Disk devloop12 7.84 MiB, 8216576 bytes, 16048 sectors
Units sectors of 1 512 = 512 bytes
Sector size (logicalphysical) 512 bytes 512 bytes
IO size (minimumoptimal) 512 bytes 512 bytes

------------------------------------------------...

[Alfonso Sanchez-Beato (alfonsosanchezbeato)]

Thanks a lot for the logs. What has changed in snapd 2.61.1 is not the check for matching gadget to disk sizes in itself but when the check happens: in the past in some cases this check was not being performed when it should. For instance, the newer snapd would not have let you update the gadget defining a 2.5 GB seed partition of a device where the partition is actually 2 GBs. But unfortunately this update already happened.

The way to fix this would be to distribute a new gadget compatible with both disk layouts. For that, there is a new "min-size" field that can be used when describing a partition. So you would have something like:

- name: ubuntu-seed
  min-size: 2000M
  size: 2500M
  ...

Once the new gadget is installed in the systems, it should be possible to update pc-kernel.

[Alexander Shevtsov (alshevtsov)]

Yes, you're right, no issues after releasing/installing new gadget with min-size: 2000M option. So, moving forward we can increase size parameter if necessary and keep min-size parameter equal 2G, this will allow us to support all possible disk layouts within a given range, correct?

[Alfonso Sanchez-Beato (alfonsosanchezbeato)]

That's correct, you just need to make sure that the ranges specified in the gadget include the real disk partitions sizes for all the devices that use that gadget.

[Alexander Shevtsov (alshevtsov)]

Got it, the issues is resolved, thanks again.