snapd hangs with 100% CPU usage when image has a custom gadget snap

[Renat]

When we add a gadget snap to an image by applying an `--extra-snaps` argument to the ubuntu-image - everything works fine, but when ubuntu-image downloads our gadget snap from the store - during the first boot snapd hangs when tries to mount our snap.

Here is the stack trace:

https://paste.ubuntu.com/24222873/

We're using our custom gadget snap from the custom store to build our image.

Also, everything works fine if we install an official pi3 gadget snap.

arch: armhf
gadget: pi3

[Kyle Nitzsche (knitzsche)]

Can you please check/confirm that you get the same problem when you first download your custom gadget snap from the store manually and then build it into your image using the ubuntu-image command with the --extra-snaps YOURDOWLOADEDSNAP.snap arg. Note that the ".snap" file extension is needed to ensure the local filesys snap is used.

[John Lenton (chipaca)]

Hello!

Sorry it's acting up on you. To help with this I *really* could use the contents of the log for snapd. Whatever's acting up is probably earlier in the logs than the stack trace itself; snapd is already using 10x the expected amount of memory by the time you get the OOM.

[Renat (renat2017)]

Hi, Kyle!

I can confirm that supplying a gadget snap downloaded from the store works fine.

Here is the command line:

ubuntu-image --channel=beta -o /build/screenly-v2-pi3-2017-03-22-19-58-47.img --extra-snaps=fEsLeh0lHx7eHjKhD2YIUw5hc9iigdQP_3.snap pi3.model

My ubuntu-image and snapcraft versions

snapper@a7562ad41eb5:~$ ubuntu-image --version
ubuntu-image 0.15+16.04ubuntu1
snapper@a7562ad41eb5:~$ snapcraft --version
2.27.1

So, the issue appears only if ubuntu-image downloads the gadget snap from the store.

[Renat (renat2017)]

Hi, John!

Here are snapd logs: https://paste.ubuntu.com/24229421/

The issue seems to be appearing when the gadget snap is being mounted.

As you can see from the previous comment - it happens only with a gadget snap downloaded from the Ubuntu store.

Also, other snaps are mounted just fine, even if ubuntu-image downloads them from the store.

Thanks,
Renat

[John Lenton (chipaca)]

Renat, any chance we can look at the snap itself? Hoping to reproduce this.

[Renat (renat2017)]

I am not sure that I am allowed to send you the image/snap, sorry for this=(

But, perhaps, I can prepare more information.

Just give me some time, please.

Thanks,
Renat

[Renat (renat2017)]

Okay, I am allowed to send you an image.

I will send you the link to the image in the personal message.

Thanks,
Renat

[John Lenton (chipaca)]

Renat, have you sent me this? I can't find it in my email (but I get a lot).

[Renat (renat2017)]

Sorry, John, I just wanted to do final tests. Sent you a download link in a personal message.

[John Lenton (chipaca)]

(setting back to "new" while we work on reproducing it)

[John Lenton (chipaca)]

So, this OOMing is definitely a bug we need to address, and it's the reason you're not being able to debug this issue

However, having said that, we've identified a number of issues with the image, one of which is probably the cause for it to never finish first boot. We'll be reaching out in private (to not dirty up this bug, which should remain about the OOM).

[Renat (renat2017)]

Hi, guys!

Thanks for the help. Removing the snapd-control interface from the prepare-device hook's plugs list fixed the issue.

I added that interface to remove AppArmor messages during the prepare-device process.

Best regards,
Renat

[Woodrow Shen (woodrow-shen)]

Hi all,

I also encountered the issue with armhf board, but the image with local gadget still happened the same situation. Currently, I can't figure out any way to workaround this.

[Woodrow Shen (woodrow-shen)]

Hi all,

After testing some experimental images, I have the new findings to share with bug:

1.Use ubuntu-image to build the image without any extra snap.(only include core/gadget/kernel, and they're downloaded from the store). Then image can work normally without the symptom of above bug.

2.I installed network-manager snap under system, and got some security / kernel issues.

3.Use ubuntu-image to build the image with `extra-snaps`. If we preload the network-manager snap with `extra-snaps` where it can't be installed completely, it could cause CPU hang issue.

FYI