Snapcraft issue during initrd driver generation

[Domotz Ubuntu Core]

Hello,
we have an issue after the build of our kernel (used in production on private Ubuntu Core Store) during the generation of the initrd driver:

Snapcraft 2.35, 64-bit Ubuntu 16.04.3 LTS

Command launched:
"snapcraft -d --target-arch=armhf"

Using the following snapcraft.yaml:

name: domotz-platform-kernel
confinement: strict
grade: stable
version: 4.9.5
summary: "A domotz-platform kernel built from source"
description: This is a kernel 4.9.5 patched to use Ubuntu Core Platform on H3
type: kernel
architectures: [ armhf ]

parts:
    kernel:
        plugin: kernel
        source: ***private-repository***
        source-tag: 1.0.0

        source-type: git
        source-depth: 15
        kdefconfig: [ domotz_fingbox_defconfig ]
        kconfigs:
            - CONFIG_LOCALVERSION="-domotz-platform"
            - CONFIG_DEBUG_INFO=n
            - CONFIG_SQUASHFS=m
            - CONFIG_DEVPTS_MULTIPLE_INSTANCES=y
            - CONFIG_BT_BCM=m
            - CONFIG_MAC80211_LEDS=y
            - CONFIG_RFKILL_LEDS=y
            - CONFIG_LEDS_TRIGGERS=y
            - CONFIG_RTL8189FS=n
            - CONFIG_REALTEK_PHY=n
            - CONFIG_SUN8I_THS=m
            - CONFIG_THERMAL=m
            - CONFIG_BINFMT_MISC=y
            - CONFIG_CPUFREQ_DT=m
            - CONFIG_ATH9K_HW=m
            - CONFIG_ATH9K_COMMON=m
            - CONFIG_ATH9K_HTC=m
            - CONFIG_ATH9K=m
            - CONFIG_MAC80211=m
            - CONFIG_FIRMWARE_IN_KERNEL=n
            - CONFIG_CC_STACKPROTECTOR_STRONG=y

        kernel-initrd-modules:
            - squashfs

        kernel-image-target: zImage
        kernel-device-trees:
            - sun8i-h3-domotz-fingbox-v2
        build-packages: [bc, kmod, cpio]
        after: [firmware]

    firmware:
        plugin: tar-content
        source: firmware.tar
        destination: firmware

--- Stacktrace --
Generating driver initrd for kernel release: 4.9.5-domotz-platform+
Parallel unsquashfs: Using 8 processors
2 inodes (2 blocks) to write

[========================================================================================|] 2/2 100%

created 0 files
created 4 directories
created 2 symlinks
created 0 devices
created 0 fifos
Traceback (most recent call last):
  File "/usr/bin/snapcraft", line 9, in <module>
    load_entry_point('snapcraft==2.35', 'console_scripts', 'snapcraft')()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 542, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2569, in load_entry_point
    return ep.load()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2229, in load
    return self.resolve()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2235, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/lib/python3/dist-packages/snapcraft/cli/__main__.py", line 19, in <module>
    run(prog_name='snapcraft')
  File "/usr/lib/python3/dist-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1037, in invoke
    return Command.invoke(self, ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/lib/python3/dist-packages/click/core.py", line 86, in augment_usage_errors
    yield
  File "/usr/lib/python3/dist-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/lib/python3/dist-packages/snapcraft/cli/__init__.py", line 124, in run
    ctx.forward(lifecyclecli.commands['snap'])
  File "/usr/lib/python3/dist-packages/click/core.py", line 552, in forward
    return self.invoke(cmd, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/lib/python3/dist-packages/click/core.py", line 86, in augment_usage_errors
    yield
  File "/usr/lib/python3/dist-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/snapcraft/cli/lifecycle.py", line 140, in snap
    project_options, directory=directory, output=output)
  File "/usr/lib/python3/dist-packages/snapcraft/internal/lifecycle/_packer.py", line 45, in snap
    execute('prime', project_options)
  File "/usr/lib/python3/dist-packages/snapcraft/internal/lifecycle/_runner.py", line 80, in execute
    _Executor(config, project_options).run(step, part_names)
  File "/usr/lib/python3/dist-packages/snapcraft/internal/lifecycle/_runner.py", line 175, in run
    self._run_step(step, part, part_names)
  File "/usr/lib/python3/dist-packages/snapcraft/internal/lifecycle/_runner.py", line 212, in _run_step
    getattr(part, step)()
  File "/usr/lib/python3/dist-packages/snapcraft/internal/pluginhandler/__init__.py", line 330, in build
    self.plugin.build()
  File "/usr/lib/python3/dist-packages/snapcraft/plugins/kbuild.py", line 258, in build
    self.do_install()
  File "/usr/lib/python3/dist-packages/snapcraft/plugins/kernel.py", line 469, in do_install
    self._make_initrd()
  File "/usr/lib/python3/dist-packages/snapcraft/plugins/kernel.py", line 282, in _make_initrd
    initrd_unpacked_path = self._unpack_generic_initrd()
  File "/usr/lib/python3/dist-packages/snapcraft/plugins/kernel.py", line 255, in _unpack_generic_initrd
    tmp_initrd_path, os.strerror(mime_detector.errno())))
RuntimeError: Unable to determine mime type for '/tmp/tmpkaqag_76/squashfs-root/usr/lib/ubuntu-core-generic-initrd/initrd.img-core': No such file or directory

After some investigation seems that the "os.snap" downloaded in the following way:
Starting new HTTPS connection (1): api.snapcraft.io
"GET /api/v1/snaps/details/core?channel=stable&fields=status%2Canon_download_url%2Cdownload_url%2Cdownload_sha3_384%2Cdownload_sha512%2Csnap_id%2Crevision%2Crelease HTTP/1.1" 200 589
Downloading core
Starting new HTTPS connection (1): api.snapcraft.io
"HEAD /api/v1/snaps/download/99T7MUlRhtI3U0QFgl5mXXESAiSwt776_3606.snap HTTP/1.1" 302 0
Starting new HTTPS connection (1): 068ed04f23.site.internapcdn.net
"GET /download-snap/99T7MUlRhtI3U0QFgl5mXXESAiSwt776_3606.snap?t=2017-12-19T18:00:00Z&h=8a4ace120f0db994bfd8632df9cf7e47f41edffe HTTP/1.1" 200 74252288
Downloading 'os.snap'[=========================================================================] 100%
Successfully downloaded core at /home/vagrant/workdir/jerry/kernel/parts/kernel/src/os.snap

has initrd inside "boot" folder but Snapcraft searches it into "usr/lib/ubuntu-core-generic-initrd/initrd.img-core" that is actually a simlink "usr/lib/ubuntu-core-generic-initrd/initrd.img-core -> ../../../boot/initrd.img-core"

Since the unsquashfs command selectively extract folders from the "os.snap" we have added "boot" folder to the command.

--- PATCH (unusquashfs.patch) ---

--- snapcraft/plugins/kernel.py 2017-12-20 10:19:57.970472232 +0000
+++ snapcraft/plugins/kernel_new.py 2017-12-20 10:22:26.724812233 +0000
@@ -237,7 +237,7 @@

         with tempfile.TemporaryDirectory() as temp_dir:
             subprocess.check_call([
- 'unsquashfs', self.os_snap, os.path.dirname(initrd_path)],
+ 'unsquashfs', self.os_snap, os.path.dirname(initrd_path), 'boot'],
                 cwd=temp_dir)

             tmp_initrd_path = os.path.join(

Do you agree that this is the best solution or I miss something?

[Ondrej Kubik (ondrak)]

I can confirm this being issue for many other kernel snaps

[Ville Ilvonen (vilvo)]

I had the same issue but I tested with a version from edge where it's already fixed.

Got this help from @ogra
"install the snapcraft snap from edge ... "sudo snap install snapcraft --classic --edge"
you can either use both side by side ... "snapcraft" calls the deb version "/snap/bin/snapcraft" the snap version ... or you can switch completely by simply using "sudo apt purge snapcraft" to only use the snap version (then there is only one snapcraft in the path so "snapcraft" always calls the snap version)"

With this help I can confirm the issue is fixed with `snapcraft, version 2.38+git29.aa116ef` from edge-channel.