ubuntu-image crashes with 'No space left on device'

[seanlano]

I'm following the guide at https://docs.ubuntu.com/core/en/guides/build-device/board-enablement

I want to build Ubuntu Core for a custom device, but first I figured I'd get it working with a known device - in this case, the RPi3.

By following the guide, I've downloaded the example pi3 model assertion, with this command:

    curl -H "Accept: application/x.ubuntu.assertion" "https://assertions.ubuntu.com/v1/assertions/model/16/canonical/pi3" > pi3.model

Then, using this command I can start to build the image:

    sudo ubuntu-image snap -c stable --image-size 4G pi3.model

(this is a little different to the command in the website, but that command also fails with the same error)

The output is:

    Fetching core
    Fetching pi2-kernel
    Fetching pi3
    COMMAND FAILED: sudo cp -dR --preserve=mode,timestamps,ownership /tmp/tmpkq364y9x/root/* /tmp/tmpy8utfhze/root-mount

    cp: error writing '/tmp/tmpy8utfhze/root-mount/system-data/var/lib/snapd/snaps/core_3884.snap': No space left on device
    cp: error writing '/tmp/tmpy8utfhze/root-mount/system-data/var/lib/snapd/snaps/pi2-kernel_22.snap': No space left on device
    cp: cannot create directory '/tmp/tmpy8utfhze/root-mount/system-data/var/lib/snapd/seed': No space left on device
    cp: cannot create directory '/tmp/tmpy8utfhze/root-mount/system-data/boot': No space left on device

    Crash in state machine
    Traceback (most recent call last):
      File "/snap/ubuntu-image/95/lib/python3/site-packages/ubuntu_image/__main__.py", line 341, in main
        list(state_machine)
      File "/snap/ubuntu-image/95/lib/python3/site-packages/ubuntu_image/state.py", line 82, in __next__
        step()
      File "/snap/ubuntu-image/95/lib/python3/site-packages/ubuntu_image/common_builder.py", line 337, in populate_filesystems
        self._populate_one_volume(name, volume)
      File "/snap/ubuntu-image/95/lib/python3/site-packages/ubuntu_image/common_builder.py", line 290, in _populate_one_volume
        preserve_ownership=True)
      File "/snap/ubuntu-image/95/lib/python3/site-packages/ubuntu_image/helpers.py", line 211, in mkfs_ext4
        preserve_flags, contents_dir, mountpoint), shell=True)
      File "/snap/ubuntu-image/95/lib/python3/site-packages/ubuntu_image/helpers.py", line 112, in run
        proc.check_returncode()
      File "/snap/ubuntu-image/95/usr/lib/python3.5/subprocess.py", line 349, in check_returncode
        self.stderr)
    subprocess.CalledProcessError: Command 'sudo cp -dR --preserve=mode,timestamps,ownership /tmp/tmpkq364y9x/root/* /tmp/tmpy8utfhze/root-mount' returned non-zero exit status 1

If I run "df -h" during this process, very briefly I can find this:

    Filesystem Size Used Avail Use% Mounted on
    udev 7.8G 0 7.8G 0% /dev
    tmpfs 1.6G 9.7M 1.6G 1% /run
    /dev/sda1 432G 259G 157G 63% /
    tmpfs 7.8G 130M 7.7G 2% /dev/shm
    tmpfs 5.0M 4.0K 5.0M 1% /run/lock
    tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup
    ...
    /dev/loop8 3.9M 52K 3.3M 2% /tmp/tmpvn70tzdq/root-mount <<<<<<<<

So it does seem like there really is not enough space in /dev/loop8 - but how am I supposed to fix this? I definitely do have enough actual free disk space.

I tried changing the gadget snap config under the "volumes" section to have size "256 M" instead of 128, but that didn't change anything.

Is there some step I'm missing from the guide? Or is this a bug?

[seanlano (seanlano)]

I should mention too, I installed it as a classic snap:

name: ubuntu-image
summary: Create Ubuntu images
publisher: canonical
contact: <email address hidden>
description: |
  Use this tool to create Ubuntu images.
snap-id: 4RW78vIax8JW5S8HkYsa8lNbv68uPaYX
commands:
  - ubuntu-image
tracking: beta
installed: 1.3+snap1 (95) 35MB classic
refreshed: 2017-12-01 06:49:58 +1100 AEDT
channels:
  stable: –
  candidate: –
  beta: 1.3+snap1 (95) 35MB classic
  edge: 1.3+snap1 (98) 34MB classic

[seanlano (seanlano)]

I tried the deb package available in 16.04, and also tried it on a different 17.10 machine again from the deb package - still I get the same problem, "No space left on device"

[Steve Langasek (vorlon)]

On Wed, Feb 07, 2018 at 11:06:24PM -0000, seanlano wrote:
> I tried the deb package available in 16.04, and also tried it on a
> different 17.10 machine again from the deb package - still I get the
> same problem, "No space left on device"

It looks like ubuntu-image may be misinterpreting your '--image-size 4G'
option. Could you try without and see if this makes a difference in the
behavior?

[seanlano (seanlano)]

Tried with just "sudo ubuntu-image snap -c stable cm3.model", still the same result:

    cp: error writing '/tmp/tmppj_io0n3/root-mount/system-data/var/lib/snapd/snaps/core_3884.snap': No space left on device
    cp: error writing '/tmp/tmppj_io0n3/root-mount/system-data/var/lib/snapd/snaps/pi2-kernel_22.snap': No space left on device
    cp: cannot create directory '/tmp/tmppj_io0n3/root-mount/system-data/var/lib/snapd/seed': No space left on device
    cp: cannot create directory '/tmp/tmppj_io0n3/root-mount/system-data/boot': No space left on device

[seanlano (seanlano)]

I've tried too with a different assertion, e.g. the dragonboard from "https://assertions.ubuntu.com/v1/assertions/model/16/canonical/dragonboard"

Again, still get the same error with the command "sudo ubuntu-image snap dragonboard.model"

[seanlano (seanlano)]

I've just tried again, using a kernel and gadget snap that I've compiled myself, I still get the same error:

    COMMAND FAILED: sudo cp -dR --preserve=mode,timestamps,ownership /tmp/tmpnvj93y2_/root/* /tmp/tmpbaz9qg5_/root-mount

    cp: error writing '/tmp/tmpbaz9qg5_/root-mount/system-data/var/lib/snapd/snaps/core_4020.snap': No space left on device
    cp: error writing '/tmp/tmpbaz9qg5_/root-mount/system-data/var/lib/snapd/snaps/extel-kernel-sama5-test-rev1_x1.snap': No space left on device
    cp: cannot create directory '/tmp/tmpbaz9qg5_/root-mount/system-data/var/lib/snapd/seed': No space left on device
    cp: cannot create directory '/tmp/tmpbaz9qg5_/root-mount/system-data/boot': No space left on device

[seanlano (seanlano)]

After some discussion in https://forum.snapcraft.io/t/ubuntu-image-no-space-left-on-device/4092, I tried adding this line to my /etc/fstab:

    none /tmp tmpfs noexec,nosuid,size=25%,mode=0755 0 0

After a reboot, the ubuntu-image command now works properly. Before this, I did not have a separate /tmp mountpoint listed in df, now I do.

Why does ubuntu-image care how /tmp is mounted?

[Łukasz Zemczak (sil2100)]

I am unable to reproduce your issue on any of my machines - I have tried on 16.04 ubuntu-image 1.3+16.04ubuntu2 deb, ubuntu-image snap 1.3+snap1 from beta and on 17.10 I have tried ubuntu-image 1.3+17.10ubuntu1 deb. In all cases the pi3 image builds correctly without any issues. I also have no explicit mount point present for /tmp during the runs.

Could you please give us more information about the machines you have been running ubuntu-image on? After removing the line in /etc/fstab are you still able to reproduce the issue?

Thanks!

[seanlano (seanlano)]

Ah, OK - I just tried two brand new VMs with Ubuntu 17.10. On one I left everything as default, which gives an ext4 filesystem. On the other I changed it to use btrfs, but otherwise left it as defaults.

The ext4-based system was able to create the pi3 Ubuntu Core image just fine - but the btrfs system failed with the exact same output as above.

My system uses btrfs - but I guess your system must be using ext4, and so the problem doesn't occur. Hence, I suppose this is actually somehow a problem with btrfs, but how?