machinectl pull-raw fails: No space left on device

[Dimitri John Ledkov]

$ machinectl pull-raw https://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-disk1.img

fails to expand /var/lib/machines.raw or /var/lib/machines, and thus fails to complete:

SHA256 checksum of https://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-disk1.img is valid.
gpg: Signature made Wed 21 Oct 2015 16:06:28 BST using RSA key ID 7DB87C81
gpg: Good signature from "UEC Image Automatic Signing Key <email address hidden>" [unknown]
gpg: WARNING: Using untrusted key!
Signature verification succeeded.
Unpacking QCOW2 file.
Failed to convert qcow2 image: No space left on device
Exiting.

[Dimitri John Ledkov (xnox)]

wily, 225-1ubuntu9

[Martin Pitt (pitti)]

I tried that command (on current xenial, you didn't specify an Ubuntu release/package version), and it works fine:

$ machinectl pull-raw https://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-disk1.img
Download of https://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-disk1.img complete.
SHA256 checksum of https://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-disk1.img is valid.
gpg: Signatur vom Mi 28 Okt 2015 03:01:23 CET mittels RSA-Schlüssel ID 7DB87C81
gpg: Korrekte Signatur von "UEC Image Automatic Signing Key <email address hidden>" [unbekannt]
gpg: WARNUNG: Ein Schlüssel ohne gesichertes Vertrauen wird benutzt!
Signature verification succeeded.
Unpacking QCOW2 file.
Created new local image 'trusty-server-cloudimg-amd64-disk1'.
Operation completed successfully.
Exiting.

$ machinectl list-images
NAME TYPE RO USAGE CREATED MODIFIED
trusty-server-cloudimg-amd64-disk1 raw no 815.5M n/a Do 2015-10-29 14:23:43 CET

Actually trying to start the container fails due to

Okt 29 14:26:25 donald systemd[1]: Starting Container trusty-server-cloudimg-amd64-disk1...
Okt 29 14:26:25 donald systemd-nspawn[10918]: Failed to open /dev/loop-control: Operation not permitted
Okt 29 14:26:25 donald systemd[1]: systemd-nspawn@trusty\x2dserver\x2dcloudimg\x2damd64\x2ddisk1.service: Main process exited, code=exited, status=1/FAILURE

But that sounds like a separate problem.

> Failed to convert qcow2 image: No space left on device

That sounds like the actual cause for what you see? :-) Can you confirm that it works with enough disk space?

[Dimitri John Ledkov (xnox)]

Comment #1 specifies release, and package version (wily, release).

The disk space error I presume is of /var/lib/machines, which is managed by the systemd-machined entirely. It is a loop mounted btrfs file, which systemd should be automatically growing to correctly accommodate increases in the disk usage, which is clearly not working correctly here.

The current space reserved is:

$ sudo btrfs filesystem df /var/lib/machines
System, single: total=4.00MiB, used=4.00KiB
Data+Metadata, single: total=496.00MiB, used=32.00KiB
GlobalReserve, single: total=4.00MiB, used=0.00B

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Philip Freeman (phil-marble)]

marble@bravo:~$ sudo machinectl pull-tar https://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-root.tar.gz
Enqueued transfer job 1. Press C-c to continue download in background.
Pulling 'https://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-root.tar.gz', saving as 'trusty-server-cloudimg-amd64-root'.
Downloading 186.4M for https://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-root.tar.gz.
Downloading 253B for https://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-root.nspawn.
HTTP request to https://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-root.nspawn failed with code 404.
Settings file could not be retrieved, proceeding without.
Downloading 260B for https://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-root.tar.gz.sha256.
Downloading 836B for https://cloud-images.ubuntu.com/trusty/current/SHA256SUMS.gpg.
Download of https://cloud-images.ubuntu.com/trusty/current/SHA256SUMS.gpg complete.
Set up default quota hierarchy for /var/lib/machines/.#tarb32b45621ce56d91.
Downloading 3.0K for https://cloud-images.ubuntu.com/trusty/current/SHA256SUMS.
Download of https://cloud-images.ubuntu.com/trusty/current/SHA256SUMS complete.
Got 1% of https://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-root.tar.gz. 5min 33s left at 566.3K/s.
Got 2% of https://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-root.tar.gz. 3min 52s left at 804.6K/s.
Got 3% of https://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-root.tar.gz. 3min 21s left at 920.3K/s.
Got 4% of https://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-root.tar.gz. 3min 9s left at 969.1K/s.
Got 5% of https://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-root.tar.gz. 2min 57s left at 1021.9K/s.
Got 6% of https://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-root.tar.gz. 2min 49s left at 1.0M/s.
Got 7% of https://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-root.tar.gz. 2min 48s left at 1.0M/s.
<SNIP>
Got 51% of https://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-root.tar.gz. 1min 54s left at 814.3K/s.
tar: usr/share/ca-certificates/mozilla/Staat_der_Nederlanden_Root_CA_-_G2.crt: Cannot open: No space left on device
tar: usr/share/ca-certificates/mozilla/OpenTrust_Root_CA_G2.crt: Cannot open: No space left on device
tar: usr/share/ca-certificates/mozilla/CA_Disig_Root_R1.crt: Cannot open: No space left on device
tar: usr/share/ca-certificates/mozilla/AddTrust_Public_Services_Root.crt: Cannot open: No space left on device
<SNIP>

marble@bravo:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 16G 0 16G 0% /dev
tmpfs 3.2G 1.4M 3.2G 1% /run
/dev/sda2 439G 11G 406G 3% /
tmpfs 16G 0 16G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/loop0 89M 89M 0 100% /snap/core/7270
/dev/sda1 511M 6.1M 505M 2% /boot/efi
tmpfs 3.2G 0 3.2G ...

[Philip Freeman (phil-marble)]

This effects me on a brand new Ubuntu Server 18.04.3 Install

[Philip Freeman (phil-marble)]

Apparently this will never get fixed upstream:

From https://github.com/systemd/systemd/issues/5859#issuecomment-553377937

poettering: "So we dropped the automatic btrfs loopback file support in machined a while back. Let's close this hence."