machinectl pull-tar/pull-raw Operation not permitted

[Kane Foss]

[impact]

machinectl pull-tar does not work, ever

[test case]

see comment 2

[regression potential]

problems/failures during pull-tar operation

[scope]

needed only in j

fixed (indirectly) by upstream commit referenced in original description, which is included in v250, so fixed already in k

pull-tar does not fail on f; no fix needed there

[original description]

There is a bug in systemd 249, where one can't pull any images. This was fixed in version 250, and never got backported. (FIX: https://github.com/systemd/systemd/commit/c40d82abf7b23803aa7394a7a7e24c40c32af851)

Hopefully this can be addressed.

ProblemType: Bug
DistroRelease: Ubuntu 22.04
Package: systemd-container 249.11-0ubuntu3.1
ProcVersionSignature: Ubuntu 5.15.0-35.36-generic 5.15.35
Uname: Linux 5.15.0-35-generic x86_64
NonfreeKernelModules: zfs zunicode zcommon znvpair zavl icp
ApportVersion: 2.20.11-0ubuntu82.1
Architecture: amd64
CasperMD5CheckResult: pass
CurrentDesktop: ubuntu:GNOME
Date: Sat Jun 4 04:51:42 2022
InstallationDate: Installed on 2022-06-01 (2 days ago)
InstallationMedia: Ubuntu 22.04 LTS "Jammy Jellyfish" - Release amd64 (20220419)
SourcePackage: systemd
UpgradeStatus: No upgrade log present (probably fresh install)

[Kane Foss (kanefoss)]

To add to that, this is the error when trying to pull an image

$ machinectl pull-tar --verify=no https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64-root.tar.xz test
...
Got 98% of https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64-root.tar.xz. 1s left at 3.6M/s.
Got 99% of https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64-root.tar.xz. 200ms left at 3.6M/s.
Download of https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64-root.tar.xz complete.
Failed to rename to final image name to /var/lib/machines/.tar-https:\x2f\x2fcloud-images\x2eubuntu\x2ecom\x2fjammy\x2fcurrent\x2fjammy-server-cloudimg-amd64-root\x2etar\x2exz.\x2216765cac-5e045b7e29b80\x22: Operation not permitted
Exiting.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[murmel (murmel0)]

this seems to be still bugged, are any more logs required to get this fixed?

[Nick Rosbrook (enr0n)]

Murmel - What release of Ubuntu are you running? This has only been fixed on Kinetic, not Jammy.

[murmel (murmel0)]

I am on 22.04, same as OP. as Kinetic is on 251 (even back when that bug was reported) anyway, there shouldn't have been a need to fix it in Kinetic.

[Nick Rosbrook (enr0n)]

Sorry, I wrote my comment in a confusing way. No additional patch was needed in Kinetic for this issue. However, the fix has not been released for Jammy yet, but a patch has been provided that will be included in the next systemd SRU.

[murmel (murmel0)]

Oh, thanks for clarifying :)

[Łukasz Zemczak (sil2100)]

Hello Kane, or anyone else affected,

Accepted systemd into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/249.11-0ubuntu3.10 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted systemd (249.11-0ubuntu3.10) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

apt/2.4.10 (armhf)
casync/2+20201210-1build1 (ppc64el)
comitup/1.15-1 (armhf)
dbus/1.12.20-2ubuntu4.1 (armhf)
initramfs-tools/0.140ubuntu13.4 (s390x)
linux-azure-5.19/5.19.0-1027.30~22.04.2 (arm64)
linux-gcp-6.2/6.2.0-1011.11~22.04.3 (arm64)
linux-lowlatency/5.15.0-83.92 (arm64)
linux-lowlatency-hwe-5.19/5.19.0-1030.30 (arm64)
linux-nvidia-tegra/5.15.0-1016.16 (arm64)
linux-oracle-5.19/5.19.0-1027.30 (arm64)
mkosi/unknown (s390x)
munin/2.0.57-1ubuntu2 (armhf)
netplan.io/0.105-0ubuntu2~22.04.3 (arm64)
prometheus-postfix-exporter/unknown (s390x)
samba/2:4.15.13+dfsg-0ubuntu1.4 (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Nick Rosbrook (enr0n)]

I verified the fix using systemd-container 249.11-0ubuntu3.10 from jammy-proposed:

nr@clean-jammy-amd64:~$ apt-cache policy systemd-container
systemd-container:
  Installed: 249.11-0ubuntu3.10
  Candidate: 249.11-0ubuntu3.10
  Version table:
 *** 249.11-0ubuntu3.10 500
        500 http://archive.ubuntu.com/ubuntu jammy-proposed/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu jammy-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     249.11-0ubuntu3.9 500
        500 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
     249.11-0ubuntu3.7 500
        500 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages
     249.11-0ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages
nr@clean-jammy-amd64:~$ machinectl pull-tar --verify=no https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64-root.tar.xz test
Enqueued transfer job 1. Press C-c to continue download in background.
Pulling 'https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64-root.tar.xz', saving as 'test'.
Downloading 390.1M for https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64-root.tar.xz.
HTTP request to https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64-root.nspawn failed with code 404.
Settings file could not be retrieved, proceeding without.
Got 1% of https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64-root.tar.xz. 2min 45s left at 2.3M/s.
Got 2% of https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64-root.tar.xz. 4min 4s left at 1.5M/s.
Got 3% of https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64-root.tar.xz. 3min 34s left at 1.7M/s.
Got 4% of https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64-root.tar.xz. 3min 5s left at 2.0M/s.
Got 5% of https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64-root.tar.xz. 2min 36s left at 2.3M/s.
Got 6% of https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64-root.tar.xz. 2min 10s left at 2.7M/s.
Got 9% of https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64-root.tar.xz. 1min 47s left at 3.3M/s.
Got 11% of https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64-root.tar.xz. 1min 34s left at 3.6M/s.
Got 13% of https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64-root.tar.xz. 1min 21s left at 4.1M/s.
Got 16% of https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64-root.tar.xz. 1min 10s left at 4.6M/s.
Got 18% of https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64-root.tar.xz. 1min 3s left at 5.0M/s.
Got 21% of https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64-root.tar.xz. 56s left at 5.4M/s.
Got 24% of https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64-root.tar.xz. 50s left at 5.7M/s.
Got 27% of https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64-root.tar.xz. 47s left at 6.0M/s.
Got 29% of https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64-root.tar.xz. 43s l...

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 249.11-0ubuntu3.10

---------------
systemd (249.11-0ubuntu3.10) jammy; urgency=medium

  [ Nick Rosbrook ]
  * debian/tests/tests-in-lxd: use --reuse flag in lxc publish (LP: #2023229)
    File: debian/tests/tests-in-lxd
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=85b2ceddff1a6cc1ddbca8a1b7e5381d146e6313
  * pstore: only try to load efi_pstore module (LP: #2023462)
    File: debian/patches/lp1978079-efi-pstore-not-cleared-on-boot.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=7600bbfb1e8a399e5aeb1010a20deda3e5a06c89
  * shutdown: get only active md arrays. (LP: #2025563)
    File: debian/patches/lp2025563-shutdown-get-only-active-md-arrays.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=416a9245c8f0efbedcc4395cada23cb09c685ec3
  * udev-rules: fix nvme symlink creation on namespace changes (LP: #2028180)
    File: debian/patches/lp2028180-udev-rules-fix-nvme-symlink-creation-on-namespace-changes.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=26e85b944da9098e66fc0c39f64ee40254c0c278
  * core: reorder systemd arguments on reexec (LP: #2013543)
    File: debian/patches/lp2013543-core-reorder-systemd-arguments-on-reexec.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=19ba0f20d311642596dc65fa5d6eb96a2d4be280
  * network/dhcp4: do not ignore the gateway even if the destination is in same network (LP: #2009743)
    Files:
    - debian/patches/lp2009743/network-dhcp4-do-not-ignore-the-gateway-even-if-the-desti.patch
    - debian/patches/lp2009743/test-network-add-one-more-testcase-for-DHCPv4-classless-r.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=73e774a2fb99c82df6d0edd770bb84ab735ec2f0
  * Drop debian/UBUNTU-wait-online-exit-if-no-links-are-managed.patch (LP: #1982218)
    File: debian/patches/debian/UBUNTU-wait-online-exit-if-no-links-are-managed.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=cf82f08feea456e1c65895b34bffa8c33d421588
  * debian/systemd.postint: do not daemon re-exec if we could hit LP: #2013543
    File: debian/systemd.postinst
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=be484dab06d590b1792a8f016f4292373d0174b7
  * binfmt: fix systemd-binfmt in LXD containers (LP: #1999275)
    Files:
    - debian/patches/lp1999275/binfmt-check-if-binfmt-is-mounted-before-applying-rules.patch
    - debian/patches/lp1999275/binfmt-util-also-check-if-binfmt-is-mounted-in-read-write.patch
    - debian/patches/lp1999275/binfmt-util-split-out-binfmt_mounted.patch
    - debian/patches/lp1999275/unit-check-more-specific-path-to-be-written-by-systemd-bi.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=0b9eadf05ee6db0e009dacbc71521480095880b3

  [ Dan Streetman ]
  * Fix machinectl pull-tar and import-tar (LP: #1977630)
    Author: Dan Streetman
    File: debian/patches/lp1977630-fix_machinectl_pull_tar.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+sourc...

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for systemd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Steve Langasek (vorlon)]

The release of this SRU has been rolled back in jammy because of bug #2035406 which appears to be a regression introduced in the new version of systemd.

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 249.11-0ubuntu3.10

---------------
systemd (249.11-0ubuntu3.10) jammy; urgency=medium

  [ Nick Rosbrook ]
  * debian/tests/tests-in-lxd: use --reuse flag in lxc publish (LP: #2023229)
    File: debian/tests/tests-in-lxd
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=85b2ceddff1a6cc1ddbca8a1b7e5381d146e6313
  * pstore: only try to load efi_pstore module (LP: #2023462)
    File: debian/patches/lp1978079-efi-pstore-not-cleared-on-boot.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=7600bbfb1e8a399e5aeb1010a20deda3e5a06c89
  * shutdown: get only active md arrays. (LP: #2025563)
    File: debian/patches/lp2025563-shutdown-get-only-active-md-arrays.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=416a9245c8f0efbedcc4395cada23cb09c685ec3
  * udev-rules: fix nvme symlink creation on namespace changes (LP: #2028180)
    File: debian/patches/lp2028180-udev-rules-fix-nvme-symlink-creation-on-namespace-changes.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=26e85b944da9098e66fc0c39f64ee40254c0c278
  * core: reorder systemd arguments on reexec (LP: #2013543)
    File: debian/patches/lp2013543-core-reorder-systemd-arguments-on-reexec.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=19ba0f20d311642596dc65fa5d6eb96a2d4be280
  * network/dhcp4: do not ignore the gateway even if the destination is in same network (LP: #2009743)
    Files:
    - debian/patches/lp2009743/network-dhcp4-do-not-ignore-the-gateway-even-if-the-desti.patch
    - debian/patches/lp2009743/test-network-add-one-more-testcase-for-DHCPv4-classless-r.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=73e774a2fb99c82df6d0edd770bb84ab735ec2f0
  * Drop debian/UBUNTU-wait-online-exit-if-no-links-are-managed.patch (LP: #1982218)
    File: debian/patches/debian/UBUNTU-wait-online-exit-if-no-links-are-managed.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=cf82f08feea456e1c65895b34bffa8c33d421588
  * debian/systemd.postint: do not daemon re-exec if we could hit LP: #2013543
    File: debian/systemd.postinst
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=be484dab06d590b1792a8f016f4292373d0174b7
  * binfmt: fix systemd-binfmt in LXD containers (LP: #1999275)
    Files:
    - debian/patches/lp1999275/binfmt-check-if-binfmt-is-mounted-before-applying-rules.patch
    - debian/patches/lp1999275/binfmt-util-also-check-if-binfmt-is-mounted-in-read-write.patch
    - debian/patches/lp1999275/binfmt-util-split-out-binfmt_mounted.patch
    - debian/patches/lp1999275/unit-check-more-specific-path-to-be-written-by-systemd-bi.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=0b9eadf05ee6db0e009dacbc71521480095880b3

  [ Dan Streetman ]
  * Fix machinectl pull-tar and import-tar (LP: #1977630)
    Author: Dan Streetman
    File: debian/patches/lp1977630-fix_machinectl_pull_tar.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+sourc...