[3.1+snap7] Attempts to write contents into a file which is not a directory with trailing slash

[Erich Eickmeyer]

I'm getting this error:

$ sudo ubuntu-image -v classic -w workdir/ -O output/ ubuntu-images/ubuntu-pi-arm64.yaml
[0] parse_image_definition
[1] calculate_states
[2] make_temporary_directories
[3] determine_output_directory
[4] build_gadget_tree
[5] prepare_gadget_tree
[6] load_gadget_yaml
[7] verify_artifact_names
[8] germinate
[9] create_chroot
[10] install_packages
[11] prepare_image
Fetching firefox (3422)
Fetching gnome-42-2204 (143)
Fetching gtk-common-themes (1535)
Fetching snap-store (1047)
Fetching snapd-desktop-integration (85)
Fetching core22 (867)
Fetching bare (5)
Fetching snapd (20298)
[12] preseed_image
[13] clean_rootfs
[14] customize_sources_list
[15] customize_fstab
[16] perform_manual_customization
[17] set_default_locale
[18] populate_rootfs_contents
[19] calculate_rootfs_size
[20] populate_bootfs_contents
Error: Error in mountedFilesystem.Write(): cannot write filesystem content of source:install/boot-assets/: cannot specify trailing / for a source which is not a directory

$ snap info ubuntu-image
name: ubuntu-image
summary: Create Ubuntu images
publisher: Canonical✓
store-url: https://snapcraft.io/ubuntu-image
contact: <email address hidden>
license: unset
description: |
  ubuntu-image is a tool used for creation of various Ubuntu preinstalled images, both classic and
  snap-based.
commands:
  - ubuntu-image
snap-id: 4RW78vIax8JW5S8HkYsa8lNbv68uPaYX
tracking: latest/edge
refresh-date: yesterday at 10:53 PST
channels:
  latest/stable: 3.1 2023-11-28 (658) 40MB classic
  latest/candidate: 3.1 2023-11-28 (658) 40MB classic
  latest/beta: 3.1 2023-11-28 (658) 40MB classic
  latest/edge: 3.1+snap3 2023-11-28 (668) 41MB classic
  2/stable: 2.2+snap10 2023-04-06 (322) 8MB classic
  2/candidate: 2.2+snap10 2023-04-06 (322) 8MB classic
  2/beta: 2.2+snap10 2023-04-06 (322) 8MB classic
  2/edge: 2.2+snap10 2023-04-06 (322) 8MB classic
installed: 3.1+snap3 (668) 41MB classic

[Steve Langasek (vorlon)]

Not clear if this is an issue in ubuntu-image or in https://github.com/snapcore/pi-gadget.git (though if it's the latter, it must be due to a behavior change in ubuntu-image). Dave, can you take a look?

[Paul Mars (upils)]

What version of ubuntu-pi-arm64.yaml did you use?

I tried to reproduce and encountered another error at the `perform_manual_customization` step, using the recently modified one from the ubuntu/noble branch, so I cannot reproduce yet.

$ sudo ubuntu-image -v classic -w workdir/ -O output/ ubuntu-images/ubuntu-pi-arm64.yaml
[...]
[16] perform_manual_customization
Executing command "/usr/sbin/chroot /home/paul/scratch/debug/chroot /usr/sbin/oem-config-prepare"
⚠️ /proc/ is not mounted. This is not a supported mode of operation. Please fix
your invocation environment to mount /proc/ and /sys/ properly. Proceeding anyway.
Your mileage may vary.
Created symlink /etc/systemd/system/oem-config.target.wants/oem-config.service → /lib/systemd/system/oem-config.service.
⚠️ /proc/ is not mounted. This is not a supported mode of operation. Please fix
your invocation environment to mount /proc/ and /sys/ properly. Proceeding anyway.
Your mileage may vary.
The unit files have no installation config (WantedBy=, RequiredBy=, Also=,
Alias= settings in the [Install] section, and DefaultInstance= for template
units). This means they are not meant to be enabled or disabled using systemctl.

Possible reasons for having this kind of units are:
• A unit may be statically enabled by being symlinked from another unit's
  .wants/ or .requires/ directory.
• A unit's purpose may be to act as a helper for some other unit which has
  a requirement dependency on it.
• A unit may be started when needed via activation (socket, path, timer,
  D-Bus, udev, scripted systemctl call, ...).
• In case of template units, the unit is meant to be enabled with some
  instance name specified.
⚠️ /proc/ is not mounted. This is not a supported mode of operation. Please fix
your invocation environment to mount /proc/ and /sys/ properly. Proceeding anyway.
Your mileage may vary.
Created symlink /etc/systemd/system/default.target → /lib/systemd/system/oem-config.target.
sudo: unknown user oem
sudo: error initializing audit plugin sudoers_audit
duration: 318.843115ms
Error: Error running script "/usr/sbin/chroot /home/paul/scratch/debug/chroot /usr/sbin/oem-config-prepare". Error is exit status 1. Full output below:
⚠️ /proc/ is not mounted. This is not a supported mode of operation. Please fix
your invocation environment to mount /proc/ and /sys/ properly. Proceeding anyway.
Your mileage may vary.
Created symlink /etc/systemd/system/oem-config.target.wants/oem-config.service → /lib/systemd/system/oem-config.service.
⚠️ /proc/ is not mounted. This is not a supported mode of operation. Please fix
your invocation environment to mount /proc/ and /sys/ properly. Proceeding anyway.
Your mileage may vary.
The unit files have no installation config (WantedBy=, RequiredBy=, Also=,
Alias= settings in the [Install] section, and DefaultInstance= for template
units). This means they are not meant to be enabled or disabled using systemctl.

Possible reasons for having this kind of units are:
• A unit may be statically enabled by being symlinked from another unit's
  .wants/ or .requires/ directory.
• A unit's purpose may be to act as a helper for some other unit which has
  a requirement dependency on i...

[Steve Langasek (vorlon)]

On Mon, Dec 04, 2023 at 11:23:30AM -0000, Paul Mars wrote:
> What version of ubuntu-pi-arm64.yaml did you use?

> I tried to reproduce and encountered another error at the
> `perform_manual_customization` step, using the recently modified one
> from the ubuntu/noble branch, so I cannot reproduce yet.

Please note that the only recent change there was to point the noble branch
to actually build for noble instead of mantic, so should have no effect on
the behavior of ubuntu-image wrt perform_manual_customization. (Even the
oem-config-prepare command from oem-config is still at the same version
between mantic and noble.)

> sudo: unknown user oem
> sudo: error initializing audit plugin sudoers_audit

Well, the oem-config-prepare script certainly depends on invoking `sudo -u
oem` when $DISPLAY is set. You could try unsetting $DISPLAY before invoking
ubuntu-image.

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer https://www.debian.org/
<email address hidden> <email address hidden>

[Erich Eickmeyer (eeickmeyer)]

Shows Revision 3 on Noble. I'll re-run and see if I can get a different result.

[Erich Eickmeyer (eeickmeyer)]

Results from my re-run, using the latest pull of ubuntu-images:

Basically same results as the bug report

$ sudo ubuntu-image -v classic -w workdir/ -O output/ ubuntu-images/ubuntu-pi-arm64.yaml
[0] parse_image_definition
[1] calculate_states
[2] make_temporary_directories
===snip===
[17] set_default_locale
[18] populate_rootfs_contents
[19] calculate_rootfs_size
[20] populate_bootfs_contents
Error: Error in mountedFilesystem.Write(): cannot write filesystem content of source: install/boot-assets/: cannot specify trailing / for a source which is not a directory

[Erich Eickmeyer (eeickmeyer)]

Same results with 3.1+snap5

[Erich Eickmeyer (eeickmeyer)]

Same results with 3.1+snap6

[Paul Mars (upils)]

I was able to reproduce. I plan on investigating it in January.

After a very quick look, I suspect this [0] change in snapd to maybe be responsible. But in 3.1 we also harmonized the use of several paths during the build so we may have messed up something.

Improvements in our testing suite should soon be merged and will help us detect this kind of issue early (in fact we currently reproduce this error in our tests).

@Erich no need for you to keep testing with future versions in edge until we tell here that this should be fixed.

[0] https://github.com/snapcore/snapd/pull/13106/files

[Steve Langasek (vorlon)]

We haven't been able to build raspi images in noble since 20231128. Marking this critical.

[Paul Mars (upils)]

A PR solving this is now in review https://github.com/canonical/ubuntu-image/pull/175

[Łukasz Zemczak (sil2100)]

After a round of reviews, we have decided not to fix this in ubuntu-image but instead streamline the gadget code itself. I merged the required change yesterday and confirmed today that the image builds are back working again.

[Erich Eickmeyer (eeickmeyer)]

Thanks Łukasz!

I would say that, if the gadget code was the problem, that this bug should be closed as "Invalid", but that's just me being a pedantic bug nerd.

I will get to testing locally here and report back, and also will be commenting on the existing Edubuntu RasPi MP in ubuntu-images (https://code.launchpad.net/~edubuntu-dev/ubuntu-images/+git/ubuntu-images/+merge/456547) for those builds as well.