Core22 Snap builds don't output error logs

[Farshid Tavakolizadeh]

Launchpad build logs don't include error logs for a snap that has core22 base. I understand that this is more of a Snapcraft issue, but I am reporting it here because it directly affects the usability of Launchpad.

This happens because Snapcraft 7 behaves differently depending on the snap base. When core22 is used, the error logs are written to a file. However, when building a core18/core20 snap, the errors are printed to the standard output/error as expected.

Snapcraft 7 introduces a flag that can be used to increase verbosity and get build logs on the standard output/error, but that flag is backward-incompatible with respect to the snap base. See https://bugs.launchpad.net/snapcraft/+bug/1980515

This limitation makes it very hard to debug and fix build errors as there is no access to local filesystem on an ephemeral CI instance.

Here is an example output, which includes very little helpful information on the error:

Executing parts lifecycle: build device-usb-camera
'override-build' in part 'device-usb-camera' failed with code 2.
Review the scriptlet and make sure it's correct.
Full execution log: '/root/.cache/snapcraft/log/snapcraft-20220708-060352.465359.log'
Build failed
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/lpbuildd/target/build_snap.py", line 196, in run
    self.build()
  File "/usr/lib/python3/dist-packages/lpbuildd/target/build_snap.py", line 182, in build
    self.run_build_command(
  File "/usr/lib/python3/dist-packages/lpbuildd/target/operation.py", line 46, in run_build_command
    return self.backend.run(args, cwd=cwd, env=full_env, **kwargs)
  File "/usr/lib/python3/dist-packages/lpbuildd/target/lxd.py", line 533, in run
    subprocess.check_call(cmd, **kwargs)
  File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)

Full build log: https://launchpadlibrarian.net/611876831/buildlog_snap_ubuntu_focal_amd64_edgex-device-usb-camera_BUILDING.txt.gz

As discussed internally, Snapcraft 7 may change to:
- print build error by default (like the versions before it)
- accept the -v/--verbose flag regardless of the core version (https://bugs.launchpad.net/snapcraft/+bug/1980515)
- detect when there is no TTY terminal and print logs
- add support for an environment variable to enable verbose mode, so that it can be set for Snapcraft 7/core22 and lower alike.

One workaround is to check the base value in snapcraft.yaml and snapcraft versions and pass the -v/--verbose flag selectively! But Launchpad doesn't seem to have an interface to pass flags to snapcraft, when using the UI or Python launchpadlib.

[Colin Watson (cjwatson)]

My understanding is that this is being worked on by Snapcraft developers, so reassigning it over there since it's not something we can sensibly do anything about.