Can't update repositories when building for different arch from host

[Dilyn]

Rnning `snapcraft --use-lxd --build-for=arm64` fails at the package repository update step. Currently (snapcraft rev. 9121) `dpkg` will be used to add a new architecture (when building either in destructive mode or in LXD). For whatever reason, when `apt update` is run the incorrect URL is used for the new architecture (it points to archive.ubuntu.com instead of ports.ubuntu.com).

A simple resolution I've found is making the deb sources in `/etc/apt/sources.list` more explicit by adding `[arch=amd64]` to each line.

Considering snapcraft already modifies the image by running updates and adding foreign architectures, it seems reasonable to at least make the repositories the image uses a bit more explicit architecture-wise.

The particular lines in question where I think this happens:
https://github.com/canonical/craft-archives/blob/cc4a9346eee627948d76ade16041f96b4ad70ff1/craft_archives/repo/apt_sources_manager.py#L220-L224

[Callahan Kovacs (mr-cal)]

Hi Dilyn, thanks for the bug report. Can you provide the snapcraft.yaml that you're using?

I'm suspecting you're using the `package-repositories` keyword.

[Dilyn (dilyn-corner)]

I am indeed using the package-repositories keyword to add the snappy-dev PPA. Here's (most) of the snapcraft.yaml:

name: foo-kernel
summary: The foo kernel
description: |
  The foo kernel packaged as a snap
grade: stable
build-base: core22
confinement: strict
type: kernel
adopt-info: kernel

architectures:
  - build-on: [amd64, arm64]
    build-for: [arm64]

package-repositories:
  - type: apt
    suites: [jammy]
    components: [main]
    architectures: [arm64]
    key-id: 78E1918602959B9C59103100F1831DDAFC42E99D
    url: http://ppa.launchpad.net/snappy-dev/image/ubuntu

parts:
  firmware:
    plugin: dump
    source: https://git.launchpad.net/~canonical-kernel-snaps/+git/kernel-snaps-uc22
    source-type: git
    source-branch: main
    source-depth: 1
    stage-packages:
      - linux-firmware
      - wireless-regdb
    prime:
      - firmware
    organize:
        lib/firmware: firmware

  kernel:
    after: [firmware]
    plugin: kernel
    source: git+ssh://git.launchpad.net/<red>
    source-depth: 1
    source-type: git
    source-branch: master-next
    kernel-kconfigflavour: "<red>"
    kernel-with-firmware: false
    kernel-compiler: aarch64-linux-gnu-gcc
    kernel-image-target: Image
    kernel-initrd-compression: gz
    kernel-initrd-stage-firmware: true
    kernel-device-trees:
      - <red>
    kernel-kdefconfig:
      - defconfig
    build-packages:
      - bison
      - debhelper
      - device-tree-compiler
      - dpkg-dev
      - fakeroot
      - flex
      - initramfs-tools-core
      - libfdt-dev
      - libssl-dev
      - u-boot-tools
      - wget
      - to arm64:
        - binutils-aarch64-linux-gnu
        - gcc-aarch64-linux-gnu
      - else:
        - gcc
    override-pull: |
      craftctl default

      . debian/debian.env

      deb_ver=$(dpkg-parsechangelog -l ${DEBIAN}/changelog -S version)
      craftctl set version=local-${deb_ver}

      fakeroot debian/rules clean

      # Kernel plugin will error if this file does not exist
      touch debian.ein/config/config.common.ports
    stage:
      - -modules
    prime:
      - -Image*
      - -System.map*
      - -config*
      - -dtbs
      - -initrd.img*
    organize:
      kernel.img: Image

{snip - module and extra firmware, FIT image}

[Callahan Kovacs (mr-cal)]

Perfect, thanks. Have you seen this bug? https://bugs.launchpad.net/snapcraft/+bug/2009278

I believe your bug has the same root cause.

[Dilyn (dilyn-corner)]

This problem is indeed issue 1 reported by Michal!

[Dilyn (dilyn-corner)]

An extension of this issue also exists, take the below example snapcraft.yaml:

name: my-snap-name # you probably want to 'snapcraft register <name>'
base: core22 # the base snap is the execution environment for this snap
version: '0.1' # just for humans, typically '1.2+git' or '1.3.2'
summary: Single-line elevator pitch for your amazing snap # 79 char long summary
description: |
  This is my-snap's description. You have a paragraph or two to tell the
  most important story about your snap. Keep it under 100 words though,
  we live in tweetspace and your description wants to look good in the snap
  store.

grade: devel # must be 'stable' to release into candidate/stable channels
confinement: devmode # use 'strict' once you have the right plugs and slots

architectures:
  - build-on: amd64
    build-for: arm64

parts:
  my-part:
    # See 'snapcraft plugins'
    plugin: nil
    stage-packages:
      - libgtk-3-0:${CRAFT_TARGET_ARCH}

After using my proposed solution in comment #1 of modifying the /etc/apt/sources.list file to identify all of the correct packages, we notice that even though libgtk-3-0 is specifically for arm64, we end up with several amd64 debs in our final snap:

ls ../parts/my-part/stage_packages/*amd64*.deb
../parts/my-part/stage_packages/dconf-gsettings-backend_0.40.0-3_amd64.deb
../parts/my-part/stage_packages/dconf-service_0.40.0-3_amd64.deb
../parts/my-part/stage_packages/fontconfig_2.13.1-4.2ubuntu5_amd64.deb
../parts/my-part/stage_packages/gtk-update-icon-cache_3.24.33-1ubuntu2_amd64.deb
../parts/my-part/stage_packages/libdconf1_0.40.0-3_amd64.deb
../parts/my-part/stage_packages/libdeflate0_1.10-2_amd64.deb
../parts/my-part/stage_packages/libfontconfig1_2.13.1-4.2ubuntu5_amd64.deb
../parts/my-part/stage_packages/libgdk-pixbuf-2.0-0_2.42.8+dfsg-1ubuntu0.2_amd64.deb
../parts/my-part/stage_packages/libicu70_70.1-2_amd64.deb
../parts/my-part/stage_packages/libjbig0_2.1-3.1ubuntu0.22.04.1_amd64.deb
../parts/my-part/stage_packages/libjpeg-turbo8_2.1.2-0ubuntu1_amd64.deb
../parts/my-part/stage_packages/libjpeg8_8c-2ubuntu10_amd64.deb
../parts/my-part/stage_packages/libtiff5_4.3.0-6ubuntu0.4_amd64.deb
../parts/my-part/stage_packages/libwebp7_1.2.2-2ubuntu0.22.04.1_amd64.deb
../parts/my-part/stage_packages/libxml2_2.9.13+dfsg-1ubuntu0.3_amd64.deb
../parts/my-part/stage_packages/shared-mime-info_2.1-2_amd64.deb

Listing all of these debs explicitly in our stage-packages list trims the list a bit, but we still end up with some amd64 debs:

ls -1 ../parts/my-part/stage_packages/*amd64*.deb
../parts/my-part/stage_packages/dconf-gsettings-backend_0.40.0-3_amd64.deb
../parts/my-part/stage_packages/dconf-service_0.40.0-3_amd64.deb
../parts/my-part/stage_packages/libdconf1_0.40.0-3_amd64.deb
../parts/my-part/stage_packages/libicu70_70.1-2_amd64.deb
../parts/my-part/stage_packages/libxml2_2.9.13+dfsg-1ubuntu0.3_amd64.deb
../parts/my-part/stage_packages/shared-mime-info_2.1-2_amd64.deb

What's interesting is that sometimes I end up with a greater of fewer number of staged amd64 debs. It doesn't seem deterministic, and there's some odd behavior with stage-package cleanup (packages won't be cleaned if a new arch is specified for them, f...

[Dilyn (dilyn-corner)]

It seems that this issue is resolved with 7.5.2

[Callahan Kovacs (mr-cal)]

Glad it's working for you. The change should have been released with snapcraft 7.5.1.

@tigarmo improved this behavior (from the snapcraft side) with https://github.com/canonical/craft-archives/pull/96 and https://github.com/snapcore/snapcraft/pull/4303.

I think the launchpad side of this is still open because `dpkg --add-architecture` shouldn't break `apt update` and I think it has to do with what architectures are advertised in repos. (@tigarmo has more details)

[Dilyn (dilyn-corner)]

I think this is fully resolved by this commit:
https://github.com/snapcore/snapcraft/commit/1db0f397b49c95800ac606a9f06b2179170fcd51