Can't mix ports.ubuntu.com and archive.ubuntu.com (building multi-arch snaps)

[Michał Sawicz]

The use case is building snaps containing both 32- and 64-bit packages (i.e. to support older software in the graphics-core22 stack [1]).

I've not found a way to build a amd64 + i386 (or arm64 + armhf) snap, because either repository refresh fails (case A), or snapcraft holds on to an archive cache from before `dpkg --add-architecture` (case B).

## Case A

Given this YAML:
(using the ARM example as this stops working when building a snap for an architecture that doesn't exist in the added repository - this way you can reproduce on amd64)

```yaml
name: arch-test
base: core22
version: '0.1'
summary: Architecture test
description: |
  Multi-architecture test.

grade: devel
confinement: devmode

package-repositories:
  - type: apt
    url: http://ports.ubuntu.com/ubuntu-ports/
    suites: [jammy]
    components: [main, universe]
    architectures: [armhf]
    key-id: F6ECB3762474EDA9D21B7022871920D1991BC93C
    key-server: keyserver.ubuntu.com

parts:
  part:
    plugin: nil
    stage-packages:
    - libc6:armhf
```

```plain
$ snapcraft
Launching instance...
Add repository architecture: armhf
Failed to refresh package list: failed to run apt update.
Failed to execute pack in instance.
Full execution log: '/home/michal/.cache/snapcraft/log/snapcraft-20230304-212001.588931.log'
```

Because:

```plain
2023-03-04 21:20:35.447 :: 2023-03-04 20:20:35.054 :: E: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/jammy/main/binary-armhf/Packages 404 Not Found [IP: 185.125.190.36 80]
2023-03-04 21:20:35.447 :: 2023-03-04 20:20:35.054 :: E: Failed to fetch http://security.ubuntu.com/ubuntu/dists/jammy-security/main/binary-armhf/Packages 404 Not Found [IP: 185.125.190.39 80]
2023-03-04 21:20:35.447 :: 2023-03-04 20:20:35.054 :: E: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/jammy-updates/main/binary-armhf/Packages 404 Not Found [IP: 185.125.190.39 80]
2023-03-04 21:20:35.447 :: 2023-03-04 20:20:35.054 :: E: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/jammy-backports/main/binary-armhf/Packages 404 Not Found [IP: 185.125.190.39 80]
2023-03-04 21:20:35.447 :: 2023-03-04 20:20:35.054 :: E: Some index files failed to download. They have been ignored, or old ones used instead.
```

The apt repository definitions in the build base are not architecture-specific, so adding an architecture outside of that repository results in above errors.

## Case B

Another approach:

```yaml
# ... (no package-repositories)

parts:
  aaa:
    plugin: nil
    stage-packages:
    - libc6

  arch-add:
    plugin: nil
    override-pull: |
      if [ "${CRAFT_TARGET_ARCH}" == "amd64" ]; then
        dpkg --add-architecture i386
        apt-get update
      fi

  pack-i386:
    after: [arch-add]
    plugin: nil
    stage-packages:
    - libc6:i386
```

This will fail to find `libc6:i386`, because Snapcraft seems to hold to an apt cache from before processing parts:

```plain
Launching instance...
Executed: pull arch-add
Stage package not found in part 'pack': libc6:i386.
Failed to execute pack in instance.
Full execution log: '/home/michal/.cache/snapcraft/log/snapcraft-20230304-213302.303889.log'
```

Re-running the build will complete, because Snapcraft will have reloaded the now-multiarch cache.

[1] https://github.com/MirServer/glvnd-core22/

[Michał Sawicz (saviq)]

There may be two bugs here in reality:
1) non-architecture-specific repositories on the build bases
   - Snapcraft could mangle them to avoid the problem
   - the build bases could have [arch=<>] bits added to the default repos
   - Snapcraft could allow advanced grammar for `package-repositories`

2) Snapcraft holding on to apt caches across parts
   - this may be found to be by design
   - but re-reading the cache in `pull` steps could also help

[Michał Sawicz (saviq)]

I've managed to find a workaround - as long as the `dpkg --add-architecture` executes _before_ any `stage-packages` is processed (enforced through `after:` or… changing the part names so they execute in a different order - they're run alphabetically), things work again.

[Callahan Kovacs (mr-cal)]

Thanks for the bug report, Michal!

I'm able to reproduce both issues. Problem #1 is an issue; Snapcraft is not handling package-repositories correctly in cross-architecture builds.

Problem #2 is interesting. It looks like craft-parts doesn't update its package cache between parts. I'm guessing there was an assumption that a part would not call `dpkg` or `apt` in an override script. If issue #1 was fixed, then users would not run into issue #2.

[Sergio Schvezov (sergiusens)]

We need grammar on package-repositories

[Tiago Nobrega (tigarmo)]

Hi Michal, the latest version of stable Snapcraft (7.5.2) updates the handling of architectures in core22 snaps: `dpkg --add-architecture` is only called if the new arch is "compatible" with the host one (so amd64 + i386, and arm64 + armhf). Can you please confirm whether this addresses your case "A"?

[Michał Sawicz (saviq)]

@tigarmo thanks for the heads up, it does look like this works indeed!

This allows me to use `package-repositories:` entries that don't necessarily correspond to the the current build architecture.

Advanced grammar would still be nice, and avoid some unnecessary downloads :)

[Tiago Nobrega (tigarmo)]

Thanks for confirming! We'll close this and possibly work on the advanced grammar in a separate ticket.

[Callahan Kovacs (mr-cal)]

Marking as "fix released" on behalf of @tigarmo