In AWS using spaces and fan network for a private network does not allow LXC containers to start

[Xav Paice]

Using Juju 2.9.12

If I use public networks with the subnet allocating public IP addresses, there's no problem. As soon as I start making network spaces in AWS, fan networking doesn't appear to work the way I'd expect.

When I add a private network:
- new subnet, no public IP address allocation
- add a NAT gateway to the public subnet
- add a route table for the private network that points 0.0.0.0/0 at the nat gateway

My new subnet is 10.0.132.0/24.

Added space:

juju add-space oam 10.0.132.0/24
juju reload-spaces

$ juju spaces
Name Space ID Subnets
alpha 0 10.0.129.0/24
                 10.0.130.0/24
                 10.0.131.0/24
                 252.2.0.0/15
                 252.4.0.0/15
                 252.6.0.0/15
oam 1 10.0.132.0/24
                 252.8.0.0/15

Add a new application

juju deploy cs:ubuntu --series focal --constraints "spaces=oam zones=eu-west-2a instance-type=t3.medium" --bind "oam" ubuntu

Machine 12 builds fine, and I can 'juju ssh' to it.

juju deploy cs:ubuntu --to lxd:12 --series focal --constraints "spaces=oam zones=eu-west-2a" --bind "oam" ubuntu-container

Unit fails to build:
machine-12: 23:12:57 INFO juju.container-setup initial container setup with ids: [12/lxd/0]
  machine-12: 23:12:57 INFO juju.packaging.manager Running: snap info lxd
  machine-12: 23:12:57 INFO juju.container.lxd LXD snap is already installed (channel: latest/stable); skipping package installation
  machine-12: 23:12:57 WARNING juju.container-setup not stopping machine agent container watcher due to error: setting up container dependencies on host machine: Not Found
  machine-12: 23:12:57 ERROR juju.container-setup starting container provisioner for lxd: setting up container dependencies on host machine: Not Found

Tried the exact same thing but with --series bionic, and got a different error:

```
machine-14: 00:43:27 WARNING juju.worker.provisioner machine 14/lxd/0 failed to start: unable to setup network: host machine "14" has no available FAN devices in space(s) "oam"

$ juju ssh 14 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 06:58:cc:b6:6d:6c brd ff:ff:ff:ff:ff:ff
    inet 10.0.132.193/24 brd 10.0.132.255 scope global dynamic ens5
       valid_lft 2176sec preferred_lft 2176sec
    inet6 fe80::458:ccff:feb6:6d6c/64 scope link
       valid_lft forever preferred_lft forever
3: fan-252: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether 32:6f:32:44:48:fd brd ff:ff:ff:ff:ff:ff
    inet 252.9.130.1/8 scope global fan-252
       valid_lft forever preferred_lft forever
    inet6 fe80::306f:32ff:fe44:48fd/64 scope link
       valid_lft forever preferred_lft forever
4: ftun0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8951 qdisc noqueue master fan-252 state UNKNOWN group default qlen 1000
    link/ether 32:6f:32:44:48:fd brd ff:ff:ff:ff:ff:ff
    inet6 fe80::306f:32ff:fe44:48fd/64 scope link
       valid_lft forever preferred_lft forever
5: lxdbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 82:a2:04:fd:70:a6 brd ff:ff:ff:ff:ff:ff
    inet 10.233.185.1/24 scope global lxdbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::80a2:4ff:fefd:70a6/64 scope link
       valid_lft forever preferred_lft forever
Connection to 10.0.132.193 closed.
```

It looks like the fan subnet 252.8.0.0/15 doesn't quite match up with what's configured on the machine 252.9.130.1/8, it's in the range but the subnet masks don't match.

[Joseph Phillips (manadart)]

The first issue has been raised already here:
https://bugs.launchpad.net/bugs/1942864

For the second, if you run "juju-subnets", does it show that the underlay 10.0.132.0/24 is in the constrained zone eu-west-2a?

[Joseph Phillips (manadart)]

https://github.com/juju/juju/pull/13319 should do the trick.