lxd-support interface doesn't appear to get properly connected/ready

[Chris Patterson]

This sequence happens many times on our spread tests and on one install of LXD, `waitready` fails because it appears that lxd-interface wasn't properly connected/ready. It is the first time I've encountered such an error, but thought it worth reporting.

+ apt-get install -y snapd

Reading package lists...

Building dependency tree...

Reading state information...

snapd is already the newest version (2.42.1+18.04).

snapd set to manually installed.

The following packages were automatically installed and are no longer required:

  dns-root-data dnsmasq-base ebtables grub-pc-bin liblxc-common liblxc1

  libnuma1 libuv1 lxcfs lxd-client uidmap xdelta3

Use 'apt autoremove' to remove them.

0 upgraded, 0 newly installed, 0 to remove and 79 not upgraded.

+ '[' ubuntu-18.04-64 = ubuntu-16.04-64 ']'

+ '[' ubuntu-18.04-64 = ubuntu-18.04-64 ']'

+ apt-get remove --purge --yes lxd lxd-client

Reading package lists...

Building dependency tree...

Reading state information...

Package 'lxd' is not installed, so not removed

The following packages were automatically installed and are no longer required:

  dns-root-data dnsmasq-base ebtables grub-pc-bin liblxc-common liblxc1

  libnuma1 libuv1 lxcfs uidmap xdelta3

Use 'apt autoremove' to remove them.

The following packages will be REMOVED:

  lxd-client*

0 upgraded, 0 newly installed, 1 to remove and 79 not upgraded.

After this operation, 11.0 MB disk space will be freed.

(Reading database ... 145633 files and directories currently installed.)

Removing lxd-client (3.0.3-0ubuntu1~18.04.1) ...

Processing triggers for man-db (2.8.3-2ubuntu0.1) ...

+ snap install lxd

2020-03-26T13:23:59Z INFO Waiting for restart...

2020-03-26T13:24:00Z INFO Waiting for restart...

lxd 3.23 from Canonical* installed

+ lxd waitready --timeout=30

cat: /proc/self/attr/current: Permission denied

/snap/lxd/13987/commands/lxd: 6: exec: aa-exec: Permission denied

[Maciej Borzecki (maciek-borzecki)]

I suspect a new core is being pulled, wonder if that increases the chances of this problem appearing?

[Callahan Kovacs (mr-cal)]

We're still struggling with this bug in our spread tests. When running spread tests for *craft applications, we occasionally see the following failure:

+ snap install lxd
error: cannot perform the following tasks:
- Run configure hook of "lxd" snap if present (run hook "configure":
-----
cat: /proc/self/attr/current: Permission denied
/snap/lxd/23680/snap/hooks/configure: 5: exec: aa-exec: Permission denied
-----)

The failure is occurring in clean VMs with a stock Ubuntu image (18.04, 20.04, and 22.04). The failure rate for these steps is very low - less than 1% of the time. In our github workflow, we run ~350 spread tests, so the failure of the overall work flow is around 10-15% (although some weeks it has been closer to 50%).

The environment setup is brief. We apply the following steps to the image:

  apt-get install -y snapd
  snap install snapd
  snap wait system seed.loaded

  if [ "$SPREAD_SYSTEM" = "ubuntu-18.04-64" ] || [ "$SPREAD_SYSTEM" = "ubuntu-20.04-64" ]; then
      # Remove lxd and lxd-client deb packages as our implementation (pylxd) does not
      # nicely handle the snap and deb being installed at the same time.
      apt-get remove --purge --yes lxd lxd-client
  fi
  # Install and setup the lxd snap
  snap install lxd

------------------

Attempted solutions

I tried the 3 primary suggestions from this thread (https://forum.snapcraft.io/t/snapped-lxd-has-stopped-working-aa-exec-permission-denied/2356) with no luck:

1. Running snap interfaces (now snap connections)

This was suggested to check if the LXD snap is connected to lxd-support interface. This displays nothing, because the LXD snap is not installed yet.

2. Running snap install core20 and snap revert core20

This gives the error:
```
  + snap revert core20

  error: cannot revert "core20": no revision to revert to
```

3. Running snap install core20 and snap refresh core20

This still produces the same exec: aa-exec: Permission denied error.

I’ve captured the journal of a failure here (https://paste.ubuntu.com/p/XnyFRsvVfH/), with the failing section posted below:

```
oct142128-679423 audit[1763]: AVC apparmor="DENIED" operation="open" profile="snap.lxd.activate" name="/proc/1763/attr/current" pid=1763 comm="cat" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
oct142128-679423 audit[1764]: AVC apparmor="DENIED" operation="exec" profile="snap.lxd.activate" name="/usr/bin/aa-exec" pid=1764 comm="daemon.activate" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
oct142128-679423 audit[1764]: AVC apparmor="DENIED" operation="exec" profile="snap.lxd.activate" name="/usr/bin/aa-exec" pid=1764 comm="daemon.activate" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
oct142128-679423 kernel: kauditd_printk_skb: 4 callbacks suppressed
oct142128-679423 kernel: audit: type=1400 audit(1665782994.672:46): apparmor="DENIED" operation="open" profile="snap.lxd.activate" name="/proc/1763/attr/current" pid=1763 comm="cat" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
oct142128-679423 kernel: audit: type=1400 audit(1665782994.672:47): apparmor="DENIED" operation="exec" profile="snap.lxd.activate" name="/usr/bin/aa-exec" pid=1764 comm="daemon.activate" requested_m...