Apparmor profile not generated correctly after failed snap refresh

[Daniel Lagreca]

After one or more cycles of a snap refresh failing, the snap refresh eventually succeeded but was left in a non-functional state because the apparmor profile was not correctly generated.

---
The snap was failing to refresh due to a bug in the post-refresh hook, and was getting rolled back. This happened 4 times each day on the scheduled refresh interval. As far as I know, the snap was correctly rolled back each time.

A fix was pushed and then the snap refresh succeeded, leaving the latest version of the snap installed. However the app armor profile (/writable/system-data/var/lib/snapd/apparmor/profiles/snap.<SNAPNAME>.<SNAPSERVICENAME>) was not correctly re-generated. Specifically this line:
```
@{SNAP_REVISION}="680"
```

Was set to the previous revision (which had been repeatedly rolled-back), and not the currently installed revision. This was causing all access to the installed snaps filesystem to be blocked by apparmor.

We saw this on about 10% of our deployed devices, which points at some kind of non-deterministic issue.

The issue, when detected, can be manually resolved by disabling and then enabling the snap, which forces the regeneration of the apparmor profile.

---
In the debug output uploaded, "results.log" the failing snap was "everactive-evernet", specifically it could not access this file:
```
13T19:09:32Z" level=panic msg="Error opening database: open /var/snap/everactive-evernet/676/evernetreceiver.db: permission denied"
```

[Daniel Lagreca (dlagreca)]

We have seen this same issue happen again since this bug was filed. It actually does not appear to be related to the snap refresh failing before it succeeds. That is, we have seen the apparmor profile fail to get updated after only one successful snap refresh to a newer revision.

Is there any further debugging information, or logs we can get that may help us understand the root of the issue?

Thank you!!

[Daniel Lagreca (dlagreca)]

Hello!

We are continuing to see this same issue happen as we push snaps to the snap store. This is affecting our system reliability and is a very big issue for us.

*Is there any way we can get it escalated?*

Again - Thank you so much for any help!!

[Michael Vogt (mvo)]

Thanks for your bugreport. What is the best way for us to try to reproduce the issue?

From your problem description it sounds like:
- Create a snap with a failing post-refresh hook
- then refresh to that a couple of times to it and see it rollback
- then refresh to a correct one and observe that it has the wrong apparmor profile?

Or are there any other hints how we could reproduce this?

[Daniel Lagreca (dlagreca)]

Michael,

Unfortunately we have been unable to deterministically reproduce it.

We have seen it happen on a strictly successful snap refresh, so the failed refresh does not seem to be necessary to reproduce the issue (but could still increase the odds of reproduction).

I'd try refreshing in a loop, and checking that the apparmor profile is correctly generated each time. If you have access to our private snap store, I can point you to the exact snap which is causing the issue, so you can use it in testing.