snapd 2.60.2 - cannot create user data directory: /home/DOMAIN/USERNAME/snap/APP/VERSION: Permission denied

[Patrick Sudderth]

Just received reports from ~80 users that an auto-upgrade of snapd from 2.59.5 to 2.60.2, which occurred last night (Ubuntu 2204 Pro), has broken all snaps. Only way to resolve is to "sudo snap revert snapd". No changes have been made to AppArmor. Error displayed is:

cannot create user data directory: /home/DOMAIN/USERNAME/snap/APP/VERSION: Permission denied

apparmor config: /home/DOMAIN/

root@hostname:/home/DOMAIN/USER# snap list snapd --all
Name Version Rev Tracking Publisher Notes
snapd 2.59.5 19457 latest/stable canonical✓ snapd,disabled
snapd 2.60.2 19993 latest/stable canonical✓ snapd
root@hostname:/home/DOMAIN/USER# snap revert snapd

[Patrick Sudderth (patricksudderth)]

I assume this is related to this note in the release notes. However, there's no documentation on how to configure apparmor for snapd:
"Apparmor userspace is vendored inside the snapd snap"

[Shane Terpening (shane-terpening)]

Downgrading snap to 2.59.5 allows package installation and correct launch. Upgrading to 2.60.2 breaks package installed after upgrade. It is repeatable.

[Patrick Sudderth (patricksudderth)]

@shane - Agreed. Although we now can't find a way to go back to 2.59.5 on at least one of our VM's. Revert doesn't work because we tried the edge version and only two versions appear to be cached locally.

[Shane Terpening (shane-terpening)]

@Patrick thanks for sharing that. I could see that becoming an issue for me as well. I looked at the apparmor settings and the snap homedir settings yesterday, and was able to determine that snap is respecting the home directory location, as changing it to a non-existent directory produced a different error than the permissions issue. I put in an e-mail to try to get traction on this issue as it impacts common enterprise configurations and the impact is not correctly conveyed by "this bug impacts you and 2 other people"

[Patrick Sudderth (patricksudderth)]

Thank you! Yes, in our case, the /home/DOMAIN/Username is needed as these are AD-Joined Citrix VDI VM's.

[Sergio Cazzolato (sergio-j-cazzolato)]

Hi, Patrick, thanks for raising this issue. Could you please share the list of snaps installed in those machines? I'll try to reproduce it.

Also could you please paste the full journal log "sudo journalctl -u snapd"

Thanks

[Patrick Sudderth (patricksudderth)]

@sergio : We've been testing with Firefox and IntelliJ but I believe it's all. Here's an excerpt from our VDI runbook which captures all the master image deployed snaps.

snap install terraform --classic
snap install pycharm-community --classic
snap install postman
snap install dbeaver-ce
snap install datagrip --classic
snap install sublime-text --classic
snap install beekeeper-studio
snap install termius-app
snap install gitkraken --classic
snap install intellij-idea-community --classic
snap install powershell --classic
snap install azbrowse --classic
snap install storage-explorer
snap install kontena-lens --classic
snap install juju --classic
snap install kubectl --classic
snap install helm3
snap install firefox
snap install snap-store
snap install intellij-idea-ultimate --classic
snap install --edge qts3browser

[Patrick Sudderth (patricksudderth)]

@sergio : here's everything recent:

Jun 29 16:59:05 ip-10-129-3-42 systemd[1]: Stopping Snap Daemon...
Jun 29 16:59:05 ip-10-129-3-42 systemd[1]: snapd.service: Deactivated successfully.
...skipping...
Aug 22 12:34:46 HOSTNAME systemd[1]: Stopping Snap Daemon...
Aug 22 12:34:46 HOSTNAME systemd[1]: snapd.service: Deactivated successfully.
Aug 22 12:34:46 HOSTNAME systemd[1]: Stopped Snap Daemon.
Aug 22 12:34:46 HOSTNAME systemd[1]: snapd.service: Consumed 1min 19.519s CPU time.
-- Boot 5bba8b27d83c462a978af19930e08101 --
Aug 22 12:35:53 HOSTNAME systemd[1]: Starting Snap Daemon...
Aug 22 12:35:55 HOSTNAME snapd[789]: overlord.go:272: Acquiring state lock file
Aug 22 12:35:55 HOSTNAME snapd[789]: overlord.go:277: Acquired state lock file
Aug 22 12:35:55 HOSTNAME snapd[789]: daemon.go:247: started snapd/2.60.2 (series 16; classic) ubuntu/22.04 (amd64) linux/6.2.0-1009-aws.
Aug 22 12:35:55 HOSTNAME snapd[789]: daemon.go:340: adjusting startup timeout by 3m25s (pessimistic estimate of 30s plus 5s per snap)
Aug 22 12:35:56 HOSTNAME snapd[789]: backends.go:58: AppArmor status: apparmor is enabled and all features are available (using snapd provided apparmor_parser)
Aug 22 12:35:56 HOSTNAME systemd[1]: Started Snap Daemon.
Aug 22 14:00:23 HOSTNAME snapd[789]: main.go:155: Exiting on terminated signal.
Aug 22 14:00:23 HOSTNAME snapd[789]: overlord.go:516: Released state lock file
Aug 22 14:00:23 HOSTNAME systemd[1]: Stopping Snap Daemon...
Aug 22 14:00:23 HOSTNAME systemd[1]: snapd.service: Deactivated successfully.
Aug 22 14:00:23 HOSTNAME systemd[1]: Stopped Snap Daemon.
Aug 22 14:00:23 HOSTNAME systemd[1]: snapd.service: Consumed 2.361s CPU time.
-- Boot 713524b2c82340a2a28d111201bd752d --
Aug 22 15:01:22 HOSTNAME systemd[1]: Starting Snap Daemon...
Aug 22 15:01:24 HOSTNAME snapd[765]: overlord.go:272: Acquiring state lock file
Aug 22 15:01:24 HOSTNAME snapd[765]: overlord.go:277: Acquired state lock file
Aug 22 15:01:24 HOSTNAME snapd[765]: daemon.go:247: started snapd/2.60.2 (series 16; classic) ubuntu/22.04 (amd64) linux/6.2.0-1009-aws.
Aug 22 15:01:24 HOSTNAME snapd[765]: daemon.go:340: adjusting startup timeout by 3m25s (pessimistic estimate of 30s plus 5s per snap)
Aug 22 15:01:24 HOSTNAME snapd[765]: backends.go:58: AppArmor status: apparmor is enabled and all features are available (using snapd provided apparmor_parser)
Aug 22 15:01:25 HOSTNAME systemd[1]: Started Snap Daemon.
Aug 22 15:02:22 HOSTNAME snapd[765]: storehelpers.go:773: cannot refresh: snap has no updates available: "amazon-ssm-agent", "azbrowse", "bare", "beekeeper-studio", "canonical-livepatch", "core", ">
Aug 22 15:03:43 HOSTNAME snapd[765]: handlers.go:2338: Downgrading snapd to version "2.60.1+git1186.g661258f", discarding all existing snap AppArmor profiles
Aug 22 15:03:43 HOSTNAME snapd[765]: daemon.go:521: gracefully waiting for running hooks
Aug 22 15:03:43 HOSTNAME snapd[765]: daemon.go:523: done waiting for running hooks
Aug 22 15:03:43 HOSTNAME snapd[765]: overlord.go:516: Released state lock file
Aug 22 15:03:43 HOSTNAME systemd[1]: snapd.service: Deactivated successfully.
Aug 22 15:03:43 HOSTNAME systemd[1]: snapd.service: Consumed 5.884s CPU time.
Aug 22 15:03...

[Fabio Augusto Miranda Martins (fabio.martins)]

Hi Patrick and Shane,

Isn't the problem you're dealing with what's described in https://snapcraft.io/docs/home-outside-home ?

I ran a test in a VM where the home directory for the user fabio is /home/canonical/fabio.

With snapd 2.55, I also used to get the same problem you described:

fabio@jammy-snap-bug:~$ sudo snap install kubectl --classic
kubectl 1.28.0 from Canonical✓ installed

fabio@jammy-snap-bug:~$ kubectl
cannot create user data directory: /home/canonical/fabio/snap/kubectl/3049: Permission denied

fabio@jammy-snap-bug:~$ snap list snapd --all
Name Version Rev Tracking Publisher Notes
snapd 2.55.3 15534 latest/stable canonical✓ snapd

I can't really fix it while at 2.55 because the homedirs option didn't exist back then:

fabio@jammy-snap-bug:~$ sudo snap set system homedirs=/home/canonical/
error: cannot perform the following tasks:
- Run configure hook of "core" snap (run hook "configure": cannot set "core.homedirs": unsupported system option)

If I upgrade to 2.60 I still get the same problem:

Make snap "snapd" (19993) available to the system -
2023-08-23T21:09:31Z INFO Waiting for automatic snapd restart...
snapd 2.60.2 from Canonical✓ refreshed
fabio@jammy-snap-bug:~$ kubectl config get
cannot create user data directory: /home/canonical/fabio/snap/kubectl/3049: Permission denied

But I can fix it with setting homedirs:

fabio@jammy-snap-bug:~$ sudo snap set system homedirs=/home/canonical/
fabio@jammy-snap-bug:~$ kubectl config get
error: unknown command "get"
See 'kubectl config -h' for help and examples

Not sure if what you're dealing with is what I described or something different that I may be missing.

Regards,
Fabio Martins

[Patrick Sudderth (patricksudderth)]

@fabio : I thought that might be the case... didn't work.

USERNAME@HOSTNAME:~$ firefox
cannot create user data directory: /home/DOMAIN/USERNAME/snap/firefox/3026: Permission denied
USERNAME@HOSTNAME:~$ sudo snap set system homedirs=/home/DOMAIN/
[sudo] password for USERNAME:
USERNAME@HOSTNAME:~$ firefox
update.go:85: cannot change mount namespace according to change mount (/var/lib/snapd/hostfs/usr/share/gimp/2.0/help /usr/share/gimp/2.0/help none bind,ro 0 0): cannot open directory "/var/lib": permission denied
update.go:85: cannot change mount namespace according to change mount (/var/lib/snapd/hostfs/usr/share/libreoffice/help /usr/share/libreoffice/help none bind,ro 0 0): cannot open directory "/var/lib": permission denied
update.go:85: cannot change mount namespace according to change mount (/var/lib/snapd/hostfs/usr/share/xubuntu-docs /usr/share/xubuntu-docs none bind,ro 0 0): cannot open directory "/var/lib": permission denied
cannot create user data directory: /home/DOMAIN/USERNAME/snap/firefox/3026: Permission denied
USERNAME@HOSTNAME:~$

[Shane Terpening (shane-terpening)]

@fabio, +1 to Patrick's sentiment:

$ sudo snap get system homedirs
/home/DOMAIN/

$ snap install vlc
vlc 3.0.18 from VideoLAN✓ installed
$ vlc
cannot create user data directory: /home/DOMAIN/USERNAME/snap/vlc/3078: Permission denied

This behavior persists across any number of snaps.

[Shane Terpening (shane-terpening)]

Also worth noting, the directories exist:
$ ls /home/DOMAIN/USERNAME/snap/vlc/
3078 3677 common current

[Fabio Augusto Miranda Martins (fabio.martins)]

Hi Shane and Patrick,

Thanks for the details. I've been able to reproduce the problem and we'll be investigating it. FYI, this has also been reported by other users and is being discussed here as well:

https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/2032668

https://forum.snapcraft.io/t/upgrade-snapd-from-2-59-5-to-2-60-2-gives-apparmor-denied-on-app-launch-for-every-app/36492/8

[Sergio Cazzolato (sergio-j-cazzolato)]

I tried to reproduce the error but I couldn't.

Could you please share the output of this script https://github.com/snapcore/snapd/blob/master/debug-tools/snap-debug-info.sh

[Shane Terpening (shane-terpening)]

Thanks for updating Fabio. I am investigating holding my fleet temporarily on snap 2.59.5. Is there and considerations for this that might not be obvious

[HG (hgncl)]

Symptoms are: I click on the launch icon for Firefox and it doesn't open. I run `firefox` in the terminal and I get:

$ firefox
cannot create user data directory: /home/my.ad.domain/domainaccount/snap/firefox/3026: Permission denied

That directory is present on the system with the following permissions:

$ ls -l ~/snap/firefox
total 36
drwxr-xr-x 4 domainaccount krb5users 4096 Feb 27 15:08 2356
drwxr-xr-x 4 domainaccount krb5users 4096 Mar 23 21:53 2432
drwxr-xr-x 4 domainaccount krb5users 4096 Apr 5 12:16 2487
drwxr-xr-x 4 domainaccount krb5users 4096 Apr 23 12:29 2579
drwxr-xr-x 4 domainaccount krb5users 4096 May 30 18:01 2710
drwxr-xr-x 4 domainaccount krb5users 4096 Jun 26 09:38 2800
drwxr-xr-x 4 domainaccount krb5users 4096 Aug 16 12:44 2987
drwxr-xr-x 4 domainaccount krb5users 4096 Aug 24 12:46 3026
drwxr-xr-x 12 domainaccount krb5users 4096 Feb 27 15:08 common
lrwxrwxrwx 1 domainaccount krb5users 4 Aug 24 09:44 current -> 3026

In the system journal, the launch attempts are accompanied by :

ug 24 19:29:23 phylab16-014 systemd[18099]: Started Application launched by gnome-shell.
Aug 24 19:29:23 phylab16-014 systemd[18099]: Started snap.firefox.firefox-ca2161bb-300b-43e8-86f6-4646171c4e5e.scope.
Aug 24 19:29:23 phylab16-014 audit: BPF prog-id=172 op=LOAD
Aug 24 19:29:23 phylab16-014 audit[59226]: SYSCALL arch=c000003e syscall=321 success=yes exit=9 a0=5 a1=7ffe932ec460 a2=80 a3=1000 items=0 ppid=18239 pid=59226 auid=364137 uid=364137 gid=1999 euid=0 suid=0 fsuid=0 egid=1999 sgid=1999 fsgid=1999 tty=(none) ses=73 comm="snap-confine" exe="/snap/snapd/19993/usr/lib/snapd/snap-confine" subj=/snap/snapd/19993/usr/lib/snapd/snap-confine key=(null)
Aug 24 19:29:23 phylab16-014 audit: PROCTITLE proctitle=2F736E61702F736E6170642F31393939332F7573722F6C69622F736E6170642F736E61702D636F6E66696E65002D2D6261736500636F7265323000736E61702E66697265666F782E66697265666F78002F7573722F6C69622F736E6170642F736E61702D657865630066697265666F78
Aug 24 19:29:23 phylab16-014 audit[59226]: AVC apparmor="DENIED" operation="open" class="file" profile="/snap/snapd/19993/usr/lib/snapd/snap-confine" name="/home/my.ad.domain/domainaccount/" pid=59226 comm="snap-confine" requested_mask="r" denied_mask="r" fsuid=364137 ouid=364137
Aug 24 19:29:23 phylab16-014 firefox_firefox.desktop[59226]: cannot create user data directory: /home/my.ad.domain/domainaccount/snap/firefox/3026: Permission denied
Aug 24 19:29:23 phylab16-014 audit[59226]: SYSCALL arch=c000003e syscall=257 success=no exit=-13 a0=4 a1=d40927 a2=b0000 a3=0 items=0 ppid=18239 pid=59226 auid=364137 uid=364137 gid=1999 euid=364137 suid=0 fsuid=364137 egid=1999 sgid=1999 fsgid=1999 tty=(none) ses=73 comm="snap-confine" exe="/snap/snapd/19993/usr/lib/snapd/snap-confine" subj=/snap/snapd/19993/usr/lib/snapd/snap-confine key=(null)
Aug 24 19:29:23 phylab16-014 audit: PROCTITLE proctitle=2F736E61702F736E6170642F31393939332F7573722F6C69622F736E6170642F736E61702D636F6E66696E65002D2D6261736500636F7265323000736E61702E66697265666F782E66697265666F78002F7573722F6C69622F736E6170642F736E61702D657865630066697265666F78
Aug 24 19:29:23 phylab16-014 audit: BPF prog-id=172 op=UNLOAD

About my system:

root@phylab16-014:...

[Patrick Sudderth (patricksudderth)]

Does anyone have a way to get back to 2.59.5? Revert isn't working on a few of our VM's where we've tried other versions and overwritten the local 2.59.5.

[Fabio Augusto Miranda Martins (fabio.martins)]

There you go, Sergio: https://pastebin.canonical.com/p/pdfrBrRN52/

[Patrick Sudderth (patricksudderth)]

USERNAME@HOSTNAME:~/Downloads$ wget https://github.com/snapcore/snapd/blob/master/debug-tools/snap-debug-info.sh
--2023-08-24 13:11:06-- https://github.com/snapcore/snapd/blob/master/debug-tools/snap-debug-info.sh
Resolving github.com (github.com)... 140.82.113.4
Connecting to github.com (github.com)|140.82.113.4|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 23510 (23K) [text/plain]
Saving to: ‘snap-debug-info.sh’

snap-debug-info.sh 100%[===================>] 22.96K --.-KB/s in 0s

2023-08-24 13:11:07 (182 MB/s) - ‘snap-debug-info.sh’ saved [23510/23510]

USERNAME@HOSTNAME:~/Downloads$ chmod +x snap-debug-info.sh
USERNAME@HOSTNAME:~/Downloads$ ./snap-debug-info.sh
./snap-debug-info.sh: line 1: payload:allShortcutsEnabled:false: command not found
USERNAME@HOSTNAME:~/Downloads$

[Lenard Fudala (lmf-mx)]

@Fabio Since the channels are all updated

$ snap info snapd
...
channels:
  latest/stable: 2.60.2 2023-08-22 (19993) 42MB -
  latest/candidate: 2.60.2 2023-08-10 (19993) 42MB -
  latest/beta: 2.60.2 2023-08-04 (19993) 42MB -
  latest/edge: 2.60.2+git1248.gdcb8ad2 2023-08-24 (20081) 42MB -

you would have to manually grab a version. https://launchpad.net/ubuntu/+source/snapd/+publishinghistory

[HG (hgncl)]

@sergio-j-cazzolato I have output from snap-debug-info.sh both with 2.59 and 2.60 installed. It contains a lot of information about my system, including usernames and AD domain names, and I don't trust myself to find-replace in files of this size. Is there somewhere more private that I can send these files to? I don't have access to Canonical's pastebin.

[HG (hgncl)]

@patricksudderth You have to run the script as root or with sudo in front of the command, and redirect the output to a file. I said `sudo ./snap-debug-info.sh &> somefile.log` to get stderr as well as stdout.

[HG (hgncl)]

According to the NEWS.md on Github, this was added as a feature.

"Apparmor userspace is vendored inside the snapd snap"

https://github.com/snapcore/snapd/blob/5a0f7c76112412ebfae5ed8b22b51fc6f3b14b70/NEWS.md#new-in-snapd-260

[Patrick Sudderth (patricksudderth)]

@H : I don't see any personal data in this file. Anything obscure that I should pull before posting?

[Patrick Sudderth (patricksudderth)]

@H : Saw that in the release notes. The bug is with people who aren't using /home/USERNAME and, instead, /home/DOMAIN/USERNAME. This is easy to manage with the native apparmor (dpkg-reconfigure apparmor). No documentation on how to do it with this version of snapd. Setting the home directory with snap doesn't work either, as noted in the thread above.

[HG (hgncl)]

@patricksudderth If you are OK with the entire internet knowing your home directory path, hostname, and username, then post away. The home directory path in mine contains the Active Directory domain of my employer and my AD account name, so I wish to conceal it.

I concur that this needs to work with non-standard home directory paths. It won't be a simple case of "revert a commit" though.

[Ville Ruohonen (vruohonen)]

Our university runs ~100 or so of VMware Horizon desktops with Ubuntu. This issue affects all of them.