create-backup not working with default backup-dir

[Andreas Hamacher]

juju create-backup
ERROR while creating backup archive: while dumping juju state database: rename /tmp/snap.juju-db/tmp/jujuBackup-388657813/juju-backup/dump /tmp/jujuBackup-388657813/juju-backup/dump: no such file or directoryackup
ERROR while creating backup archive: while dumping juju state database: rename /tmp/snap.juju-db/tmp/jujuBackup-388657813/juju-backup/dump /tmp/jujuBackup-388657813/juju-backup/dump: no such file or directory

I found a workaround in https://bugs.launchpad.net/juju/+bug/1959705

Apparmor prevents the usage of /tmp
changing the backup-dir as suggested in 1959705 helps.
The purpose of this bug report is to promote a better default. e.g. backup-dir=/var/snap/juju-db/common/ ( likely depending on the OS or controller version )

Controller Version: 2.9.37
OS: jammy
Cloud/region: localhost/localhost

[John A Meinel (jameinel)]

The issue is that the juju-db snap doesn't have the right to access /tmp on the controller machine, so we should be staging the backup somewhere else, possibly /var/lib/juju/backups, or something along those lines. Either that, or we should be giving juju-db access to tmp in the snap definition.

[Ian Booth (wallyworld)]

On Dec 1, our CI tests also started failing

ERROR while creating backup archive: while dumping juju state database: rename /tmp/snap.juju-db/tmp/jujuBackup-3436056306/juju-backup/dump /tmp/jujuBackup-3436056306/juju-backup/dump: no such file or directory

Until then the tests were fine and we've not made changes in this area. Juju uses $TMPDIR by default as a base dir (unless backup-dir is set) and I thought that the snap env would be set up such that TMPDIR was set correctly to point to somewhere snap specific that was writeable.

If I shell into a confined snap, I see that TMPDIR is set to /tmp. So this seems like an upstream change to me.

This commit landed when things stated breaking. Seems could be related.

https://git.launchpad.net/snapd/commit/?id=a9812c467597159dd7fdbbb2da16b41c1a002962

I filed an upstream bug

https://bugs.launchpad.net/snapd/+bug/1999109

[Ian Booth (wallyworld)]

I added some extra code to juju to deal with the issue

[Ian Booth (wallyworld)]

https://github.com/juju/juju/pull/15006

[Mateusz Kozakowski (hypeitnow)]

Hi, any chance of a pr to juju beta channel?

[Mateusz Kozakowski (hypeitnow)]

Can an approach be helpful here?
mkdir /var/snap/juju-db/common/tmp
sudo mount --bind /tmp /var/snap/juju-db/common/tmp

EDIT

Unfortunately does not work

root@juju-maas-2:~# ll /var/snap/juju-db/common/tmp/
total 761264
drwxrwxrwt 15 root root 4096 Dec 17 14:42 ./
drwx------ 5 root root 4096 Dec 17 12:56 ../
drwxrwxrwt 2 root root 4096 Dec 13 09:31 .ICE-unix/
drwxrwxrwt 2 root root 4096 Dec 13 09:31 .Test-unix/
drwxrwxrwt 2 root root 4096 Dec 13 09:31 .X11-unix/
drwxrwxrwt 2 root root 4096 Dec 13 09:31 .XIM-unix/
drwxrwxrwt 2 root root 4096 Dec 13 09:31 .font-unix/
drwxr-xr-x 4 root root 4096 Dec 16 23:40 bkp/
-rw-r--r-- 1 root root 779465185 Dec 16 23:44 juju-controller.tgz
-rw------- 1 root root 0 Dec 13 09:35 juju-machine-lock
drwx------ 4 root root 4096 Dec 13 09:31 snap-private-tmp/
drwx------ 3 root root 4096 Dec 13 09:31 systemd-private-c45b760838cb4927870de045cc36ddc2-ModemManager.service-I2Tlzz/
drwx------ 3 root root 4096 Dec 13 09:31 systemd-private-c45b760838cb4927870de045cc36ddc2-systemd-logind.service-YycNbR/
drwx------ 3 root root 4096 Dec 13 09:31 systemd-private-c45b760838cb4927870de045cc36ddc2-systemd-resolved.service-6GvnaA/
drwx------ 3 root root 4096 Dec 13 09:31 systemd-private-c45b760838cb4927870de045cc36ddc2-systemd-timesyncd.service-wsZS07/
drwx------ 3 root root 4096 Dec 13 17:06 systemd-private-c45b760838cb4927870de045cc36ddc2-upower.service-SChbOO/
drwx------ 2 root root 4096 Dec 13 09:35 tmux-0/
root@juju-maas-2:~# ll /tmp
total 761264
drwxrwxrwt 15 root root 4096 Dec 17 14:42 ./
drwxr-xr-x 19 root root 4096 Dec 13 09:31 ../
drwxrwxrwt 2 root root 4096 Dec 13 09:31 .ICE-unix/
drwxrwxrwt 2 root root 4096 Dec 13 09:31 .Test-unix/
drwxrwxrwt 2 root root 4096 Dec 13 09:31 .X11-unix/
drwxrwxrwt 2 root root 4096 Dec 13 09:31 .XIM-unix/
drwxrwxrwt 2 root root 4096 Dec 13 09:31 .font-unix/
drwxr-xr-x 4 root root 4096 Dec 16 23:40 bkp/
-rw-r--r-- 1 root root 779465185 Dec 16 23:44 juju-controller.tgz
-rw------- 1 root root 0 Dec 13 09:35 juju-machine-lock
drwx------ 4 root root 4096 Dec 13 09:31 snap-private-tmp/
drwx------ 3 root root 4096 Dec 13 09:31 systemd-private-c45b760838cb4927870de045cc36ddc2-ModemManager.service-I2Tlzz/
drwx------ 3 root root 4096 Dec 13 09:31 systemd-private-c45b760838cb4927870de045cc36ddc2-systemd-logind.service-YycNbR/
drwx------ 3 root root 4096 Dec 13 09:31 systemd-private-c45b760838cb4927870de045cc36ddc2-systemd-resolved.service-6GvnaA/
drwx------ 3 root root 4096 Dec 13 09:31 systemd-private-c45b760838cb4927870de045cc36ddc2-systemd-timesyncd.service-wsZS07/
drwx------ 3 root root 4096 Dec 13 17:06 systemd-private-c45b760838cb4927870de045cc36ddc2-upower.service-SChbOO/
drwx------ 2 root root 4096 Dec 13 09:35 tmux-0/
root@juju-maas-2:~# df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 364M 1.3M 363M 1% /run
/dev/sda2 18G 13G 4.7G 73% /
tmpfs 1.8G 0 1.8G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 50M 13M ...

[Ian Booth (wallyworld)]

RE: comment#5, the fix is already in the 2.9 and 3.0 edge snaps, so you can try those if you want access now.

we don't do beta releases. The next 2.9 release will be a 2.9.38 candidate early next year, and 3.0.3 shortly after that.

[Mateusz Kozakowski (hypeitnow)]

Would
snap refresh juju --channel 3.0/edge
be enough? Or do I need to reinstall the client?
What about the controllers, do I need to remove and bootstrap again? I am afraid I have to
Right now in client I have 2 revisions
root@juju-client:~# ll /var/snap/juju/
total 20
drwxr-xr-x 5 root root 4096 Dec 18 07:47 ./
drwxr-xr-x 8 root root 4096 Dec 11 21:40 ../
drwxr-xr-x 2 root root 4096 Oct 25 18:39 21474/
drwxr-xr-x 2 root root 4096 Oct 25 18:39 21739/
drwxr-xr-x 2 root root 4096 Nov 5 17:12 common/
lrwxrwxrwx 1 root root 5 Dec 18 07:47 current -> 21739/

I am not an expert in snaps field, so any help appreciated

Thank you

[Erik Lönroth (erik-lonroth)]

Is this related to the error output we have?

ERROR while creating backup archive: while dumping juju state database: rename /tmp/snap.juju-db/tmp/jujuBackup-487338168/juju-backup/dump /tmp/jujuBackup-487338168/juju-backup/dump: no such file or directory

Also discussed here on discourse: https://discourse.charmhub.io/t/failing-create-controller-backup/7831

[Giuseppe Petralia (peppepetra)]

I am on juju 2.9.32 and hitting this bug.

As I workaround I can change the backup dir with:

juju model-config -m controller backup-dir="/var/snap/juju-db/current"

and then I am able to create a backup

[Haw Loeung (hloeung)]

Confirmed the workaround above works, setting `backup-dir` to "/var/snap/juju-db/current".

Thanks.