Comment 14 for bug 1852502

Revision history for this message
nikhil kshirsagar (nkshirsagar) wrote (last edit ):

Adding some information on the bug as advised by John (@jam). A user who is hitting this has the following environment,

-$ juju --version
2.8.13-bionic-amd64

$ juju status
Model Controller Cloud/Region Version SLA Timestamp Notes
controller sj1-prod-maas-01-juju-controller maas_cloud 2.9.16 unsupported 09:49:29-08:00 upgrade available: 2.9.22

Machine State DNS Inst id Series AZ Message
0 started 10.100.40.233 kq34k7 bionic default Deployed
1 started 10.100.40.234 juju-2 bionic default Deployed
2 started 10.100.40.235 juju-3 bionic default Deployed

juju:PRIMARY> show databases
admin 0.000GB
backups 0.498GB
blobstore 0.454GB
config 0.000GB
juju 1.084GB
local 1.311GB
logs 0.205GB
juju:PRIMARY> show collections
system.keys
system.users
system.version
juju:PRIMARY>

They run a script to take daily backups of the juju controller and then a command to only keep the latest backup but the backups are filling up the disk on all the 3 juju controller nodes. I believe the script failure causes backups to keep piling up because the script to only keep the latest backup only runs upon successful backup.

even if they run this command manually they get the message that there is only most current backup.

# juju remove-backup -m sj1-prod-maas-01-juju-controller:admin/controller --keep-latest
WARNING no backups to remove, 20210921-190247.c06182d1-ffc0-454f-8418-dac55ad882a8 most current

But when they login into the controllers they see juju backup directories for every backup run in /tmp
:/tmp$ ls -ld jujuBackup*
drwx------ 3 root root 4096 Dec 20 10:58 jujuBackup-016549046
drwx------ 3 root root 4096 Dec 20 11:37 jujuBackup-042799675
drwx------ 3 root root 4096 Dec 20 10:58 jujuBackup-063462045
drwx------ 3 root root 4096 Dec 20 10:58 jujuBackup-270640115

These are the commands used

/snap/bin/juju create-backup -m sj1-prod-maas-01-juju-controller:admin/controller --filename $DEST_DIR/juju-backup-sj1-prod-maas-01-juju-controller-$TIME.tar.gz "sj1-prod-maas-01-juju-controller" 2>&1

juju remove-backup -m sj1-prod-maas-01-juju-
controller:admin/controller --keep-latest