"juju backups" shows no backup, yet the "backup" mongodb database is 20GB big

[Junien F]

Hi,

The following is happening on a 2.3.8 controller :

$ juju backups
No backups to display.

Yet, on mongo :

juju:PRIMARY> show databases
[...]
backups 23.559GB

The space is in the "backups.chunks" collection, as expected :
backups.backups.chunks, storage = 24123 MB, index = 2 MB, total = 24124 MB

Also :
juju:PRIMARY> db.backups.files.find().pretty()
{
        "_id" : ObjectId("58ad38dec9ce0c289c734ea1"),
        "chunkSize" : 261120,
        "uploadDate" : ISODate("2017-02-22T07:19:19.725Z"),
        "length" : NumberLong("3822417218"),
        "md5" : "b7eab51f1901853bed387a4bd680acef",
        "filename" : "<uuid>"
}

How can we get rid of this ?

Thanks

[John A Meinel (jameinel)]

This feels like something started creating backup files, and then we removed the *reference* to the backups, but didn't actually remove the backup content.

[Paul Collins (pjdc)]

I stumbled across this again today, on the very same controller (based on the result of db.backups.files.find()) although this time the backups database only had 8GB in it. Per thumper and rick_h, I dropped the backups database to release the space. The controller was running 2.4.5 this time.

[Paul Collins (pjdc)]

Various non-chunks sections of the database, in case they help (Canonical-only links, sorry):

https://pastebin.canonical.com/p/jfQykfS5j4/ backups.files (same as above)
https://pastebin.canonical.com/p/QRcKdJmKD6/ storedResources
https://pastebin.canonical.com/p/Qpy4N2Mbsc/ txns

[Branislav Neskovic (branislav.neskovic)]

We have same issue on controller 2.8.7!

[John A Meinel (jameinel)]

Was this an upgrade from an older version, or only running 2.8 code?

On Tue, Apr 6, 2021 at 8:50 AM Branislav Neskovic <
<email address hidden>> wrote:

> We have same issue on controller 2.8.7!
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1789840
>
> Title:
> "juju backups" shows no backup, yet the "backup" mongodb database is
> 20GB big
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1789840/+subscriptions
>

[Branislav Neskovic (branislav.neskovic)]

Its 2.8.7 installation. I have checked on fresh 2.8.9 and there is the same situation.

[Branislav Neskovic (branislav.neskovic)]

Sorry. I saw this now, Remote backup was not created. And it was downloaded to local dir.

k8s@test2-controller:~$ juju create-backup -m test2-controller:controller

backup ID:
backup format version: 1
juju version: 2.8.7
series: focal

controller UUID: 7d956a97-a20e-4029-8887-aaa8b6d5efba
model UUID: a9aeeafb-5984-4d16-8a83-86dfda011a75
machine ID: 0
created on host: test2-controller

checksum: s/ulMkFEp2E8SGKNNpgUr1RIQwg=
checksum format: SHA-1, base64 encoded
size (B): 101226925
stored: 0001-01-01 00:00:00 +0000 UTC
started: 2021-04-07 07:35:48.730818889 +0000 UTC
finished: 2021-04-07 07:37:20.133856682 +0000 UTC

notes:

Remote backup was not created.
Downloaded to juju-backup-20210407-073548.tar.gz.

And mongo.db has no backups:

juju:PRIMARY> show databases;
admin 0.000GB
backups 0.000GB

I found this bug because my filesystem was filling up at this location /tmp/snap.juju-db/tmp .

k8s@test2-controller:~$ sudo du -h -d1 /tmp/snap.juju-db/tmp
276M /tmp/snap.juju-db/tmp/jujuBackup-218257021
276M /tmp/snap.juju-db/tmp/jujuBackup-178084792
276M /tmp/snap.juju-db/tmp/jujuBackup-847774230
.
.
.

[John A Meinel (jameinel)]

Branislav, so the issue isn't that the mongodb backups database is getting
too big, but that we aren't cleaning up /tmp after producing a backup. Is
that the case?
Can you submit that as a separate bug?

On Wed, Apr 7, 2021 at 3:55 AM Branislav Neskovic <
<email address hidden>> wrote:

> Sorry. I saw this now, Remote backup was not created. And it was
> downloaded to local dir.
>
> k8s@test2-controller:~$ juju create-backup -m
> test2-controller:controller
>
> backup ID:
> backup format version: 1
> juju version: 2.8.7
> series: focal
>
> controller UUID: 7d956a97-a20e-4029-8887-aaa8b6d5efba
> model UUID: a9aeeafb-5984-4d16-8a83-86dfda011a75
> machine ID: 0
> created on host: test2-controller
>
> checksum: s/ulMkFEp2E8SGKNNpgUr1RIQwg=
> checksum format: SHA-1, base64 encoded
> size (B): 101226925
> stored: 0001-01-01 00:00:00 +0000 UTC
> started: 2021-04-07 07:35:48.730818889 +0000 UTC
> finished: 2021-04-07 07:37:20.133856682 +0000 UTC
>
> notes:
>
> Remote backup was not created.
> Downloaded to juju-backup-20210407-073548.tar.gz.
>
> And mongo.db has no backups:
>
> juju:PRIMARY> show databases;
> admin 0.000GB
> backups 0.000GB
>
> I found this bug because my filesystem was filling up at this location
> /tmp/snap.juju-db/tmp .
>
> k8s@test2-controller:~$ sudo du -h -d1 /tmp/snap.juju-db/tmp
> 276M /tmp/snap.juju-db/tmp/jujuBackup-218257021
> 276M /tmp/snap.juju-db/tmp/jujuBackup-178084792
> 276M /tmp/snap.juju-db/tmp/jujuBackup-847774230
> .
> .
> .
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1789840
>
> Title:
> "juju backups" shows no backup, yet the "backup" mongodb database is
> 20GB big
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1789840/+subscriptions
>

[Branislav Neskovic (branislav.neskovic)]

Yes, that is coorrect...mongodb is ok but tmp is not cleaning up.
I will open case for that.