juju remove-backup should release space to the filesystem

[Junien F]

Hi,

juju backups are stored in mongodb, and can be pretty big. I removed 30GB worth of backups from a 2.2.6 controller yesterday (that was 3 backups), and the space was not released to the filesystem.

It may be worth compacting the collections in the "backups" database after removing a backup. The documentation [1] states that "compact only blocks operations for the database it is currently operating on.", so it should not block the rest of the operations.

Thanks

[1] https://docs.mongodb.com/v3.2/reference/command/compact/

[John A Meinel (jameinel)]

My gut says that there is probably a difference in Mongo 2.4 vs 3.2 and
what we could do. I believe 2.4 only supports "repairDatabase" to squish
everything, vs "compact" which can target specific collections.
I suppose in either case, backups should already be isolated in its own
database.
Going a little bit further, though. There is some concern if you were
removing lots of backups. We wouldn't really want to compact between each
removal. (I have 30, delete one, compact 29GB, delete another, compact
28GB, delete another, 27GB... etc.)
"juju remove-backup" doesn't yet support multiple backups to be removed. I
think it should.
I also wonder if it should take an optional flag about compacting the
database. (We could default it to compact and just allow you to specify
--no-compact, potentially)

We discussed in Cape Town the possibility of limits on the backup table,
akin to our other limits. (limit total number of preserved backups, total
size, etc, and having the automatically expire).

I could live with only changing it to auto compact on delete, since the
user intent is almost always to return disk space to the OS, not back to
Mongo. But I would like us to make that reasonably efficient in bulk.

On Thu, Jan 18, 2018 at 4:38 PM, Heather Lanigan <<email address hidden>
> wrote:

> ** Tags added: backup-restore
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1743986
>
> Title:
> juju remove-backup should release space to the filesystem
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1743986/+subscriptions
>

[Junien F (axino)]

Good thoughts. One nitpick though :

> I believe 2.4 only supports "repairDatabase" to squish everything

I thought this as well, but compact() appeared in 2.0, as per https://docs.mongodb.com/v2.4/reference/command/compact/

[Junien F (axino)]

Heh, although https://docs.mongodb.com/v3.2/reference/command/compact/ does say "On WiredTiger databases, this command will release unneeded disk space to the operating system." and WT appeared in 3.0

[John A Meinel (jameinel)]

mgopurge has code that switches between compact and repairDatabase based on
version, which makes me think that compact didn't actually do what we want
on 2.4. I haven't personally investigated it.

On Fri, Jan 19, 2018 at 9:19 AM, Junien Fridrick <<email address hidden>
> wrote:

> Heh, although https://docs.mongodb.com/v3.2/reference/command/compact/
> does say "On WiredTiger databases, this command will release unneeded
> disk space to the operating system." and WT appeared in 3.0
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1743986
>
> Title:
> juju remove-backup should release space to the filesystem
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1743986/+subscriptions
>

[Mario Splivalo (mariosplivalo)]

Juju should not be storing backups in the database by default, in my opinion.
We had a lot of issues dealing with customers that do regular backups (on a weekly/monthly basis), and the disk usage rapidly goes up in such situations.

Backups should be downloaded locally, and from there they could be easily restored if needed.

[Peter Sabaini (peter-sabaini)]

I'd like to second Marios sentiment here. The disk space usage can be quite unexpected, and the effect is exacerbated by the fact that space is not automatically free'd in mongodb. In fact, I have a juju 1.25.13 env here where even after deleting backups and running db.repairDatabase() (which is the documented way of freeing up disk in mongodb) the disk space that has been used is not reclaimed. And on the other hand, having backups of a database in the same database doesn't provide much benefit in a disaster recovery scenario.

[Erlon R. Cruz (sombrafam)]

I agree that juju should not be storing backups. It not only takes a lot of space that is not released after deleting the backups but also backups are retroactive (juju backup backups the backups collections) making DB and backup size explode in size.