Instance files should be tombstoned and cleaned up later instead of immediately deleted

[Vish Ishaya]

This is a huge deployment concern. Recovering from accidental deletions by users is impossible if we immediately delete the instance files.

[justinsb (justin-fathomdb)]

Agree this is important.

Do you think there should be a administrative command that e.g. gets run in cron job or do you think we should schedule it ourselves somehow? I like the former, because it feels simpler and easier for everyone, but it's really just pushing more work onto the packagers / deployers.

Anyone from Rackspace want to weigh in with a recommended policy? Do we just leave the disk image lying around for - say - 24 hours? Do we archive it off to Swift? What happens if people launch and shut down a lot of machines quickly - could that DoS the cloud?

[Vish Ishaya (vishvananda)]

I like the manage command + cron job. Perhaps enabling tombstoning with a flag, because it is pretty useless in dev mode.

On Mar 22, 2011, at 11:18 PM, justinsb wrote:

> Agree this is important.
>
> Do you think there should be a administrative command that e.g. gets run
> in cron job or do you think we should schedule it ourselves somehow? I
> like the former, because it feels simpler and easier for everyone, but
> it's really just pushing more work onto the packagers / deployers.
>
> Anyone from Rackspace want to weigh in with a recommended policy? Do we
> just leave the disk image lying around for - say - 24 hours? Do we
> archive it off to Swift? What happens if people launch and shut down a
> lot of machines quickly - could that DoS the cloud?
>
> --
> You received this bug notification because you are a direct subscriber
> of the bug.
> https://bugs.launchpad.net/bugs/739601
>
> Title:
> Instance files should be tombstoned and cleaned up later instead of
> immediately deleted
>
> Status in OpenStack Compute (Nova):
> New
>
> Bug description:
> This is a huge deployment concern. Recovering from accidental
> deletions by users is impossible if we immediately delete the instance
> files.
>
> To unsubscribe from this bug, go to:
> https://bugs.launchpad.net/nova/+bug/739601/+subscribe

[justinsb (justin-fathomdb)]

I've realized that of course we don't actually delete things in the database, we just set the deleted_at flag, so this might not even be particularly hard...

There might be some helpful stuff which comes out of my work on making sure instances don't disappear every time the host is restarted which is a WIP here:
lp:~justin-fathomdb/nova/restart-instance

[justinsb (justin-fathomdb)]

Particularly now a lot of the hard work has been done for us...
https://code.launchpad.net/~jk0/nova/xs-rescue-periodic-tasks/+merge/54597

[justinsb (justin-fathomdb)]

lp:~justin-fathomdb/nova/restart-instance now includes a tombstone function, which we could use instead of actually deleting data.

I'd like to hear from people with experience in running Rackspace CloudServers on what we should be doing here.

[Johannes Erdfelt (johannes.erdfelt)]

This is controlled by reclaim_instance_interval, which got merged back in September 2011. Closing this out since it's been implemented for a while.