If an external drive is used for backups, Timevault will write backups even if the drive is not present

[gaten]

Timevault 7.5-1
Ubuntu 7.10

When using an external hard drive and selecting a backup directory on it (/media/disk1/backup), if the hard drive in unmounted from the system and Timevault runs any backup procedures during that time Timevault will re-create the backup directory and begin to copy backups to that folder. This problem compounds itself when the hard drive is plugged back into the computer and cannot mount itself in the usual location because the directory already exists. Timevault will now happily fill up your root partition (or whatever partition /media is on) until there is no more space. No notification from Timevault is made.

[snookie (snookie)]

I can confirm that this happens as well.

In fact, I can't seem to get it to work on a usb device at all. Timevault must be off, the drive must be mounted, and then TimeVault must be started. Otherwise, if the drive is not mounted, the directory is created and then plugging in the USB drive will result in incorrect operation. In other words. If you start timevault, the directory /media/disk1 isn't there because the drive isn't mounted, so timevault makes that directory. If you mount the drive, the drive will mount at /media/disk-1 or some other directory.

If you reboot the computer, timevault starts and creates the directory because USB devices are not automounted at boot. The user has to click the icon on the desktop for the drive to actually mount to the media directory. So restarting the computer with the usb as backup will also not work resulting in the behavior describe above.

[Daniel Stiner (danstiner)]

So basically TimeVault should ask the user before running off and creating directories willy-nilly?
That makes perfect sense to me, but I'm not sure the best way to ask the user, because timevault runs as a service under root, with no user associated with it.
I hope SourceContact comes back and works on this project again, it has a lot of potential I think, and good luck getting a fix soon.

[Shep (c-shepherd7)]

I'm having this issue also using an external hard drive. It gets very annoying. There are a few good ways that I can think of fixing this.

The first would be to require that the Backup directory chosen in the preferences already exists. Let TimeVault create subdirectories, but not the root backup dir.

The next option would be to try to guess if they are using an external drive based on conventions. That would mean if the root backup directory exists inside /mnt or /media, then make sure the drive is mounted before backing up to it.

Last option would be to make the option of using an external drive part of the app prefs. Then handle is like above.

[Andrew Stromme (astromme)]

Well, you could determine if it is an external drive by checking when the user first sets the backup directory. Then, you could have TimeVault sit in a 'paused' mode with the applet saying "please plug in the backup drive to allow backups" until the user does just that.

[Andrew Stromme (astromme)]

Hmm, I've now got an itch to scratch it seems.

My ideas:

I've got an external drive for my laptop. I want to keep my TimeVault files on said drive, so...
TimeVault should notice that it's a removable drive when I set the root save path to the drive
TimeVault should only write my backups when the drive is plugged in
TimeVault should remind me to plug in my drive when I try and make a manual snapshot and it's not plugged in

Possibilities:

TimeVault should cache my snapshots while my drive is unplugged, and then copy them over?
TimeVault should recognize other TimeVault backups stored on drives when they are plugged in?

To achieve this I'm going to use HAL via the DBus methods.

I'll be working in this branch:
https://code.edge.launchpad.net/~astromme/timevault/timevault-external

We'll see what happens. I haven't used python in a while and I have no idea how TimeVault's code works yet =P.

If anyone has other ideas on things regarding external drives, feel free to add comments.

[Andrew Stromme (astromme)]

At the moment (revision 44 of timevault-external) timevault sets a uuid for the drive (if it's other than root) that stores the backups. Then, upon each configuration reload (Meaning at application startup or after any conf change) it checks for a valid backup drive before doing anything. If the drive is not present, it shows an error status in timevault-notifier and prompts the user to plug in the backup drive.

I still haven't figured how to make it continue to check for the backup drive (or get automatically notified when it gets plugged in). HAL will probably do this for me. Also, once the drive has been plugged in and timevault starts using it it will not give it up until the process is stopped. To change this behavior, I think I have to dig deeper into how timevault works and how it stores information on the drive. I'm afraid that it won't be easy. Although, there might be a way to set up a 'disable' event where it releases control of the drive.

[Daniel Stiner (danstiner)]

I think this is an absolutely huge step forward Andrew, I uninstalled TimeVault specifically because of problems with it creating its own happy directory to work in before the external drive was mounted, defeating the purpose of using an external drive.
I can't thank you enough for making these changes, I might just build and install your branch.
On the other hand, I think there needs to be some fundamental changes in both the backup daemon and the user interface before TimeVault could be included by default in Ubuntu, but what if? If things were fixed, this could be an incredible default app to provide Ubuntu users with peace of mind.

[Andrew Stromme (astromme)]

I agree, I think some fundamental changes are needed for me to be able to use it as my backup solution, especially in the backend. However, I think that the concepts behind TimeVault are great and exactly where backups need to be headed.

I'm glad someone appreciates the changes =). I'll try to clean up my work and implement those last few things that I mentioned yesterday.

[mikeh (mikeh-ptfd)]

>TimeVault should cache my snapshots while my drive is unplugged, and then copy them over?
>TimeVault should recognize other TimeVault backups stored on drives when they are plugged in?

I like this sort of, I am using a separate dir (/home/.private/.snapshots/...) for my backups using retrospekt, but like the timevault using inotify. This is great, snapshoting just a file when it needs it. I like that. I do like retrospekt's timeline in konqueror tho.

What would be great is the possibility of using that directory for my snapshots, then mirroring them to my usb drive when it is plugged in in teh office. Or maybe when I plug in the usb drive it makes a monthly, or weekly snap plus mirroring the currently snaps.

[Andrew Stromme (astromme)]

That is an interesting project. I wonder if TimeVault and retrospekt could
share a common backend. That would be a lot of work, though it would make
for an interesting result. I wouldn't mind having a desktop-antagonistic
backup solution.

On Tue, May 20, 2008 at 5:02 PM, mikeh <email address hidden> wrote:

> >TimeVault should cache my snapshots while my drive is unplugged, and then
> copy them over?
> >TimeVault should recognize other TimeVault backups stored on drives when
> they are plugged in?
>
> I like this sort of, I am using a separate dir
> (/home/.private/.snapshots/...) for my backups using retrospekt, but
> like the timevault using inotify. This is great, snapshoting just a file
> when it needs it. I like that. I do like retrospekt's timeline in
> konqueror tho.
>
> What would be great is the possibility of using that directory for my
> snapshots, then mirroring them to my usb drive when it is plugged in in
> teh office. Or maybe when I plug in the usb drive it makes a monthly, or
> weekly snap plus mirroring the currently snaps.
>
> --
> If an external drive is used for backups, Timevault will write backups if
> the drive is not present
> https://bugs.launchpad.net/bugs/191681
> You received this bug notification because you are a direct subscriber
> of the bug.
>

[ross (ross-rossmoore)]

I just want to add my support for getting this bug fixed. It's the one thing stopping me using TimeVault - and I imagine most non-server type users, especially laptop users. I love the concept though, and at least Andrew's development stops it mucking things up even if it isn't a totally polished solution yet.

Good luck with developing this, I look forward to seeing it in the repositories sometime soon. Let me know if I can help - although I don't know Python or HAL, so I don't suppose I can!

[Nic (ntetreau)]

Hi, just wanted to add my support to fixing this bug, it is really, really important especially for laptop users making backup to external USB drive. You might want to take a look at how Conduit has fixed this issue, has we can now use Conduit to keep a local and external drive in sync, and it will recognize the USB drive using a uuid as well (it save a .conduit file on the drive as well).

I'd be prepared to help testing, just let me know how to download/make and use your branch. Thanks again for your hard work!

[jfestrada (jfestrada)]

Hi, I've found a workarrond to solve this problem, I'll try to explain the temporary solution that I have used.
- The first thing that I made was set the mount path for my esternal hd into gnome (/media/hdusb)
- The second was create a link into my /home directory with mount directory for my external disk as target
         $ sudo ln -s /media/hdusb/backup /home/TimeVault
- At the end I changed the config for timevault setting the backup directory with the link that I had created in the second step ( /home/TimeVault).

With this configuration, timevault doesn't start when the disk is not mount, and if I plug the disk I have to restart the service to get timevault working again.

[Axel Pospischil (apos)]

It would be another very easy way to handle the way of getting control over this problem:
I think some of people out there, that are doing backups had these problems using external drives. I am using simple lock files on the backup drive to recognize if a certain drive or directory is mounted or not. Another way is the evaluation of the UUID of a mounted drive as mentioned above.

However, thinking for advanced users and the possibilities of scenarios in general: Add an option to execute a script before/after timevault starts a scheduled backup.

This will give the user / admin full control over the backup-process, mounting, unmounting, preparing of backup drives (e.g. dvdrw's) ... It will be possible to add timevault in a more complex backup chain. Scripts might be written in different languages.

Timevault should evaluate the return code of the "before"-script as followed:
 0 - (scheduled) backup will be proceeded / no error
 1 - (scheduled) backup will NOT be executed / no error message**
 2 - (scheduled) backup will NOT be executed / error message to the user and a simple log entry "script execution failed"

** this is for automated backups when the user does not want to be informed, because he knows what he is doing (e.g. switches on the external usb backup drive manually).

The "after"-script return code should be evaluated like this:
 0 - do nothing
 1 - give an error message / log entry.

For you, the developer this should be a one-liner in the code. If you like to hide this option for the average user, you could simply enable script execution via /etc/timevault.conf :
 'executeBefore': '/root/timevault/before.sh',
 'executeAfter': '/root/timevault/after.sh',

Greets Axel

[nanolight (adrian-lamy)]

(please excuse my english)

Hi! I would love to use timevault to backup/restore my /home but I can see that it causes problems with an external hard-drive.
I haven't seen any new version of TimeVault since last year, so I was wondering if the project was dead or have been slowed down?
Does the actual version of this branch is ready to use?
I'm not a developer at all, but I can say that it seems to be a very useful application, so I encourage you a lot for this work!

Sincerely

[Daniel Stiner (danstiner)]

nanolight, there is no need to excuse your English. There are some who pick up English as a second or even third language, and as such do speak it with odd grammar, bad spelling and such, but your English is in fact near superb.
The Trunk of TimeVault is currently somewhat usable, but especially for something as large as your /home directory, I would currently recommend some other kind of backup method.
Andrew Stromme has been putting in some work on revising TimeVault from the core out, but he as yet does not have a stable finished product.

[Andrew Stromme (astromme)]

On Friday 02 January 2009 13:10:21 Daniel Stiner wrote:
> nanolight, there is no need to excuse your English. There are some who pick
> up English as a second or even third language, and as such do speak it with
> odd grammar, bad spelling and such, but your English is in fact near
> superb. The Trunk of TimeVault is currently somewhat usable, but especially
> for something as large as your /home directory, I would currently recommend
> some other kind of backup method. Andrew Stromme has been putting in some
> work on revising TimeVault from the core out, but he as yet does not have a
> stable finished product.

I'll add my 2 cents here as well.

If you feel comfortable working with a non-released branch my timevault-
external branch has basic support for external devices. I haven't messed with
any of the actual backup code so it shouldn't mess with your data at all, but
I can't guarantee that. There is more information at the branch
https://code.launchpad.net/~astromme/timevault/timevault-external .

Secondly, I have been working on a different backup program that originally had
some of the same goals as timevault but has now deviated quite a bit. I have
some preliminary work up at https://launchpad.net/timevaultng but nothing
close to usable by what I would consider the 'target' audience.

With luck, I'll have more time to work on my next gen concept of timevault
more in the coming weeks.

At the moment, I would suggest one of two things.
A: Use the released version of TimeVault if you like the concept of 'always
backing up'. As long as you stop the timevault daemon while your drive is
unplugged you should be fine.

B: Learn rdiff-backup and put it in a simple script. That is what I am doing
for the time being as it is really flexible. It is also what I am building
timevaultng around so you will have a head start on the concepts that will be
useful.

Andrew Stromme

[nanolight (adrian-lamy)]

@Daniel: (Thanks for the compliments, and it's my second language).
@ everybody:
I want to use TimeVault to backup documents from my /home (Photos, music, documents...) so it seems to be ok with TV.
I think I'm going to try to install it but I never used bazaar before, so I'm going to read carefully the documentation of bazaar before continuing.
I have got another question because of my complex /home. Does TV follow hardlinks? Because many of my documents are placed on a FAT32 partition and my home is composed of folders that are in fact hardlinks to other folders on that FAT32 partition.

@Andrew: You've said that the goals of your new project TimeVaultNG has deviated from these of TimeVault, what are these new goals?
And just to know, does the actual state of this branch answers the goals that were written at the beginning? :
"TimeVault should notice that it's a removable drive when I set the root save path to the drive
TimeVault should only write my backups when the drive is plugged in
TimeVault should remind me to plug in my drive when I try and make a manual snapshot and it's not plugged in"

When you write "As long as you stop the timevault daemon while your drive is unplugged you should be fine." does it significate that I have to quit TimeVault everytime I unplug my drive?

Thanks a lot for your answers and for your work!

Adrian