handle symlinks in syncdaemon

[Lucio Torre]

syncdaemon treats symlinks as normal files or directories. syncdaemon will break if the user creates a loop. handle this.

[Rick McBride (rmcbride)]

removing placeholder as I'm shifting this to the proper project

[Vincenzo Ciancia (vincenzo-ml)]

I tried today and it does not seem to add symlinks at all.

[Kaïs Bejaoui (kais)]

I can confirm what Vincenzo Ciancia is saying. Symlinks were not synced with the server.

[Roshan George (roshan-george)]

I can confirm this too. Does someone have an older version of the client? It seems from this bug report that it used go across symlinks?

[Necromancer (denis-endis)]

Yep, without symlinks (at least for me) it's completely useless.

[Paul Sladen (sladen)]

Can somebody clarify what is happening? Is u1sync-daemon trying to spider across a symlink?

AFAICT, this symlinks should be handled in just the same way as any other filesystem/remote filesystem/rsync/etc handles them. That is the symlink should be copied (and the same with all other special nodes such as fifos/device notes) and the symlink should not be dereferenced.

[Necromancer (denis-endis)]

Nope, it DOES NOT follow symlinks.

For example, i have a symlink for my Pidgin logs to sync it with my other computers, the only way i can make it is to place a symlink from pidgin settings directory to ~/Ubuntu One/My Files.

When i use Dropbox, it syncs all files, treating symlinks as directories, which is very convenient in my case.

On the other hand, some people may want symlinked directories to be treated as symlinks, but not as directory. So maybe an option for that would be a good idea.

[John Lenton (chipaca)]

On Wed, Jun 24, 2009 at 11:52:09AM -0000, Paul Sladen wrote:
> Can somebody clarify what is happening? Is u1sync-daemon trying to
> spider across a symlink?

symlinks are ignored by the client, and not supported by the server.

> AFAICT, this symlinks should be handled in just the same way as any
> other filesystem/remote filesystem/rsync/etc handles them. That is the
> symlink should be copied (and the same with all other special nodes such
> as fifos/device notes) and the symlink should not be dereferenced.

that is the problem with symlinks: some people want them to be copied
as symlinks, and some people want them dereferenced. And although real
POSIX remote filesystems should support them, many remote/distributed
filesystems do not; to name two, neither dropbox nor SMB (afaict) do:
the first dereferences the link, and the second runs around in
circles, screaming and waving its paws in the air.

[Elliot Murphy (statik)]

We do have a lovely feature planned to address the legitimate user need that is met by dropbox dereferencing symlinks though: in a later version of syncdaemon you will be able to add arbitrary directories to the list of dirs to be kept in sync.

[Necromancer (denis-endis)]

Well, that's just great!

The only question i have is when the next version is planned to be available.

[Paul Sladen (sladen)]

IIRC, Samba >=3.0 (using CIFS) defaults to using the Unix extensions, so sends user/ground ownership, permission along with the data for readlink() when requested. Samba <3.0 (using SMB) defaults to dereferencing.

So if you're connected from a modern (~2003) Apple Mac, or Linux box, then symlinks should do the right thing. For Samba <3.0, you should be able to browse the symlinks, but not (remotely) create them.

Necromancer: Deferencing symlinks and treating them like they are hard-linked directories is probably going to end up in knots. As you've noted, you can store the actual data that wants syncing, and then symlink /to/ that.

[Kaïs Bejaoui (kais)]

Creating a symlink to the ~/Ubuntu One/My Files won't work is you're using a OS that can't mount ext3. Putting my data on my home partition means that I won't be able to access it from Windows.

So this workaround is unfortunately not really one.

[Necromancer (denis-endis)]

If symlinks are handled as files/directories then it should work fine.

Dropbox does handle ntfs links under windows just fine so as linux symlinks.

But handling symlinks as symlinks won't work, that's true.

[Paul Sladen (sladen)]

Kais: as an a-side: having your $HOME directory on a partition that doesn't handle unix permission, meta data, extended attributes or special files is a /really bad/ idea. What you're probably better doing is having your /home partition be on a standard Unix filesystem (ext*, xfs, ...) and then having a separate vfat/ntfs partition where you store you shared files. For convenience you can then doing something like:

  ln -s /mnt/somewhere ~/Desktop/shared-with-ms-window

and it'll be there on your Desktop.

[Vincenzo Ciancia (vincenzo-ml)]

Il giorno gio, 25/06/2009 alle 09.41 +0000, Necromancer ha scritto:
> If symlinks are handled as files/directories then it should work fine.
>
> Dropbox does handle ntfs links under windows just fine so as linux
> symlinks.
>
> But handling symlinks as symlinks won't work, that's true.
>

Handling symlinks as files must not be done by default, I hope. Your
primary target are ubuntu users, they can use links for linking e.g.
huge files that you certainly don't want to duplicate.

Vincenzo

[Necromancer (denis-endis)]

>Handling symlinks as files must not be done by default, I hope. Your
>primary target are ubuntu users, they can use links for linking e.g.
>huge files that you certainly don't want to duplicate.

Yeah, right. That's why they don't work now at all :)

[Genachka (genachka)]

I agree with the comments that the target folks are Ubuntu users and as such the symlinks should be supported. I was very excited to see this app come about. My primary use is to sync customer files on my laptop back my desktops. But I've already got the customer files organized just as like it on a different partition. All I would want to do is to create a symlink for them and not have to duplicate them (I couldn't even if I wanted due to size anyway). So I really hope symlinks is the direction this takes! :)

[Thea Barnes (tsbarnes)]

I was definitely hoping, given its target users, that Ubuntu One would support symlinks directly (as actual links, not duplicating or ignoring them). I was really hoping to have a nice way to keep my themes, icons, and various other things in sync between my systems without the pain of something like DropBox where every single icon that would normally be a link to another filename instead becomes a copy, and suddenly a 2mb icon theme takes up 8mb. But if Ubuntu One completely ignores symlinks then it's *even less* usable for that, since all the icons linked to multiple names would go missing entirely. I really think the best solution for the majority of users is to handle symlinks as links, not duplicating or ignoring them, which would also keep it consistent with how most Ubuntu users expect their files to work.

[John Lenton (chipaca)]

On Tue, Aug 25, 2009 at 10:34:47AM -0000, Scott Barnes wrote:
> I was definitely hoping, given its target users, that Ubuntu One would
> support symlinks directly (as actual links, not duplicating or ignoring
> them). I was really hoping to have a nice way to keep my themes, icons,
> and various other things in sync between my systems without the pain of
> something like DropBox where every single icon that would normally be a
> link to another filename instead becomes a copy, and suddenly a 2mb icon
> theme takes up 8mb. But if Ubuntu One completely ignores symlinks then
> it's *even less* usable for that, since all the icons linked to multiple
> names would go missing entirely. I really think the best solution for
> the majority of users is to handle symlinks as links, not duplicating or
> ignoring them, which would also keep it consistent with how most Ubuntu
> users expect their files to work.
>

the problem, and the reason we decided to postpone symlinks, is that
there isn't a clear "this is what users expect"; there are at least
two separate camps that argument in pretty much the same way (“you
should do it this way, because *everybody* expects it to work this
way!”), but those ways are actually opposite: Some people expect
symlinks to be stored as symlinks, some people expect them to be
followed.

We will, at some point, support symlinks. However, this won't happen
before we also support synchronizing multiple folders (rather than
just ~/"Ubuntu One", which will make the other half of the people
happy). Unfortunately these features won't make it into Karmic
(although you will be able to use them if you use our PPA).

Cheers,

[Michele (mikelito)]

If I may add some thoughts, there is a subtle problem both if one decides to follow symlinks, and if multiple directories are to be handled. Consider this scenario: say that I have two machines synced via UbuntuOne.
I want to sync
machine1:~user1/folder_a/folder_b

now, let's say I add ~user1/folder_a/folder_b to the synced list on machine1. what will then happen on machine2?
recreating a new directory with the absolute path is impossible, as I might even have a different username on machine2. Let alone what would happen if I want to sync with different OS.

Let's say then that on other machines folder_b will appear as a sub-folder of ~/Ubuntu One.
Now, what happens if I want folder_b to be in a different location on machine_2?
this gets really messy, because if I move folder_b out of Ubuntu One, what will happen to
machine1:~user1/folder_a/folder_b
?

I think that a multiple-folder approach needs to be REALLY well-thought, for example there could be a 1-1 mapping between folders on different machines, and if I share a new folder, on syncing the other clients will have to ask "where do you want me to put this new, synced folder?"

symlink-following leads to same problems. I was first a bit pissed off by this "feature" being absent, but now I second the developers on the fact that this has to be well-considered and very cleanly designed.

Cheers!

[cldx (cldx3000)]

If theres gonna be multiple folders support i think it wouldnt even be neccesary to have an actual Folder "Ubuntu One" at all. Instead Folders should just have a right-click Option for U1 saying "Synch Folder" which will synch it to U1, now logging in to an associated System would have to ask me where to keep this Folder synched like suggested above. Those Folders should have emblems for being synched, like a small cloud or U1 logo (like Dropbox has too, you gotta admit it makes sense). Maybe they could show the synch paths of all asssociated machines too if i hover the icon or if i look at the Properties...

[Arfyness (inatrik)]

1. It strikes me as very important that if symlinks are followed, it should be easy for the novice user to see a dry-run report. That way potential problems would be seen before hours of syncing with no explanation.

2. If symlinks are followed, it also seems to me to be very important to implement an EXCLUDE list. This of course being a list of paths which the sync algorithms completely ignore.

+++ I just had a vision of such a dry-run report window.... It shows an expandable tree of everything to be synchronized. Local files are green, remote files are blue, files that exist both places are black. Symlinks show up clearly different, and each item has the option to ignore it. Ignoring a file or directory turns it and everything beneath it light gray. Drag and drop a file from Konqueror or Nautilus (etc) into this window adds a branch to the tree via creation of a symlink.

Symlink inclusion and selective exclusion both seem very integral to the useful operation UbuntuOne as a whole. And both should be novice friendly. I am very much looking forward to the continued development of UbuntuOne ... hopefully soon it will be feasible for me to start using it. Maybe the selection window will even be similar to my imagined one. :o)

-- Thank you!

[nitride (vancecs)]

The fact that symlinks are not supported needs to be put in the details of the ubuntu one webpage because it is a massive limitation (The first thing I thought when I read about Ubuntu One was symlinking my websites, music, and pictures)

Without symlinks, it is completely worthless to anyone that does not want to maintain two separate directories, or move things around alot.

As a web developer, I do not want to have to copy my live directories to /home/Ubuntu One every time I make a change....

Why not just add a checkbox called "Follow Symlinks"?

Im not moving 40+ gigs of data to the ubuntu one folder, and making all my other programs play nicely. Ubuntu One must play nicely, or I won't "Pay" nicely ;) next month and thats all there is to it.

[molecule-eye (niburu1)]

There are only two conditions under which I could see a lack of symlink support as being tolerable:

1. It's not a paid-for service, so you get what you get;
2. No other similar syncing software offers symlink support.

Unfortunately both of those conditions fail. I don't see how anybody in their right mind would pay for the 50GB plan if they have to move or copy 50GB worth of data over to a specific folder just to sync it.

[Fabio Napodano (napodano)]

I totally agree with molecule-eye, and this should at least be possible in the pay version of the client

[Tobias Bradtke (webwurst)]

I would vote for the possibility to enable every directory in the home-folder to be synched with ubuntu-1. Not via symlinks, but a configuration option where you can check the several directories to be synched on or off. Default would be only the ubuntu-1 directory.

[Jasper Looije (jasper-jamu)]

I agree with Tobias Bradtke. It would solve my need for sym-links

[Stanislav Senotrusov (senotrusov)]

Another option is to jump through symlink only once if that symlinks is placed somewhere in "~/Ubuntu One", excluding references to any "~/Ubuntu One" subdirectory.

If there is a symlink somewhere inside that linked directory it should not be followed, but stored in Ubuntu One cloud as a symlink, so it can be restored as is.

So, it's a filesystem-centric way to configure which directories one want to sync. It may be more natural to someone then GUI with directory list.

[Lucio Torre (lucio.torre)]

please see https://wiki.ubuntu.com/Specs/LucidUbuntuOneSymlinksAndUDFs

[Everthon Valadão (valadao)]

I agree too with Tobias Bradtke: a GUI option to sync folders to ubuntu one will be nice!
IMHO, just like the share folder on nautilus, it should have a sync folder option too :'(

[Everthon Valadão (valadao)]

BTW, hard links works fine! it's sort of a work arround, but not for the average user...