Wrong timestamp on files copied over SSH

[dotancohen]

In Konqueror 3.5.8, when files are copied via ssh from a remote host to local, the file's date is the date that the original file was created on the remote host. However, if the file is then copied from local to a different remote host, the new file's date is today.

[Mi+kde (mi+kde)]

Version:

[Adriaan de Groot (groot)]

Not FreeBSD specific.

[2-nicolasg (2-nicolasg)]

See also bug #58950 (for tar KIO slave)

[2-nicolasg (2-nicolasg)]

See also bug #89377 (for fish KIO slave)

[2-nicolasg (2-nicolasg)]

See also bug #24443 (remove/copy in Konqy, probably a duplicate)

[2-nicolasg (2-nicolasg)]

See also bug #112375 (media KIO slave)

[David Faure (faure)]

#24443 is fixed, i.e. mtime is preserved for local files and directories.
For this bug, let's not forget to preserve mtime with other protocols for both files and dirs (can't be done before KDE4).

[Marco Cimmino (cimmo)]

Any news?

[Stephen on 3rd Rock (sonrock3)]

It's a permissions thing (assumign same problem as I've just solved in Ubuntu Gutsy gnome - I installed ntfs-config then enabled both options).

Good luck
Stephen

[dotancohen (dotancohen)]

As I am coping between two Linux machines, I doubt that ntfs-config would affect this. I should have made that clear. Local machine: Ubuntu 7.04. Remote machine: Red Hat Enterprise Linux.

[X-wstephenson (x-wstephenson)]

Ping. I'm going through the fish:// bugs and http://bugs.kde.org/show_bug.cgi?id=56213#c11 from David references this one. Can we fix this now or is it deferred til KDE5?

[David Faure (faure)]

For all slaves that implement ::copy(), like FTP or FISH, they have to preserve timestamps on their own using something like

            KDE_struct_stat dest_statbuf;
            if (KDE_stat( _dest_orig.data(), &dest_statbuf ) == 0) {
                struct utimbuf utbuf;
                utbuf.actime = dest_statbuf.st_atime; // access time, unchanged
                utbuf.modtime = dt.toTime_t(); // modification time
                utime( _dest_orig.data(), &utbuf );
            }
(as used unconditionally in kio_file, so it compiles on Windows too)

For other slaves it should work already in KDE 4.

[Yuriy Kozlov (yuriy-kozlov)]

Thank you for filing this bug. You are using fish:// to copy the files?

[Yuriy Kozlov (yuriy-kozlov)]

Also see:
http://bugs.kde.org/show_bug.cgi?id=56213
http://bugs.kde.org/show_bug.cgi?id=55804

[dotancohen (dotancohen)]

Yes, I was using the fish:// protocol. I did not realize that it is a KDE issue. Thanks.

[Yuriy Kozlov (yuriy-kozlov)]

Could you test if Dolphin/Konqueror in KDE 4.0 has the desired behavior, or if it is at all different from the current behavior in KDE 3?

[dotancohen (dotancohen)]

I cannot seem to get Kubuntu 8.04 beta installed in VirtualBox. I've tried with Fedora9 alpha and that won't install either. So I'm without KDE4 for the time being. Can someone else confirm this for me?

[Jonathan Jesse (jjesse)]

Good evening,

I am having problems trying to replicate this bug. Are you still having problems with the wrong timestamp on files copied via SSH?

Can you please update the status of this bug?

Thanks,

Jonathan

[dotancohen (dotancohen)]

I am very busy with studies, but in the coming week or two I will retriage this bug and update the bug report. Thanks.

[dotancohen (dotancohen)]

I can confirm that the behaviour of Konqueror in KDE 4.1 in Kubuntu 8.10 is the same as described in the first post of this thread. Files copied via SSH in Konqueror from a remote machine to the local machine retain their original creation dates. Files copied via SSH in Konqueror from the local machine to a remote machine have their creation dates set as "today".

This is a serious issue when working with files such as photographs with no exif information, as the date can be critical in certain applications. This could be considered a dataloss issue in those cases.

[dotancohen (dotancohen)]

Wow, the upstream bug is from early 2003:
https://bugs.kde.org/show_bug.cgi?id=55804

[Kde-2011-08 (kde-2011-08)]

I can confirm that in Konqueror in KDE 4.1, when files are copied via ssh from a remote host to local, the file's date is the date that the original file was created on the remote host. However, if the file is then copied from local to a different remote host, the new file's date is today.

In instances where file creation times are critical this is a dataloss bug, and therefore very severe.

[Kde-2011-08 (kde-2011-08)]

*** Bug 56213 has been marked as a duplicate of this bug. ***

[Thorsten Staerk (dev-staerk)]

I agree that this is severe. If I MOVE (not copy) files from a fish:// location to local, the time data gets lost as well.

[Kde-2011-08 (kde-2011-08)]

I just tested Konquerer 3.5.10, Dolphin 1.2 (KDE 4.2), and the system's cp/mv commands to determine behaviour under different copy/move circumstances (local->local, local->remote, remote->local). I found that Dolphin and Konqueror both behaved identically. However, neither of them behaved consistently from circumstance to circumstance, and none of them were consistent with the system's cp/mv behaviour!

I made a monospace table here but it looks terrible, so please see the HTML table at this page to see the results of my test:
http://dotancohen.com/eng/bugs_kde_cut_copy_paste.html

As there is no consistency between Konqi/Dolphin's own behaviour, nor between the behaviour of the cp/mv commands, I feel that this bug should be addressed quickly. File timestamps mean nothing on KDE systems so long as this bug is open.

[David Faure (faure)]

The goal of KIO is to emulate cp -a, not cp.
Thanks for the rest of the analysis -- but please be more precise when you say "remote". The implementations of fish, ftp, sftp, and smb are all different and might all need fixing for this (well assuming that they support changing a file's timestamp, which IIRC at least FTP doesn't support).

[Kde-2011-08 (kde-2011-08)]

Thank you. By "remote" I mean SSH via Konqueror's fish:// pseudoprotocol.

When I have a chance I will test the other protocols and update the table. I will then report back here. If there is anything else related that you would like me to triage, just ask.

[Thorsten Staerk (dev-staerk)]

When I move
fish://root@localhost/blah
to
/tmp/blah
the timestamp gets changed.

When I move
fish://root@localhost/blah
to
fish://root@localhost/tmp/blah
the timestamp does not get changed.

This indicates that the file:// kioslave is the instance that forgets to set the timestamp in case the file comes from a fish:// location.

[Thorsten Staerk (dev-staerk)]

There is a comment in the code:

// TODO set modification time on url.path() somehow
// see FileProtocol::put if using utime() to do that.

see kdebase/runtime/kioslave/fish/fish.cpp

[Harald Sitter (apachelogger)]

Is this supposed to work with sftp://?

http://www.derkeiler.com/Mailing-Lists/securityfocus/Secure_Shell/2003-08/0069.html
seems to indicated that sftp is capable of updating the timestamps, still KDE's kioslave seems to set them to today().

Could someone please comment on the KDE bug :)

I am lowering the importance, medium is clearly too high.

[dotancohen (dotancohen)]

I think that the fish:// protocol is in fact sftp://. The sftp put command can have the -P flag used to preserve timestamps and permissions. I will go through the three KDE bugs posted here and comment where needed.

> I am lowering the importance, medium
> is clearly too high.

This is a dataloss issue, in my opinion medium is not high enough. I have voice recordings of my daughter as a baby, and because they were moved with KDE I have lost the dates of those recordings. Some of them are very sensitive to her age, and not knowing her age at the time of recording makes the recording worthless. Any kid will tell you that water is made of hydrogen and oxygen. Hearing a 16 month old say that is precious. Now, I don't remember and don't have the date to be sure any more.

Please consider moving the priority back up. Dataloss is serious.

[Kde-2011-08 (kde-2011-08)]

The sftp put command can have the -P flag used to preserve timestamps and permissions. Furthermore, bug 56213 states that the command MDTM (or MTDM?) is supposed to preserve timestamps as well.

I am updating the title of this bug to better describe the issue. Furthermore, I am changing this from wishlist to bug, because that is what this is.

For those who need convincing that this is a bug, read on. Others stop here.

1) Dataloss. I have voice recordings of my daughter as a baby, and because they were moved with KDE I have lost the dates of those recordings. Some of them are very sensitive to her age, and not knowing her age at the time of recording makes the recording worthless. Any kid will tell you that water is made of hydrogen and oxygen. Hearing a 16 month old say that is precious. Now, I don't remember and don't have the date to be sure any more.

2) See comment #17 and comment #18

[Kde-2011-08 (kde-2011-08)]

KDE 4.3.2: Both Konqueror and Dolphin set the timestamp as NOW for files copied from remote to local over sftp:// protocol.

[Kde-2011-08 (kde-2011-08)]

Bug 169326 is a possible dupe, with regard to Bluetooth.

[Frank78ac (frank78ac)]

*** Bug 218133 has been marked as a duplicate of this bug. ***

[David Faure (faure)]

Copying from file to sftp preserves the modification time for me.
libssh-0.4.0, kde 4.4.0.

The code that calls sftp_utimes in kio_sftp::put() comes from r1010049, i.e. from the initial rewrite of kio_sftp to use libssh in August 2009.
Ah, that's after 4.3.0 was released, so 4.4 is indeed the first release with this new kio_sftp.

-> FIXED for kio_sftp.

No idea how to fix for kio_fish, I don't know the protocol.

[Kde-2011-08 (kde-2011-08)]

On my 4.3.5 machine, all operations via SFTP result in NOW timestamp for both Modified and Accessed time:

* Copy/Paste
- local->remote: Modified NOW Accessed NOW
- remote->local: Modified NOW Accessed NOW

*Cut/Paste
- local->remote: Modified NOW Accessed NOW
- remote->local: Modified NOW Accessed NOW

I will fire up a 4.4 prerelease version and test there.

[Kde-2011-08 (kde-2011-08)]

Tested on Kubuntu 9.10 with KDE SC 4.4 RC2, libssh-4 0.4.0, connected via SFTP in Dolphin and using the mouse:

* Copy/Paste
- local->remote: Modified THEN Accessed NOW
- remote->local: Modified THEN Accessed NOW

* Cut/Paste
- local->remote: Modified NOW Accessed NOW
- remote->local: Modified NOW Accessed NOW

That is the exact _opposite_ of what one would expect to find. Copy/Paste should be NOW as the user is creating a new file. Cut/Paste should be THEN as the user is merely moving a file form one place to another. In fact, the Copy/Paste operation should be optionally THEN as I can think of quite a few use cases for that.

[David Faure (faure)]

I agree, moving a file should preserve its mtime, I'll take a look at that. Surprising...

But I disagree that copying a file should change its mtime; this is what this whole report is about, preserving the mtime on copy ;-)
You didn't create a new file from scratch or modify a file, this is a copy of a file that was modified long ago. Well you seem to hint at that too, but I don't see a use case for making it optional. The goal is to always preserve the mtime.

[Kde-2011-08 (kde-2011-08)]

> But I disagree that copying a file should change its mtime; this is what
> this whole report is about, preserving the mtime on copy ;-)

No, the report is that the mtime is wrong. As per the common unix tool cp, mtime is NOW. That is in contrst to mv, which uses THEN.

However, I have no problem with the mtime on copied files being THEN. The critical issue is that the mtime on a moved file should never be NOW. If you make both move and copy operations THEN, then I agree that this long-standing dataloss bug can finally be closed! Thanks!

[Mi+kde (mi+kde)]

As the original reporter, I'd like to opine, that the mtime should always be preserved -- whether the operation is move (rename), or copy (the equivalent of `cp -p').

There are other timestamps maintained by the filesystem, from which the time of copying (or moving) can be discerned, if needed. But, if the two files are identical (one is the copy of the other), the timestamps on them should be identical too -- copying does not modify the file, even if `cp' (without the `-p' flag) thinks, that it does...

BTW, on certain filesystems, such as FreeBSD's default FFS, there are also flags (such as append-only, no-dump, undeletable, etc.), which should be preserved too, whenever possible.

[David Faure (faure)]

Dotan: KDE emulates cp -a, not cp, by choice.

Back to the bug: I confirm that kde-mv loses the mtime. Testcase for fast bug reproduction:

touch -r /boot /tmp/myfile ; kde-mv /tmp/myfile sftp://localhost/$HOME/tmp ; ll ~/tmp/myfile

Debugging now.

[David Faure (faure)]

SVN commit 1084346 by dfaure:

Preserve modification time when moving files [and when the kioslave's put() supports it, like kio_sftp, but unlike kio_fish]
This was already working for copying, but not for moving (across protocols).
Fixed for: 4.4.0, hopefully
CCBUG: 55804

 M +4 -0 copyjob.cpp

WebSVN link: http://websvn.kde.org/?view=rev&revision=1084346

[David Faure (faure)]

This commit also fixed kde-mv from fish to local.

Only kde-mv/kde-cp from local to fish is still broken, due to the not implemented use of "modificationtime" in fish's put(). Reassigning.

[Kde-2011-08 (kde-2011-08)]

Thank you David! This bug was a showstopper for me, I cannot wait to get this on production systems. Thanks!

[Kde-2011-08 (kde-2011-08)]

Has there been any progress for local to fish? I don't mean to push, but I would like to see the issue resolved :)

Thanks!

[Kde-2011-08 (kde-2011-08)]

I'm not sure if this is a new issue or not, so I'll ask:
In KDE 4.5.1 it turns out that copying (copy-paste) from remote to local via FTP in Dolphin gives a timestamp of NOW. Is that this same issue, or should I file another issue?

Thanks.

[D-nl-k (d-nl-k)]

How comes that a 9 year old bug is not fixed?
Are the kiddies too busy programming wobbling windows and other nonsense stuff?
It's a shame.
Solution suggest: If ther's no one smart enough to program a working sftp in the KDE team why not just utilise the working scp that comes with any Linux distribution? No need to re invent the wheel.

[Christoph-maxiom (christoph-maxiom)]

Paul, that sounds like a suggestion you could try. Please add the patch to reviewboard.

[David Faure (faure)]

Hmm? This bug is already fixed for kio_sftp, see comment #30.

[Kde-2011-08 (kde-2011-08)]

Hi David. This bug is not resolved for uploading local files to remote servers via the fish:// pseudo-protocol, see comment 33. This actually just bit me this week and got someone very mad at me! Paul, do you also go by another name and live in Beersheba? :)

As for comment 34 (remote to local via FTP in Dolphin) I'll have to triage a bit later when I have access to an FTP server. I'll also test via SSH to be complete.

[D-nl-k (d-nl-k)]

As far as I can say neither upload nor download is fixed (tried sftp with konqueror and krusader under Kubuntu precise). It's a real showstopper as no directory sync is possible that way. One can copy back and forth 'till eternity.
Christoph, if I was able to provide a patch I would have done years before. Unfortunately I'm not in Linux development, just some perl.
...and no, Dotan. I don't even know what Beersheba is.

[Mathieu d'Acremont (mathieu-dacremont)]

Synchronization over ssh with krusader was working fine under Fedora 17. Timestamps were preserved. The bug has affected me since I switched to Ubuntu 12.04. So it might be specific to Ubuntu.

[Harald Sitter (apachelogger)]

Closing in favor of upstream report. This ought to be fixed in kdelibs upstream, nothing we can do about it on Kubuntu level.

[dotancohen (dotancohen)]

Harald, if you look at the comment preceding yours, you will see that the bug is Ubuntu-specific. Users of other distros, such as Fedora, do not suffer the issue.

[Harald Sitter (apachelogger)]

Except there is an upstream bug that confirms it not to be specific to any distro.

[Alpha0x89 (alpha0x89)]

Time stamps used to be preserved for me with KDE 4.6 (openSUSE 11.4) when copying files with Krusader over SFTP from or to a local directory, except when using the Synchronize Directories Tool. After upgrading to KDE 4.8 the time stamps are no longer preserved even when uploading over SFTP from a panel.

(In about two months we can celebrate the tenth birthday of this bug report.)

[D-nl-k (d-nl-k)]

(In reply to comment #40)
> (In about two months we can celebrate the tenth birthday of this bug report.)

Hey, great! 10th anniversary.
Maybe we sould take the chance to rename KDE into BUG.
This is not the only bug that hasn't been fixed for years. (I only say Amarok)

[DA (adawit)]

(In reply to comment #41)
> (In reply to comment #40)
> > (In about two months we can celebrate the tenth birthday of this bug report.)
>
> Hey, great! 10th anniversary.
> Maybe we sould take the chance to rename KDE into BUG.
> This is not the only bug that hasn't been fixed for years. (I only say
> Amarok)

If you do not have anything constructive to say, please keep it to yourself. Adding noise in a bug report will not get it fixed! In case you are not aware, this is a volunteer project.

[DA (adawit)]

I will reiterate what David stated already. Each of the kioslaves are their own independent modules maintained by different people. So if there is a problem in any of them it would have to be fixed in that protocol's ioslave itself.

As such if you find issues with a specific protocol, it would be wise to open a bug report against that specific protocol's ioslave, so the issue might get handled in a timely fashion. Please note that most developers are only subscribed to get bug reports against things they maintain ; so a generic bug report like this one might never reach them at all.

Anyhow, if the kio_sftp ioslave is broken again, I am to blame since I added an optimization that made it a bit faster to copy files so long as either the source or the destination is a local file. This was done for the KDE v4.8 release. I will check and fix that.

I will also see what can be done about the ftp ioslave.

[DA (adawit)]

Git commit a3b8bc5ec9ecffa45b2d64789a4d571b7af79cc7 by Dawit Alemayehu.
Committed on 10/06/2013 at 21:51.
Pushed by adawit into branch 'KDE/4.10'.

Set the "modified" (aka mtime) property for direct move jobs, i.e. where the
ioslave has implemented copyFromFile/copyToFile.
REVIEW: 110944

M +6 -0 kio/kio/job.cpp

http://commits.kde.org/kdelibs/a3b8bc5ec9ecffa45b2d64789a4d571b7af79cc7

[DA (adawit)]

Git commit f0187df8cbcca0f613a23431018255813b129ad1 by Dawit Alemayehu.
Committed on 10/06/2013 at 23:41.
Pushed by adawit into branch 'KDE/4.10'.

Set the proper mtime when moving a file from sftp -> file.
REVIEW: 110947

M +10 -0 kioslave/sftp/kio_sftp.cpp

http://commits.kde.org/kde-runtime/f0187df8cbcca0f613a23431018255813b129ad1

[3-asn (3-asn)]

*** Bug 321010 has been marked as a duplicate of this bug. ***

[Kde-2011-08 (kde-2011-08)]

Thank you for looking into this, Dawit!

[DA (adawit)]

(In reply to comment #47)
> Thank you for looking into this, Dawit!

No problem, but please note that my fix is only going to address kio_sftp and kio_ftp. And even then for kio_ftp, I can only address the issue for moving files from the ftp server to the local file system. To see why I was not able to do the same for the other case, checkout http://git.reviewboard.kde.org/r/111016/.

After the code listed above is committed, the timestamp set on copied/moved files through

kio_file will be:
============
* Copy/Paste
- local->remote: Modified THEN Accessed NOW
- remote->local: Modified THEN Accessed NOW

*Cut/Paste
- local->remote: Modified THEN Accessed NOW
- remote->local: Modified THEN Accessed NOW

kio_sftp will be:
=============
* Copy/Paste
- local->remote: Modified NOW Accessed NOW
- remote->local: Modified NOW Accessed NOW

*Cut/Paste
- local->remote: Modified THEN Accessed NOW
- remote->local: Modified THEN Accessed NOW

kio_ftp will be:
=============
* Copy/Paste
- local->remote: Modified NOW Accessed NOW
- remote->local: Modified NOW Accessed NOW

*Cut/Paste
- local->remote: Modified NOW Accessed NOW [ requires new implementation ]
- remote->local: Modified THEN Accessed NOW

[DA (adawit)]

Git commit 4a75f8b6dbc36a858b5a05e4f65d39d07b0dc98c by Dawit Alemayehu.
Committed on 14/06/2013 at 06:22.
Pushed by adawit into branch 'KDE/4.10'.

Set the modified timestamp on ftp -> file moves.
REVIEW: 111016

M +15 -0 kioslave/ftp/ftp.cpp

http://commits.kde.org/kdelibs/4a75f8b6dbc36a858b5a05e4f65d39d07b0dc98c

[DA (adawit)]

Git commit ef3de4a99c3af0557f51afe61e30addb592f2d4a by Dawit Alemayehu.
Committed on 04/09/2013 at 05:00.
Pushed by adawit into branch 'KDE/4.11'.

Set modified meta-data for all copy jobs, not just move operations.
REVIEW: 112528

M +2 -2 kio/kio/job.cpp

http://commits.kde.org/kdelibs/ef3de4a99c3af0557f51afe61e30addb592f2d4a

[Michael Scholz (mailto-mikes)]

For me it seems the bug ist still open ....because I have this in Plasma 5. It's a really serious bug for digikam users after realizing that copied files from SD card loose their create-date info..; :-(

[Vadims-zenins (vadims-zenins)]

Ubuntu 14.04.4 LTS
Qt: 4.8.6
KDE: 4.14.13

Krusader
Version 2.4.0-beta3 "Single Step"
Using KDE Development Platform 4.14.13

When I synchronize directories to smb timestamp changed to new one.
Bug is still open.

[N-kde (n-kde)]

Happy 19th anniversary.

Arch Linux
KDE 5.24.3
Qt 5.15.3 & 6.2.3
kio 5.91.0
kio-extras 21.12.3
krusader 2.7.2 & 2.8.0.dev