gvfsd-gphoto2 copies files to RAM (unsuitable for really large files)

[Tyrael]

Binary package hint: gvfs

Copy video files from my Canon Digital IXUS 90 IS with 16GB SD Card is a very hard problem.
It uses about 1 or 2 times the file dimension that you are copying.

I have 2GB of memory (1GB RAM + 1 GB SWAP) and I can't backup some videos because they are too big.

Also copying files from the computer to a media device (camera, smartphone etc) is broken for the same reason.

[Sebastien Bacher (seb128)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. Please try to obtain a valgrind log following the instructions at https://wiki.ubuntu.com/Valgrind and attach the file to the bug report. This will greatly help us in tracking down your problem.

[Sebastien Bacher (seb128)]

what ubuntu version do you use?

[Tyrael (marco-crociani)]

Ubuntu 8.10

"Start the program under control of memcheck:

G_SLICE=always-malloc G_DEBUG=gc-friendly valgrind -v --tool=memcheck --leak-check=full --num-callers=40 --log-file=valgrind.log <program> <arguments> "

How can I do this in this particular case?

[Sebastien Bacher (seb128)]

the issue is a known bug in intrepid but should be fixed in jaunty, closing the bug, you can reopen if you still get the issue after upgrade

[Tyrael (marco-crociani)]

Reopened. This bug is still present in Jaunty.

[Sebastien Bacher (seb128)]

could you get a valgrind log?

[Tyrael (marco-crociani)]

"Start the program under control of memcheck:

G_SLICE=always-malloc G_DEBUG=gc-friendly valgrind -v --tool=memcheck --leak-check=full --num-callers=40 --log-file=valgrind.log <program> <arguments> "

How can I do this in this particular case?

[Sebastien Bacher (seb128)]

you can rename gvfsd-photo to gvfsd-photo.binary and have a wrapper calling valgrind on the binary

[Sebastien Bacher (seb128)]

We are closing this bug report as it lacks the information, described in the previous comments, we need to investigate the problem further. However, please reopen it if you can give us the missing information and don't hesitate to submit bug reports in the future.

[j^ (j)]

i still have this issue in jaunty, its more that files are loaded into memory before being written to disk, which does not work for files that can be up to 4GB.

[Steve Kieu (msh-computing)]

Not quite sure why nobody actioned on this, I got problem with this too with Lucid.

[zak k (y2kbugger)]

This still exists. 7 gigs of ram filled and then the swap. What can I do to give you more information on this bug.

[zak k (y2kbugger)]

http://ubuntuforums.org/showthread.php?t=1237122

[Simon Schaak (sschaak)]

Still present in oneiric. At least 6 releases are affected...

[Sebastien Bacher (seb128)]

somebody having the issue should report it to the people writting the code on the GNOME bugzilla then

[Chris Guidry (chris-theguidrys)]

Definitely still a bug in precise. It appears to me that the gphoto/gvfs is copying the files into memory in order to transfer them.

[Lukáš Chmela (lukaschmela)]

I have a Valgrind log using a wrapper for /usr/lib/gvfs/gvfsd-gphoto2 as Sebastien adviced.
However, I have not managed to copy more than a 19MB (RAW photo file), thus I am not sure if it will help. All bigger files (videos) ended up with time out error as the Valgrind memchecks take probably too long (Nautilus seems to require getting a response within 30 seconds after making a copy request).

Test run on Ubuntu 12.10 where the memory problem persists - I have tried coppying a 1.7GB video file using gphoto2 protocol in Nautilus (without Valgrind wrapper) and it begun consuming memory right after entering the directory with the video file (for obtaining a preview) - leaving the directory didn't stop raising memory usage - it raised somewhere to 5GB of RAM (used only by gvfsd-gphoto2) after what I closed the Nautilus window - this unallocated all the 5GB of memory.

After entering the directory again, there was the video file with default video icon showing no video preview.
After pasting the video file into a local directory gvfsd-gphoto2 begun consuming memory again but the Nautilus dialog showed 0 bytes copied. Then after 30 seconds the copy dialog has been closed showing the time out error as with Valgrind (this is new behavior, it didn't happen at least with Oneiric where I was able to copy the file if I had enough memory - so now, I'm NOT ABLE TO COPY A BIG FILE FROM A CAMERA AT ALL). After closing the error window stopping the copy operation, however, gvfsd-gphoto2 continued allocating memory up to the 1.7GB - then all the memory has been freed immediately.

Please ask if you need another information, I hope to get this very old issue finally fixed - for now, I'm marking it confirmed.

Thank you in advance.

[Sebastien Bacher (seb128)]

Thanks for the log, there is no obvious big issue there though ... it would still be useful to report the bug upstream though

[Lukáš Chmela (lukaschmela)]

Upstream bug report: https://bugzilla.gnome.org/show_bug.cgi?id=642814

[e633 (e633)]

Filled 7GB of RAM before i noticed when transferring mp3s to my USB Samsung YP-Q1JCB mp3 player.
Due to the fact that i do not have a swap file the system slowed down to a crawl and some other applications crashed. Managed to save it though. Noticed the player had finished transferring and had been somehow unmounted. I remounted it and started putting files in it again. Watched the damn thing grow to 800MB then to 2.5GB then to 5GB... when i decided i had enough and to unmount & remount the mp3 player.
Logs show nothing unusual.
Definitely affects me.
My specs: http://pastebin.com/f0rjeg9p.
Gonna try valgrind as soon as i can.

[upromis (promisman)]

I have managed to transfer 3.8GB video file from camera, but it first filled whole RAM (4GB) and then some swap, after that the progress bar in nautilus started to work transfering the cashed data. Also in this period disk activity lamp on the camera stop blinking.
So, it first transfers the data to RAM, then it writes it to HDD. Which is bad choice, since the data can be in gigabytes.

This was in Ubuntu 12.04.
But, in Ubuntu 12.10, on a different computer it didn't succeeded even it has 4GB of RAM and swap. It timeouted.

[Yariv Barkan (yarivbarkan)]

I'm trying to transfer large video files from my computer to my Samsung Galaxy S2 smartphone.

As @upromis said, apparently it first tries to copy the file to RAM. Since I only have 4GB RAM, and video files are very often larger (especially high-definition which is very common these days), it is effectively impossible to transfer these files.

[e633 (e633)]

... and this bug is rated "low"...

[gene_wood (gene.wood)]

The one workaround I've found for this is to spin up a Windows XP VM in Virtualbox and pass the USB device through to windows, then copy the files from a Virtualbox shared folder, using windows, onto the device. This bypasses gvfs entirely.

Such a shame that stuff like this sits broken for years making it impossible to recommend that normal users use Linux on the desktop. Unfortunately, fixing this bug is beyond my abilities.

[Paolo Montrasio (paolo-paolomontrasio)]

I just run into this bug transferring a video to a Galaxy Tab 3. Very annoying. I'd rate it High instead of Medium because it effectively prevents us to transfer large files to phones and tablets over USB. Only WiFi or bluetooth are left but they are slower.

A workaround for Yariv, with the Galaxy S2. You're lucky because that phone is one of the last ones that can be mounted as real USB drives and not only over MTP.

Tap on Settings, More Settings, USB utilities, Connect storage to PC. Connect the USB cable now, tap on Turn on USB storage, OK. It's quicker than it looks and you get an external USB drive mounted in /media and visible in Nautilus. You get two drives if you have an SD card in the phone. This is much better than copying over MTP because it directly access the phone file system, it's faster and you can access every file and folder on the phone. MTP doesn't let us look at everything.

[Rolf Leggewie (r0lf)]

Upstream has finally cooked up a patch: https://bugzilla.gnome.org/show_bug.cgi?id=642814#c12 and I think this should DEFINITELY hit trusty before the release

Please apply 86162bbe4b09f517b551ff1c9207a119e91ab733 in trusty and possibly even precise.

[Rolf Leggewie (r0lf)]

https://git.gnome.org/browse/gvfs/commit/?id=86162bbe4b09f517b551ff1c9207a119e91ab733 is where you can get the patch

[Sebastien Bacher (seb128)]

The version is trusty already include that patch, closing the bug

[Gabriel Amador García (gabriel-amador-garcia)]

The bug is not fixed. I'm on xubuntu 14.04 and I'm swapping a lot.

I'm copying all my music library to my cellphone and using the system is a hell.

Proof: http://imgur.com/5hEiCHC

[Markus Demmel (az-zankapfel)]

I'm using ubuntu studio (14.04 / Trusty) with the latest updates and this bug is not fixed here!

@30: How come it is fixed in your Trusty version?

[Rolf Leggewie (r0lf)]

I just copied a 1GB trusty live CD from my local computer to a davfs-mounted webdrive. Previously, this would trigger this bug but I no longer see it now, none of the gvfs-related processes were consuming more than 40MB. I still run trusty, I verified the upstream patch is indeed applied. So, yes, this is fixed. I believe you are seeing something else. I suggest you open a new ticket with similar information as was given here.

[Gabriel Amador García (gabriel-amador-garcia)]

The problem is with MTP
El 06/10/2014 06:15, "Rolf Leggewie" <email address hidden> escribió:

> I just copied a 1GB trusty live CD from my local computer to a davfs-
> mounted webdrive. Previously, this would trigger this bug but I no
> longer see it now, none of the gvfs-related processes were consuming
> more than 40MB. I still run trusty, I verified the upstream patch is
> indeed applied. So, yes, this is fixed. I believe you are seeing
> something else. I suggest you open a new ticket with similar
> information as was given here.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/348522
>
> Title:
> gvfsd-gphoto2 copies files to RAM (unsuitable for really large files)
>
> Status in GVFS:
> Fix Released
> Status in “gvfs” package in Ubuntu:
> Fix Released
>
> Bug description:
> Binary package hint: gvfs
>
> Copy video files from my Canon Digital IXUS 90 IS with 16GB SD Card is a
> very hard problem.
> It uses about 1 or 2 times the file dimension that you are copying.
>
> I have 2GB of memory (1GB RAM + 1 GB SWAP) and I can't backup some
> videos because they are too big.
>
> Also copying files from the computer to a media device (camera,
> smartphone etc) is broken for the same reason.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/gvfs/+bug/348522/+subscriptions
>

[Rolf Leggewie (r0lf)]

On 06.10.2014 14:14, Gabriel Amador García wrote:
> The problem is with MTP

Then you should report that against the appropriate package. This
ticket is about gvfs.

[Craig McQueen (cmcqueen1975)]

I'm seeing this problem today in 14.04, when trying to copy music from Ubuntu 14.04 to an Android phone.

[Craig McQueen (cmcqueen1975)]

The memory usage seems to be proportional to the size of the files copied. When I copied 370 MB of files, according to 'top' the gvfsd-gphoto2 memory usage in RES column at the end was 420 MB. Memory is freed when I unplug USB. So a workaround is to copy only a few 100 MB of files, then unplug USB, then replug and resume.

I'm running Ubuntu 14.04 64-bit. Package gvfs-backends is version 1.20.1-1ubuntu1.