Application freezes under Ubuntu 12.10, Fedora 18 and other recent releases

[Chris Giltnane]

On Quantal 32bit. version The application runs fine right up until you hit the download button. It says file 1 or if lucky 2 of XXX photos or videos and then dims out not responding to anything other than a quit from the launch icon. However to restart you need to then kill the process using system monitor.
Version 0.4.5-1 installed.

[Damon Lynch (dlynch3)]

Thanks for your bug report. Please run this from the command line:

rapid-photo-downloader --debug

and copy and paste the results of the terminal output. Also provide
information such as what device you are downloading from and where to.

[Chris Giltnane (chris-giltnane)]

Downloading was from a directory on hard drive to a different directory. I also use renaming if that makes a difference and no thumbnail creation

[INFO/MainProcess] Rapid Photo Downloader 0.4.5
[INFO/MainProcess] Using pyexiv2 0.3.2
[INFO/MainProcess] Using exiv2 0.23.0
[INFO/MainProcess] Using exiftool 8.60
[INFO/MainProcess] Using hachoir 1.3.3
[INFO/MainProcess] No downloads have occurred so far today
[INFO/MainProcess] allocating a new mmap of length 4096
[DEBUG/MainProcess] created semlock with handle 3023564800
[DEBUG/MainProcess] created semlock with handle 3023560704
[DEBUG/MainProcess] created semlock with handle 3023556608
[DEBUG/MainProcess] created semlock with handle 3023552512
[DEBUG/MainProcess] created semlock with handle 3023548416
[DEBUG/MainProcess] created semlock with handle 3023544320
[DEBUG/MainProcess] created semlock with handle 3023540224
[DEBUG/MainProcess] created semlock with handle 3023536128
[DEBUG/MainProcess] _set_from_toolbar_state
[INFO/GetPreviewImage-1] child process calling self.run()
[DEBUG/MainProcess] Start of day is set to 3:0
[DEBUG/MainProcess] SubfolderFile PID: 5942
[INFO/MainProcess] Using manually specified path /home/chris/Photoimport
[INFO/MainProcess] # photo backup devices: 0; # video backup devices: 0
[INFO/SubfolderFile-2] child process calling self.run()
[DEBUG/SubfolderFile-2] Finished 0. Getting next task.
[DEBUG/MainProcess] created semlock with handle 2909835264
[DEBUG/MainProcess] created semlock with handle 2909831168
[DEBUG/MainProcess] created semlock with handle 2909827072
[DEBUG/MainProcess] Queue._after_fork()
[DEBUG/MainProcess] created semlock with handle 2909822976
[DEBUG/MainProcess] created semlock with handle 2909818880
[DEBUG/MainProcess] created semlock with handle 2909814784
[DEBUG/MainProcess] created semlock with handle 2909810688
[DEBUG/MainProcess] created semlock with handle 2909806592
[DEBUG/Scan-3] Queue._after_fork()
[INFO/Scan-3] child process calling self.run()
[DEBUG/MainProcess] TaskManager PID: 5943
[DEBUG/Scan-3] Scanned 100 files
[DEBUG/MainProcess] on_from_filechooser_button_selection_changed
[DEBUG/Scan-3] Scanned 200 files
[DEBUG/MainProcess] on_from_filechooser_button_selection_changed
[DEBUG/Scan-3] Scanned 300 files
[INFO/Scan-3] process shutting down
[DEBUG/Scan-3] running all "atexit" finalizers with priority >= 0
[DEBUG/Scan-3] running the remaining "atexit" finalizers
[INFO/Scan-3] process exiting with exitcode 0
[INFO/MainProcess] Found 321 photos and videos
[INFO/MainProcess] Files total 640.4MB
[DEBUG/MainProcess] Turning on display of thumbnails
Segmentation fault (core dumped)

[Damon Lynch (dlynch3)]

A core dump indicates a bug in PIL (python imaging) or pygtk itself.
These things can happen when a distro adopts a newer version of a
compiler. It does not indicate a bug in the code for Rapid Photo
Downloader.

[Chris Giltnane (chris-giltnane)]

Thanks for that Damon, what can I do to move this forward. Should I raise a bug against one of the packages you mentioned?

[Damon Lynch (dlynch3)]

I am in Central Asia right now and I'm consequently not in a position to
figure out what the precise cause of the bug is. If you know a bit about
python, you can try to look for it yourself you're feeling brave. You
might find any bug you might report will be a duplicate of an already
reported bug, however!

[kylehase (kyle-hasegawa)]

For the record, the problem is also reproduceable on precise pangolin 12.04. RPD becomes unresponsive once the download button is pressed.

[Damon Lynch (dlynch3)]

kyle-hasegawa you need to file a new bug report, including the output of this when run from the command line:

rapid-photo-downloader --debug

[Damon Lynch (dlynch3)]

Chris, do you still see the bug? If so, which file type are you downloading?

It works for me with a large variety of different file types on 12.10 beta 1.

[Chris Giltnane (chris-giltnane)]

Hi Damon I just tried it forthe first time for weeks and it worked from my HTC phone ok, very slow but it worked . I use jpg's and Samsung dng's mostly.
I will give it some serious exercise and let you know.
32 bit 12.10 beta1.
 I had reduced my import options as well, so will ramp them back up and see what happens.

[Chris Giltnane (chris-giltnane)]

Hi Damon. running in debug it process 1 file every 28 seconds or so.

[Chris Giltnane (chris-giltnane)]

To recreate freeze I had the following start position
multiple source files need to be in at least 2 sub directories, it seems to cope if there are only a few files to import but anything in excess of 100 files breaks everytime. I also had some files that had been previously imported, the application seems to slow down considerably if you have these files present. Downloading froma single source directory works everytime although very slowly.

[Damon Lynch (dlynch3)]

Are you downloading directly from a camera or are you taking the memory card and inserting it into a memory card reader? When downloading directly from a camera or a phone, Rapid Photo Downloader uses gnome system libraries. The gnome developers do the best but they have bugs. A possible workaround is to use Nautilus to copy the files onto your desktop, and then use Rapid Photo Downloader to import and rename them as you prefer.

[Chris Giltnane (chris-giltnane)]

Hi Damon I have tried both of these options, direct from the camera and
then copying into a directory on the hard drive. Seems to make no
difference. The trigger seems to be multiple input directories. So for
instance if the import directory is /dcim then if I have /dcim/1 and
dcim/2 then it seems to get to the end of the first directory and then
freeze and never get to the next directory.

There is a apport bug on launchpad dont know if that will help

https://bugs.launchpad.net/bugs/1048796

On 11/09/12 19:36, Damon Lynch wrote:
> Are you downloading directly from a camera or are you taking the memory
> card and inserting it into a memory card reader? When downloading
> directly from a camera or a phone, Rapid Photo Downloader uses gnome
> system libraries. The gnome developers do the best but they have bugs.
> A possible workaround is to use Nautilus to copy the files onto your
> desktop, and then use Rapid Photo Downloader to import and rename them
> as you prefer.
>

--
Chris Giltnane

[Damon Lynch (dlynch3)]

Is it possible for you to share a copy of the exact files using the
directory structure you describe? If so, and the files are large, a service
like dropbox may help.

[Chris Giltnane (chris-giltnane)]

Don't have dropbox but do have a ubuntu one account that we could use if
you have that. If not I may be able to give you temporary ftp access to
my NAS to get the files. I will zip up a sample and get something going.
On 11/09/12 20:52, Damon Lynch wrote:
> Is it possible for you to share a copy of the exact files using the
> directory structure you describe? If so, and the files are large, a service
> like dropbox may help.
>

--
Chris Giltnane

[Damon Lynch (dlynch3)]

I also have Ubuntu One. I've not used it for file sharing but I should be
able to figure it out ;-)

[Damon Lynch (dlynch3)]

After some time I was finally able to take a look at the test files you provided. I was unable to replicate the crash.

[Chris Giltnane (chris-giltnane)]

Thanks for looking at this. I still have the problem but next week I am
moving to a new machine and 64bit will see if i still get the issue
after that.
On 25/09/12 05:08, Damon Lynch wrote:
> After some time I was finally able to take a look at the test files you
> provided. I was unable to replicate the crash.
>

--
Chris Giltnane

[Olivier L (ceituna)]

Hello,

Using a 64bit environment, I have the same problem.

When using Pangolin (12.04), everything was ok.

After moving to Quantal in order to have a newer Digikam, RPD became useless...
The time needed to download pictures exploded.

Worst case : having multiple folders created with many dates : seems like whole process repeats again.
Main window doesn't react, even if debug terminal shows action.

Final download speed seems approaching 500 kb/s while previous use with Pangolin, we were something like 20MB/s.

[Damon Lynch (dlynch3)]

Hello Olivier,

You may need to open a new bug report. If you do, You need to provide information such as from what device you are downloading, the exact problem, output from the terminal using the --debug command line option, and that type of files you are downloading/ importing. Without this kind of information, it's impossible to diagnose any problem.

Also please note that Ubuntu 12.10 is still in Beta. Experience shows that there are bound to be problems with Python, Python modules, and the system libraries, causing crashes that are of a general nature and not specific to Rapid Photo Downloader.

[Chris Giltnane (chris-giltnane)]

You may well be right that it is an external issue in python, I just has
the same or at least very similar issue importing to shotwell using its
native importer. Using a similar target structure of Year/Month/Day. As in
RPD it seems related to the number of sub-directories under the top level
source along with multiple target directories.

On 29 September 2012 17:45, Damon Lynch <email address hidden> wrote:

> Hello Olivier,
>
> You may need to open a new bug report. If you do, You need to provide
> information such as from what device you are downloading, the exact
> problem, output from the terminal using the --debug command line
> option, and that type of files you are downloading/ importing. Without
> this kind of information, it's impossible to diagnose any problem.
>
> Also please note that Ubuntu 12.10 is still in Beta. Experience shows
> that there are bound to be problems with Python, Python modules, and the
> system libraries, causing crashes that are of a general nature and not
> specific to Rapid Photo Downloader.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1025908
>
> Title:
> Application seems to freeze after first few photo's
>
> Status in Rapid Photo Downloader:
> New
>
> Bug description:
> On Quantal 32bit. version The application runs fine right up until you
> hit the download button. It says file 1 or if lucky 2 of XXX photos or
> videos and then dims out not responding to anything other than a quit from
> the launch icon. However to restart you need to then kill the process using
> system monitor.
> Version 0.4.5-1 installed.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/rapid/+bug/1025908/+subscriptions
>

[Chris Giltnane (chris-giltnane)]

I installed 12.04 the problem went away.
Installed 12.10 64Bit, I was on 32bit before and the problem returned so it looks like a problem in an external dependency.

[Damon Lynch (dlynch3)]

Whatever the cause of the problem, it's a real problem, apparently making
Rapid Photo Downloader unusable on 12.10. Unfortunately for the next three
weeks I'm racing to meet some academic deadlines and am therefore unable to
diagnose the root cause of the problem. I hope to be able to take a
serious look at in November.

[m1chel (hu6hzq0zv8rd-7fo7ak9-t16qijz8x59b)]

i can confirm this bug. Since i upgraded to ubuntu 12.10 (pangolin), the r-p-d wont work anymore.

[Bob Leers (bobslee)]

Hi Damon,
Do you have any plans to work on this the upcoming weeks?

[Damon Lynch (dlynch3)]

Bob are you offering to help?

[Bob Leers (bobslee)]

i'm willingly, but don't have lotta spare time.

wat could i do?
reading debug log and check code calling dependencies? right now i dunno.

Damon Lynch <email address hidden> schreef:

>Bob are you offering to help?
>
>--
>You received this bug notification because you are subscribed to the
>bug
>report.
>https://bugs.launchpad.net/bugs/1025908
>
>Title:
> Application freezes under Ubuntu 12.10
>
>Status in Rapid Photo Downloader:
> Confirmed
>
>Bug description:
>On Quantal 32bit. version The application runs fine right up until you
>hit the download button. It says file 1 or if lucky 2 of XXX photos or
>videos and then dims out not responding to anything other than a quit
>from the launch icon. However to restart you need to then kill the
>process using system monitor.
> Version 0.4.5-1 installed.
>
>To manage notifications about this bug go to:
>https://bugs.launchpad.net/rapid/+bug/1025908/+subscriptions

--
groeten, Bob

[Damon Lynch (dlynch3)]

If you know python, the biggest help right now would be to profile the
application to identify what is causing it to run so slowly. Once the cause
is known, fixing it will hopefully not be difficult. If you don't know
python however, then it's a task best left to someone who does.

[Bob Leers (bobslee)]

I know a bit of Python.
So I profiled the whole program (func call): rapid.start(), with cProfile.
Right I'm not able to interpret these results - it's almost midnight GMT+2 here.
I hope I have some free time next days. Then I'll let you know.

[Bob Leers (bobslee)]

ofcourse one should profile other parts/funcs instead of starting the program..
i need some sleep.
--
groeten, Bob

[Bob Leers (bobslee)]

I spend several hours in profiling (cProfile and runsnakerun), but I couldn't find any clues.
Sorry, I give up.. due to my inexperience and not enough spare time.

I hope you could work on it sometime.
Thanx.

[Damon Lynch (dlynch3)]

Thanks for your efforts Bob! Debugging a multi-process program with code
that runs in parallel is not easy.

[Damon Lynch (dlynch3)]

I have identified the line of code that is causing the bug. It is in the file subfolder.py and is this line:

temp_file.move(download_file, self.progress_callback_no_update, cancellable=None)

That's a call to Gnome's libraries to rename / move a file. For some reason each time it runs under Ubuntu 12.10 it is taking an extremely long time to complete.

Next: figure out what to do about it!

[Bob Leers (bobslee)]

@Damon
Do you have the time to fix it? What can we expect?
Thanks for your efforts anyway.

[Trond Andresen (trond-andresen)]

I believe the error is caused by something in Gnome's library. I am unable to copy files from my eos550d using Nautilus, RPD and Shotwell. Copying from eos350d and olympus TG-810 is however working.

Copying from eos550 worked untill I upgraded fraom 12.04 to 12.10.

[Arjan (arjan-scherpenisse)]

That's strange, copying the photos from the DCIM folders with Nautilus works fine for me..

actually that is how I work around it now that rapid is broken... :-/

[Gonçalo Marrafa (goncalo-marrafa)]

I use nautilus as a workaround too and i it works fine for me too.

[Damon Lynch (dlynch3)]

@Trond the bug you found is unrelated to the current problem.

You can try a preliminary and tentative fix for Rapid Photo Downloader under Ubuntu 12.10 in this PPA:
https://launchpad.net/~dlynch3/+archive/rapid-photo-downloader-testing

Let me know if it improves anything. I don't anticipate having any more time to work on it for another 3 or 4 weeks.

[Yopland (yopland)]

This version seems to work fine on LinuxMint 14

[Henrik Toth (realthk)]

The slow-down is gone on Ubuntu 12.10 with this 0.4.6 beta 1, but there is a new bug: now the thumbnail preview duplicates every image.

[Brent McLaren (bmclaren99)]

The duplicate issue was happening to me (with the slow response) . Haven't tried the beta to see if it fixed the slowness, but when I got duplicates, I would see two entries at the top of the screen where it detected the card that was inserted. This didn't happen every time though.

[Damon Lynch (dlynch3)]

The card was inserted into a memory card reader, or some other kind of
device?

On Mon, Nov 26, 2012 at 8:29 PM, Brent McLaren
<email address hidden>wrote:

> The duplicate issue was happening to me (with the slow response) .
> Haven't tried the beta to see if it fixed the slowness, but when I got
> duplicates, I would see two entries at the top of the screen where it
> detected the card that was inserted. This didn't happen every time
> though.
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1025908
>
> Title:
> Application freezes under Ubuntu 12.10
>
> Status in Rapid Photo Downloader:
> Confirmed
>
> Bug description:
> On Quantal 32bit. version The application runs fine right up until you
> hit the download button. It says file 1 or if lucky 2 of XXX photos or
> videos and then dims out not responding to anything other than a quit from
> the launch icon. However to restart you need to then kill the process using
> system monitor.
> Version 0.4.5-1 installed.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/rapid/+bug/1025908/+subscriptions
>

--
http://www.damonlynch.net

[Brent McLaren (bmclaren99)]

Yes, card was installed in a reader. Just watched what it was doing, and I am seeing the first line show up when it auto-launches, and the second entry shows up shortly after. I can send a screen shot if that works.

[Damon Lynch (dlynch3)]

Please open a new bug to report this new issue. It sounds pretty serious.

On Mon, Nov 26, 2012 at 9:12 PM, Brent McLaren
<email address hidden>wrote:

> Yes, card was installed in a reader. Just watched what it was doing,
> and I am seeing the first line show up when it auto-launches, and the
> second entry shows up shortly after. I can send a screen shot if that
> works.
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1025908
>
> Title:
> Application freezes under Ubuntu 12.10
>
> Status in Rapid Photo Downloader:
> Confirmed
>
> Bug description:
> On Quantal 32bit. version The application runs fine right up until you
> hit the download button. It says file 1 or if lucky 2 of XXX photos or
> videos and then dims out not responding to anything other than a quit from
> the launch icon. However to restart you need to then kill the process using
> system monitor.
> Version 0.4.5-1 installed.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/rapid/+bug/1025908/+subscriptions
>

--
http://www.damonlynch.net

[Ulisse (ulipo)]

The duplication issue happens also to me, and I also use a card reader; hitting "refresh", only one of the card entries remains.
Waiting for the new bug to join the party there.

[Ulisse (ulipo)]

Ok, just tried the beta and I can confirm the "slow download" issue is gone.
Duplicated sources is still there, I opened the relative bug: #1083756

[Bob Leers (bobslee)]

Confirm.
Tested 0.4.6 beta1, with import of +/- 700 pictures. It succeeds with acceptable speed.
Thanx for your efforts Damon!

[m1chel (hu6hzq0zv8rd-7fo7ak9-t16qijz8x59b)]

works very well.... thanks Damon!

[mabunix (majobu)]

Same problem as described here on Archlinux 64 Bit. Is there a tarball available to test the beta?

[sbolten (stephan-bolten)]

Updated to 0.4.6 beta1 - download working now but still very slow compared to a download of the exact same pics to the exact same target structure on Ubuntu 12.04

[Gonçalo Marrafa (goncalo-marrafa)]

Where can i get 0.4.6 b1? It isn't on the PPA.

[Damon Lynch (dlynch3)]

https://launchpad.net/~dlynch3/+archive/rapid-photo-downloader-testing

[Damon Lynch (dlynch3)]

I have now released 0.4.6 beta 1 as a tarball too:
https://launchpad.net/rapid/trunk/0.4.6beta1/+download/rapid-photo-downloader-0.4.6%7Eb1.tar.gz

[Damon Lynch (dlynch3)]

sbolten I need more details. If you can share with me the exact files you
are trying to download, and the exact file renaming and subfolder creation
scheme you are using, that will be most helpful.

[mabunix (majobu)]

Damon, thanks for the tarball. I just gave it a try and I can confirm the duplication bug. Downloading from SD card works faster again compared to 0.4.5. I tried it with JPEGs and RAWs from a Nikon D600. System used: ArchLinux 64 Bit.

[sbolten (stephan-bolten)]

Damon, I have no clue why - but after a couple of reboots and system updates I have now a performing installation of 0.4.6 beta 1... thx for your support

[Olivier L (ceituna)]

Damon,

Thank you for the fix. Seems to solve problem commented on #19

[Miloš Popović (gpopac)]

This also solves the same Bug in Gentoo.

[Damon Lynch (dlynch3)]

Unfortunately it is still not fixed. This is an overview the current
situation: I first diagnosed the bug to a problem making calls in a child
daemon process that renames files. Each file rename would take
approximately 60 seconds to complete when using Gnome GIO, instead of
occurring virtually instantaneously like they should. A very hackish fix
involves substituting the Gnome library call for a python library call. Of
course that does not truly fix the underlying problem, but it was the best
I could do late last year under the circumstances. But extending that 'fix'
to the backup file processes doesn't work. Instead in certain scenarios
when backing up, the program freezes.

On Mon, Jan 7, 2013 at 5:30 AM, Miloš Popović <email address hidden> wrote:

> This also solves the same Bug in Gentoo.
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1025908
>
> Title:
> Application freezes under Ubuntu 12.10
>
> Status in Rapid Photo Downloader:
> Confirmed
>
> Bug description:
> On Quantal 32bit. version The application runs fine right up until you
> hit the download button. It says file 1 or if lucky 2 of XXX photos or
> videos and then dims out not responding to anything other than a quit from
> the launch icon. However to restart you need to then kill the process using
> system monitor.
> Version 0.4.5-1 installed.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/rapid/+bug/1025908/+subscriptions
>

--
http://www.damonlynch.net

[Neil Woolford (neil-neilwoolford)]

I've just updated my Ubuntu Studio installation to the very latest version, 13.10 (Raring Ringtail) and am glad to be able to report that version 0.4.6 appears to behave well.
I haven't tested it in production yet, but downloading seems fast and reliable on test.

[Damon Lynch (dlynch3)]

It does work a lot better but the solution is a band-aid and not a real
fix. It can still get into serious trouble when downloading videos or
backing up to more than one backup device, for instance. I suspect
switching to the more advanced multi-processing library in python 3.x will
one day help.
<http://www.damonlynch.net>

[Neil Woolford (neil-neilwoolford)]

For my purposes the videos aren't a problem at the moment. However first trials with an external backup device did lead to the program freezing as you mentioned earlier.
I'm very happy to get involved in further testing.

Neil