Rapid Photo Downloader

Application freezes under Ubuntu 12.10, Fedora 18 and other recent releases

Reported by Chris Giltnane on 2012-07-17
118
This bug affects 22 people
Affects Status Importance Assigned to Milestone
Rapid Photo Downloader
Critical
Damon Lynch

Bug Description

On Quantal 32bit. version The application runs fine right up until you hit the download button. It says file 1 or if lucky 2 of XXX photos or videos and then dims out not responding to anything other than a quit from the launch icon. However to restart you need to then kill the process using system monitor.
Version 0.4.5-1 installed.

Related branches

Thanks for your bug report. Please run this from the command line:

rapid-photo-downloader --debug

and copy and paste the results of the terminal output. Also provide
information such as what device you are downloading from and where to.

Downloading was from a directory on hard drive to a different directory. I also use renaming if that makes a difference and no thumbnail creation

[INFO/MainProcess] Rapid Photo Downloader 0.4.5
[INFO/MainProcess] Using pyexiv2 0.3.2
[INFO/MainProcess] Using exiv2 0.23.0
[INFO/MainProcess] Using exiftool 8.60
[INFO/MainProcess] Using hachoir 1.3.3
[INFO/MainProcess] No downloads have occurred so far today
[INFO/MainProcess] allocating a new mmap of length 4096
[DEBUG/MainProcess] created semlock with handle 3023564800
[DEBUG/MainProcess] created semlock with handle 3023560704
[DEBUG/MainProcess] created semlock with handle 3023556608
[DEBUG/MainProcess] created semlock with handle 3023552512
[DEBUG/MainProcess] created semlock with handle 3023548416
[DEBUG/MainProcess] created semlock with handle 3023544320
[DEBUG/MainProcess] created semlock with handle 3023540224
[DEBUG/MainProcess] created semlock with handle 3023536128
[DEBUG/MainProcess] _set_from_toolbar_state
[INFO/GetPreviewImage-1] child process calling self.run()
[DEBUG/MainProcess] Start of day is set to 3:0
[DEBUG/MainProcess] SubfolderFile PID: 5942
[INFO/MainProcess] Using manually specified path /home/chris/Photoimport
[INFO/MainProcess] # photo backup devices: 0; # video backup devices: 0
[INFO/SubfolderFile-2] child process calling self.run()
[DEBUG/SubfolderFile-2] Finished 0. Getting next task.
[DEBUG/MainProcess] created semlock with handle 2909835264
[DEBUG/MainProcess] created semlock with handle 2909831168
[DEBUG/MainProcess] created semlock with handle 2909827072
[DEBUG/MainProcess] Queue._after_fork()
[DEBUG/MainProcess] created semlock with handle 2909822976
[DEBUG/MainProcess] created semlock with handle 2909818880
[DEBUG/MainProcess] created semlock with handle 2909814784
[DEBUG/MainProcess] created semlock with handle 2909810688
[DEBUG/MainProcess] created semlock with handle 2909806592
[DEBUG/Scan-3] Queue._after_fork()
[INFO/Scan-3] child process calling self.run()
[DEBUG/MainProcess] TaskManager PID: 5943
[DEBUG/Scan-3] Scanned 100 files
[DEBUG/MainProcess] on_from_filechooser_button_selection_changed
[DEBUG/Scan-3] Scanned 200 files
[DEBUG/MainProcess] on_from_filechooser_button_selection_changed
[DEBUG/Scan-3] Scanned 300 files
[INFO/Scan-3] process shutting down
[DEBUG/Scan-3] running all "atexit" finalizers with priority >= 0
[DEBUG/Scan-3] running the remaining "atexit" finalizers
[INFO/Scan-3] process exiting with exitcode 0
[INFO/MainProcess] Found 321 photos and videos
[INFO/MainProcess] Files total 640.4MB
[DEBUG/MainProcess] Turning on display of thumbnails
Segmentation fault (core dumped)

A core dump indicates a bug in PIL (python imaging) or pygtk itself.
These things can happen when a distro adopts a newer version of a
compiler. It does not indicate a bug in the code for Rapid Photo
Downloader.

Thanks for that Damon, what can I do to move this forward. Should I raise a bug against one of the packages you mentioned?

I am in Central Asia right now and I'm consequently not in a position to
figure out what the precise cause of the bug is. If you know a bit about
python, you can try to look for it yourself you're feeling brave. You
might find any bug you might report will be a duplicate of an already
reported bug, however!

For the record, the problem is also reproduceable on precise pangolin 12.04. RPD becomes unresponsive once the download button is pressed.

Damon Lynch (dlynch3) wrote :

kyle-hasegawa you need to file a new bug report, including the output of this when run from the command line:

rapid-photo-downloader --debug

Damon Lynch (dlynch3) wrote :

Chris, do you still see the bug? If so, which file type are you downloading?

It works for me with a large variety of different file types on 12.10 beta 1.

Chris Giltnane (chris-giltnane) wrote :

Hi Damon I just tried it forthe first time for weeks and it worked from my HTC phone ok, very slow but it worked . I use jpg's and Samsung dng's mostly.
I will give it some serious exercise and let you know.
32 bit 12.10 beta1.
 I had reduced my import options as well, so will ramp them back up and see what happens.

Hi Damon. running in debug it process 1 file every 28 seconds or so.

To recreate freeze I had the following start position
multiple source files need to be in at least 2 sub directories, it seems to cope if there are only a few files to import but anything in excess of 100 files breaks everytime. I also had some files that had been previously imported, the application seems to slow down considerably if you have these files present. Downloading froma single source directory works everytime although very slowly.

Damon Lynch (dlynch3) wrote :

Are you downloading directly from a camera or are you taking the memory card and inserting it into a memory card reader? When downloading directly from a camera or a phone, Rapid Photo Downloader uses gnome system libraries. The gnome developers do the best but they have bugs. A possible workaround is to use Nautilus to copy the files onto your desktop, and then use Rapid Photo Downloader to import and rename them as you prefer.

Hi Damon I have tried both of these options, direct from the camera and
then copying into a directory on the hard drive. Seems to make no
difference. The trigger seems to be multiple input directories. So for
instance if the import directory is /dcim then if I have /dcim/1 and
dcim/2 then it seems to get to the end of the first directory and then
freeze and never get to the next directory.

There is a apport bug on launchpad dont know if that will help

https://bugs.launchpad.net/bugs/1048796

On 11/09/12 19:36, Damon Lynch wrote:
> Are you downloading directly from a camera or are you taking the memory
> card and inserting it into a memory card reader? When downloading
> directly from a camera or a phone, Rapid Photo Downloader uses gnome
> system libraries. The gnome developers do the best but they have bugs.
> A possible workaround is to use Nautilus to copy the files onto your
> desktop, and then use Rapid Photo Downloader to import and rename them
> as you prefer.
>

--
Chris Giltnane

Damon Lynch (dlynch3) wrote :

Is it possible for you to share a copy of the exact files using the
directory structure you describe? If so, and the files are large, a service
like dropbox may help.

Don't have dropbox but do have a ubuntu one account that we could use if
you have that. If not I may be able to give you temporary ftp access to
my NAS to get the files. I will zip up a sample and get something going.
On 11/09/12 20:52, Damon Lynch wrote:
> Is it possible for you to share a copy of the exact files using the
> directory structure you describe? If so, and the files are large, a service
> like dropbox may help.
>

--
Chris Giltnane

Damon Lynch (dlynch3) wrote :

I also have Ubuntu One. I've not used it for file sharing but I should be
able to figure it out ;-)

After some time I was finally able to take a look at the test files you provided. I was unable to replicate the crash.

Thanks for looking at this. I still have the problem but next week I am
moving to a new machine and 64bit will see if i still get the issue
after that.
On 25/09/12 05:08, Damon Lynch wrote:
> After some time I was finally able to take a look at the test files you
> provided. I was unable to replicate the crash.
>

--
Chris Giltnane

Hello,

Using a 64bit environment, I have the same problem.

When using Pangolin (12.04), everything was ok.

After moving to Quantal in order to have a newer Digikam, RPD became useless...
The time needed to download pictures exploded.

Worst case : having multiple folders created with many dates : seems like whole process repeats again.
Main window doesn't react, even if debug terminal shows action.

Final download speed seems approaching 500 kb/s while previous use with Pangolin, we were something like 20MB/s.

Damon Lynch (dlynch3) wrote :

Hello Olivier,

You may need to open a new bug report. If you do, You need to provide information such as from what device you are downloading, the exact problem, output from the terminal using the --debug command line option, and that type of files you are downloading/ importing. Without this kind of information, it's impossible to diagnose any problem.

Also please note that Ubuntu 12.10 is still in Beta. Experience shows that there are bound to be problems with Python, Python modules, and the system libraries, causing crashes that are of a general nature and not specific to Rapid Photo Downloader.

You may well be right that it is an external issue in python, I just has
the same or at least very similar issue importing to shotwell using its
native importer. Using a similar target structure of Year/Month/Day. As in
RPD it seems related to the number of sub-directories under the top level
source along with multiple target directories.

On 29 September 2012 17:45, Damon Lynch <email address hidden> wrote:

> Hello Olivier,
>
> You may need to open a new bug report. If you do, You need to provide
> information such as from what device you are downloading, the exact
> problem, output from the terminal using the --debug command line
> option, and that type of files you are downloading/ importing. Without
> this kind of information, it's impossible to diagnose any problem.
>
> Also please note that Ubuntu 12.10 is still in Beta. Experience shows
> that there are bound to be problems with Python, Python modules, and the
> system libraries, causing crashes that are of a general nature and not
> specific to Rapid Photo Downloader.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1025908
>
> Title:
> Application seems to freeze after first few photo's
>
> Status in Rapid Photo Downloader:
> New
>
> Bug description:
> On Quantal 32bit. version The application runs fine right up until you
> hit the download button. It says file 1 or if lucky 2 of XXX photos or
> videos and then dims out not responding to anything other than a quit from
> the launch icon. However to restart you need to then kill the process using
> system monitor.
> Version 0.4.5-1 installed.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/rapid/+bug/1025908/+subscriptions
>

I installed 12.04 the problem went away.
Installed 12.10 64Bit, I was on 32bit before and the problem returned so it looks like a problem in an external dependency.

Whatever the cause of the problem, it's a real problem, apparently making
Rapid Photo Downloader unusable on 12.10. Unfortunately for the next three
weeks I'm racing to meet some academic deadlines and am therefore unable to
diagnose the root cause of the problem. I hope to be able to take a
serious look at in November.

i can confirm this bug. Since i upgraded to ubuntu 12.10 (pangolin), the r-p-d wont work anymore.

Damon Lynch (dlynch3) on 2012-10-23
Changed in rapid:
status: New → Confirmed
summary: - Application seems to freeze after first few photo's
+ Application freezes under Ubuntu 12.10

Hi Damon,
Do you have any plans to work on this the upcoming weeks?

Bob Leers (bobslee) wrote :

i'm willingly, but don't have lotta spare time.

wat could i do?
reading debug log and check code calling dependencies? right now i dunno.

Damon Lynch <email address hidden> schreef:

>Bob are you offering to help?
>
>--
>You received this bug notification because you are subscribed to the
>bug
>report.
>https://bugs.launchpad.net/bugs/1025908
>
>Title:
> Application freezes under Ubuntu 12.10
>
>Status in Rapid Photo Downloader:
> Confirmed
>
>Bug description:
>On Quantal 32bit. version The application runs fine right up until you
>hit the download button. It says file 1 or if lucky 2 of XXX photos or
>videos and then dims out not responding to anything other than a quit
>from the launch icon. However to restart you need to then kill the
>process using system monitor.
> Version 0.4.5-1 installed.
>
>To manage notifications about this bug go to:
>https://bugs.launchpad.net/rapid/+bug/1025908/+subscriptions

--
groeten, Bob

Damon Lynch (dlynch3) wrote :

If you know python, the biggest help right now would be to profile the
application to identify what is causing it to run so slowly. Once the cause
is known, fixing it will hopefully not be difficult. If you don't know
python however, then it's a task best left to someone who does.

I know a bit of Python.
So I profiled the whole program (func call): rapid.start(), with cProfile.
Right I'm not able to interpret these results - it's almost midnight GMT+2 here.
I hope I have some free time next days. Then I'll let you know.

ofcourse one should profile other parts/funcs instead of starting the program..
i need some sleep.
--
groeten, Bob

I spend several hours in profiling (cProfile and runsnakerun), but I couldn't find any clues.
Sorry, I give up.. due to my inexperience and not enough spare time.

I hope you could work on it sometime.
Thanx.

Thanks for your efforts Bob! Debugging a multi-process program with code
that runs in parallel is not easy.

Damon Lynch (dlynch3) on 2012-11-19
Changed in rapid:
importance: Undecided → Critical
assignee: nobody → Damon Lynch (dlynch3)
milestone: none → 0.4.6

I have identified the line of code that is causing the bug. It is in the file subfolder.py and is this line:

temp_file.move(download_file, self.progress_callback_no_update, cancellable=None)

That's a call to Gnome's libraries to rename / move a file. For some reason each time it runs under Ubuntu 12.10 it is taking an extremely long time to complete.

Next: figure out what to do about it!

Bob Leers (bobslee) wrote :

@Damon
Do you have the time to fix it? What can we expect?
Thanks for your efforts anyway.

I believe the error is caused by something in Gnome's library. I am unable to copy files from my eos550d using Nautilus, RPD and Shotwell. Copying from eos350d and olympus TG-810 is however working.

Copying from eos550 worked untill I upgraded fraom 12.04 to 12.10.

Arjan (arjan-scherpenisse) wrote :

That's strange, copying the photos from the DCIM folders with Nautilus works fine for me..

actually that is how I work around it now that rapid is broken... :-/

I use nautilus as a workaround too and i it works fine for me too.

Damon Lynch (dlynch3) wrote :

@Trond the bug you found is unrelated to the current problem.

You can try a preliminary and tentative fix for Rapid Photo Downloader under Ubuntu 12.10 in this PPA:
https://launchpad.net/~dlynch3/+archive/rapid-photo-downloader-testing

Let me know if it improves anything. I don't anticipate having any more time to work on it for another 3 or 4 weeks.

Yopland (yopland) wrote :

This version seems to work fine on LinuxMint 14

Henrik Toth (thk) wrote :

The slow-down is gone on Ubuntu 12.10 with this 0.4.6 beta 1, but there is a new bug: now the thumbnail preview duplicates every image.

Brent McLaren (bmclaren99) wrote :

The duplicate issue was happening to me (with the slow response) . Haven't tried the beta to see if it fixed the slowness, but when I got duplicates, I would see two entries at the top of the screen where it detected the card that was inserted. This didn't happen every time though.

The card was inserted into a memory card reader, or some other kind of
device?

On Mon, Nov 26, 2012 at 8:29 PM, Brent McLaren
<email address hidden>wrote:

> The duplicate issue was happening to me (with the slow response) .
> Haven't tried the beta to see if it fixed the slowness, but when I got
> duplicates, I would see two entries at the top of the screen where it
> detected the card that was inserted. This didn't happen every time
> though.
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1025908
>
> Title:
> Application freezes under Ubuntu 12.10
>
> Status in Rapid Photo Downloader:
> Confirmed
>
> Bug description:
> On Quantal 32bit. version The application runs fine right up until you
> hit the download button. It says file 1 or if lucky 2 of XXX photos or
> videos and then dims out not responding to anything other than a quit from
> the launch icon. However to restart you need to then kill the process using
> system monitor.
> Version 0.4.5-1 installed.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/rapid/+bug/1025908/+subscriptions
>

--
http://www.damonlynch.net

Yes, card was installed in a reader. Just watched what it was doing, and I am seeing the first line show up when it auto-launches, and the second entry shows up shortly after. I can send a screen shot if that works.

Please open a new bug to report this new issue. It sounds pretty serious.

On Mon, Nov 26, 2012 at 9:12 PM, Brent McLaren
<email address hidden>wrote:

> Yes, card was installed in a reader. Just watched what it was doing,
> and I am seeing the first line show up when it auto-launches, and the
> second entry shows up shortly after. I can send a screen shot if that
> works.
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1025908
>
> Title:
> Application freezes under Ubuntu 12.10
>
> Status in Rapid Photo Downloader:
> Confirmed
>
> Bug description:
> On Quantal 32bit. version The application runs fine right up until you
> hit the download button. It says file 1 or if lucky 2 of XXX photos or
> videos and then dims out not responding to anything other than a quit from
> the launch icon. However to restart you need to then kill the process using
> system monitor.
> Version 0.4.5-1 installed.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/rapid/+bug/1025908/+subscriptions
>

--
http://www.damonlynch.net

The duplication issue happens also to me, and I also use a card reader; hitting "refresh", only one of the card entries remains.
Waiting for the new bug to join the party there.

Ulisse (ulipo) wrote :

Ok, just tried the beta and I can confirm the "slow download" issue is gone.
Duplicated sources is still there, I opened the relative bug: #1083756

Bob Leers (bobslee) wrote :

Confirm.
Tested 0.4.6 beta1, with import of +/- 700 pictures. It succeeds with acceptable speed.
Thanx for your efforts Damon!

works very well.... thanks Damon!

mabunix (majobu) wrote :

Same problem as described here on Archlinux 64 Bit. Is there a tarball available to test the beta?

sbolten (stephan-bolten) wrote :

Updated to 0.4.6 beta1 - download working now but still very slow compared to a download of the exact same pics to the exact same target structure on Ubuntu 12.04

Where can i get 0.4.6 b1? It isn't on the PPA.

Damon Lynch (dlynch3) wrote :

sbolten I need more details. If you can share with me the exact files you
are trying to download, and the exact file renaming and subfolder creation
scheme you are using, that will be most helpful.

Damon, thanks for the tarball. I just gave it a try and I can confirm the duplication bug. Downloading from SD card works faster again compared to 0.4.5. I tried it with JPEGs and RAWs from a Nikon D600. System used: ArchLinux 64 Bit.

sbolten (stephan-bolten) wrote :

Damon, I have no clue why - but after a couple of reboots and system updates I have now a performing installation of 0.4.6 beta 1... thx for your support

Olivier L (ceituna) wrote :

Damon,

Thank you for the fix. Seems to solve problem commented on #19

Miloš Popović (gpopac) wrote :

This also solves the same Bug in Gentoo.

Unfortunately it is still not fixed. This is an overview the current
situation: I first diagnosed the bug to a problem making calls in a child
daemon process that renames files. Each file rename would take
approximately 60 seconds to complete when using Gnome GIO, instead of
occurring virtually instantaneously like they should. A very hackish fix
involves substituting the Gnome library call for a python library call. Of
course that does not truly fix the underlying problem, but it was the best
I could do late last year under the circumstances. But extending that 'fix'
to the backup file processes doesn't work. Instead in certain scenarios
when backing up, the program freezes.

On Mon, Jan 7, 2013 at 5:30 AM, Miloš Popović <email address hidden> wrote:

> This also solves the same Bug in Gentoo.
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1025908
>
> Title:
> Application freezes under Ubuntu 12.10
>
> Status in Rapid Photo Downloader:
> Confirmed
>
> Bug description:
> On Quantal 32bit. version The application runs fine right up until you
> hit the download button. It says file 1 or if lucky 2 of XXX photos or
> videos and then dims out not responding to anything other than a quit from
> the launch icon. However to restart you need to then kill the process using
> system monitor.
> Version 0.4.5-1 installed.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/rapid/+bug/1025908/+subscriptions
>

--
http://www.damonlynch.net

Damon Lynch (dlynch3) on 2013-01-23
summary: - Application freezes under Ubuntu 12.10
+ Application freezes under Ubuntu 12.10, Fedora 18 and other recent
+ releases
Changed in rapid:
status: Confirmed → In Progress

I've just updated my Ubuntu Studio installation to the very latest version, 13.10 (Raring Ringtail) and am glad to be able to report that version 0.4.6 appears to behave well.
I haven't tested it in production yet, but downloading seems fast and reliable on test.

It does work a lot better but the solution is a band-aid and not a real
fix. It can still get into serious trouble when downloading videos or
backing up to more than one backup device, for instance. I suspect
switching to the more advanced multi-processing library in python 3.x will
one day help.
<http://www.damonlynch.net>

For my purposes the videos aren't a problem at the moment. However first trials with an external backup device did lead to the program freezing as you mentioned earlier.
I'm very happy to get involved in further testing.

Neil

Damon Lynch (dlynch3) on 2014-01-13
Changed in rapid:
status: In Progress → Fix Committed
milestone: 0.4.6beta1 → 0.4.9beta1
Damon Lynch (dlynch3) on 2014-01-16
Changed in rapid:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Related questions