Cache preview and thumbnail images

[Jussi Pakkanen]

Currently the Dash always reloads preview images. This includes both web results such as those from Amazon or local files such as images. It should cache those to reduce networks and battery usage. It should follow appropriate standards such as the freedesktop.org thumbnailing standard or the Media Art storage standard.

There should be only one centralised location where thumbnails are fetched and stored and that should be in the dash. This is because we want to do load images lazily, and the dash is the one piece that shows images to the user.

An example data flow would go like this:

- mediascanner scans the user's music files
- it looks up album art download addresses from last.fm
- it stores the address somewhere
- user types a music query
- the music scope returns results whose metadata list the download address
- dash loads the actual images as necessary and stores them in the cache

[Michal Hruby (mhr3)]

A few notes from saviq and me about what should the service do ideally:

• driven from dash
• driven by the media scanner (additional)
• ability to cancel requests
• consider power connection (do more on AC, less on battery)
• continue with cancelled requests?
• ultimately a system service
• ageing mechanism

[Michal Hruby (mhr3)]

Also, since mediascanner is used in the example, how will other apps deal with the not-yet-downloaded album art? I'm thinking about the music-app, which should also use mediascanner's data.

[Omer Akram (om26er)]

FWIW this does not happen under Mir.

[Michał Sawicz (saviq)]

On 18.09.2013 14:37, Michal Hruby wrote:
> Also, since mediascanner is used in the example, how will other apps
> deal with the not-yet-downloaded album art? I'm thinking about the
> music-app, which should also use mediascanner's data.

Well, I'd assume the "album art image provider", that should be part of
the SDK, would take care of all of that.

--
Michał (Saviq) Sawicz <email address hidden>
Canonical Services Ltd.

[Michał Sawicz (saviq)]

Omer, sure it does - you might be thinking of bug #1193099 that doesn't happen under Mir.

[Omer Akram (om26er)]

oh - right, yes. I meant that case only.

[Jussi Pakkanen (jpakkane)]

Cancellability is really, really hard. Is it absolutely necessary? Would a timeout value be sufficient?

[David Planella (dpm)]

We've been having that discussion for the Music app recently [1].

Right now the Music app uses its own media scanner reads the cover art from the embedded images in the media files. However, we are now migrating from the custom implementation to using mediascanner, and we found out that we can no longer do this, as mediascanner via grilo does not read locally embedded cover art data but fetches it from the net via last.fm. Related to that, we found out about the non-existing caching.

My comment is rather on the process of fetching before catching. From the bug's description, I would suggest a adding a step to look up for the cover art locally first, before accessing the network. I believe that's what the U1 Music app on Android does as well. That is:

- mediascanner scans the user's music files
=> it looks up album art from locally embedded images on the media files
- _if not available locally_, it looks up album art download addresses from last.fm
-...

Also, it's the first time I've heard from an "album art image provider" in the SDK. What is this project and who's working on it?

[1] https://code.launchpad.net/~vthompson/music-app/use-mediascanner2/+merge/185403

[Jussi Pakkanen (jpakkane)]

There is code for extracting image data out of mp3 files. It will be added to wherever this functionality ends up going.

[David Planella (dpm)]

Thanks Jussi. Would you mind elaborating a bit more on that?

It would help us very much to remove uncertainty as to whether we need to plan for adding functionality to the Grilo QML plugin the Music app uses to talk to mediascanner/grilo or whether we will get this functionality transparently from the backend.

[Michał Sawicz (saviq)]

On 19.09.2013 03:07, Jussi Pakkanen wrote:
> Cancellability is really, really hard. Is it absolutely necessary? Would
> a timeout value be sufficient?

Thing is, imagine scrolling quickly through a list of 200 items that
don't have their images available locally. That will end up queueing 200
image requests, when only the last ones are actually interesting when
the view settles. The queue should be a ~ LIFO (ideally with a few
active workers), I imagine, to cater for the fact that you're usually
interested most in the last ones that are requested.

So, if we can't cancel, all of those requests would get queued up and
will be acted upon later, when you don't even necessarily need them any
more. One question is whether there even is the possibility of
implementing cancellation in the image providers - I would assume the
QImage returned from the provider would get destroyed when not needed
any more, at which point you could tell the remote service to cancel
that request. TBH it could really only mean skipping them in the queue
and acting again on that queue onIdle (+onWiFi +onPower?, as Michal
mentioned).

Now it occurred to me - the queue should probably be unique, and promote
older items on new requests for the same asset.

How's that sound?
--
Michał (Saviq) Sawicz <email address hidden>
Canonical Services Ltd.

[Victor Thompson (vthompson)]

On the note of extracting media cover art from MP3's: The ability to extract the cover art from a URL was added to the Filemanager's FolderListModel plugin (which is installed by default) [1]. Someone could package JUST the image provider portion of it, perhaps.

[1] https://code.launchpad.net/~ubuntu-filemanager-dev/ubuntu-filemanager-app/plugin

[Jussi Pakkanen (jpakkane)]

@michal: limiting the amount of queries is a good thing but having cancellables might not be the best way to achieve that. There needs to be a rate limiter somewhere but it is uncertain what is the best place for it.

The thumbnailer should not place arbitrary limits on how much work it does. There may well be cases where someone wants to shove 1000 queries in and expect them to be done as fast as possible. Therefore the limiting on query speeds should be done outside the thumbnailer.

Furthermore we had a discussion on the API it should provide and we came to the conclusion that it should be done in a similar way to Unity-API. That is, it can not expose any implementation details (be them Qt, or anything else). The reason for this is that this service is useful for all kinds of applications and we should not arbitrarily limit it to Qt apps. (There could be Qt wrappers, of course.)

Any app using the thumbnailer could use whatever tasking or queue mechanism they choose. Basically it would boil down to having a worker thread with a QQueue of strings and resolving them one by one. Cancellation would then just be a case of calling clear on the queue. This is massively simpler to implement than trying to make third party code (gstreamer, http downloading and the like) cancellable. The only downside is that you can't cancel the in-flight query, but since there is only one, it is not an issue.

This would also decouple queueing strategy from thumbnail generation so all applications could have their own strategy and they could alter it without any code changes on the generator side.

[Michał Sawicz (saviq)]

On 19.09.2013 09:05, Jussi Pakkanen wrote:
> @michal: limiting the amount of queries is a good thing but having
> cancellables might not be the best way to achieve that. There needs to
> be a rate limiter somewhere but it is uncertain what is the best place
> for it.
>
> The thumbnailer should not place arbitrary limits on how much work it
> does. There may well be cases where someone wants to shove 1000 queries
> in and expect them to be done as fast as possible. Therefore the
> limiting on query speeds should be done outside the thumbnailer.
>
> Furthermore we had a discussion on the API it should provide and we came
> to the conclusion that it should be done in a similar way to Unity-API.
> That is, it can not expose any implementation details (be them Qt, or
> anything else). The reason for this is that this service is useful for
> all kinds of applications and we should not arbitrarily limit it to Qt
> apps. (There could be Qt wrappers, of course.)

Sure, of course - the Qt image provider implementation would just be one
of them.

> Any app using the thumbnailer could use whatever tasking or queue
> mechanism they choose. Basically it would boil down to having a worker
> thread with a QQueue of strings and resolving them one by one.
> Cancellation would then just be a case of calling clear on the queue.
> This is massively simpler to implement than trying to make third party
> code (gstreamer, http downloading and the like) cancellable. The only
> downside is that you can't cancel the in-flight query, but since there
> is only one, it is not an issue.
>
> This would also decouple queueing strategy from thumbnail generation so
> all applications could have their own strategy and they could alter it
> without any code changes on the generator side.

Right, that does give better flexibility, but creates additional work
for all the consumers.

All in all, my requirements were high level ones, and they stand. And I
don't expect to have to implement the default Qt image provider, for
which the behavior we described above seems to cover most of the
usecases - LIFO queue with the ability to drop things from the queue
when the corresponding consumer is destroyed (cancelling "in progress"
jobs is probably not needed).

--
Michał (Saviq) Sawicz <email address hidden>
Canonical Services Ltd.

[Sergio Schvezov (sergiusens)]

I bought six SIMs for the sprint here with 5€ each; they lasted 2 days while there was no "network browsing".

This bug should be made critical, our credit was wiped clean because of just rebooting and refreshing...