Provide a QML API to use with images in qrc files

[Simon Stürz]

The thumbnailer works only if I load images from the file system, not from a resource file:

    Image {
        id: backgroundImage
        anchors.fill: parent
        sourceSize: Qt.size(parent.width, parent.height)
        source: "image://thumbnailer/qrc:///images/menu-background.jpg"
    }

The default sdk template uses qrc as template.

[Michi Henning (michihenning)]

If you want a thumbnail for an image, you have to give an image: url to qml.

See https://developer.ubuntu.com/en/apps/qml/tutorials/use-ubuntu-thumbnailer/

[Michael Zanetti (mzanetti)]

Michi, I don't think this case should be dismissed so quickly. If one creates a new default app template with out SDK, putting images into qrc files is the default way to go. In order to create games that look good on big screens like tablet, but still perform well on small screens and weak hardware, the thumbnailer could be an ideal helper so that developers can provide large graphics in their package but use the thumbnailer to size them properly for the current device.

I think simon just forgot the "image://thumbnailer/" in the above example but he really meant that. I've added it.

[Michael Zanetti (mzanetti)]

I just confirmed this with a snippet of code:

Works:

Image {
  width: 100
  height: 100
  sourceSize: Qt.size(width, height)
  source: "/image.png"
}

Works:

Image {
  width: 100
  height: 100
  sourceSize: Qt.size(width, height)
  source: "image://thumbnailer/file:///image.png"
}

Works:

Image {
  width: 100
  height: 100
  sourceSize: Qt.size(width, height)
  source: "qrc:///image.png"
}

Fails:

Image {
  width: 100
  height: 100
  sourceSize: Qt.size(width, height)
  source: "image://thumbnailer/qrc:///image.png"
}

This throws the following error:

Thumbnailer: RequestImpl::dbusCallFinished(): D-Bus error: DBusInterface::GetThumbnail(): ap.png: unity::ResourceException: Thumbnailer::get_thumbnail():
    boost::filesystem::canonical: No such file or directory: "/image.png"
qrc:///Main.qml:19:5: QML QQuickImage: Thumbnailer: RequestImpl::dbusCallFinished(): D-Bus error: DBusInterface::GetThumbnail(): ap.png: unity::ResourceException: Thumbnailer::get_thumbnail():
    boost::filesystem::canonical: No such file or directory: "/image.png"

=> As stated before, putting media into a qrc file is the way the default template in the SDK suggests to do it (in order to not bother the developer with finding the apparmor confined paths on the phone) and thus should probably also work with the thumbnailer.

As the thumbnailer is remote process and thus won't have access to the in-binary resources files, the QQmlImageProvider could perhaps extract them from the binary, provide them to the thumbnailer and delete the temporarily extracted original file again.

[Michi Henning (michihenning)]

A fix for the incorrect processing of relative path names is sitting in silo 75 at the moment. It's waiting for testing sign-off.

The thumbnailer qml plug-in just does this:

QQuickImageResponse* ThumbnailGenerator::requestImageResponse(const QString& id, const QSize& requestedSize)
{
    QString src_path = QUrl(id).path();
    auto request = thumbnailer->getThumbnail(src_path, requestedSize);
    return new ThumbnailerImageResponse(request);
}

So, whatever comes back as the path component of the URL is what we use. If that path is relative, we interpret it relative to the current directory. If the QUrl(id).path() call takes care of expanding a relative qrc path to a path relative to the current dir, or to an absolute path, things should work. If path() expands it to a relative path with respect to the directory containing the qrc file, it won't.

[Michi Henning (michihenning)]

> If path() expands it to a relative path with respect to the directory containing the qrc file, it won't.

Unless the working dir of the app is the dir containing the qrc file.

[Michael Zanetti (mzanetti)]

I don't think this has to do with relative path names. The way .qrc files work is that the image files are compiled into the application binary and it sets up a virtual file system inside the binary that contains stuff from the resource file. The path "qrc:///image.png" is actually an absolute one.

[Michi Henning (michihenning)]

Well, there was definitely a bug in the thumbnailer that relates to relative pathnames, and I just picked up on the error message you copied.

If there is a virtual filesystem somewhere, the thumbnailer has no idea that it exists. In a nutshell, the thumbnailer can produce a thumbnailer for a path in the filesystem, no more, no less. If there is no path, there is no thumbnail :-(

[Albert Astals Cid (aacid)]

The thumbnailer is an external process, it can't see the qrc resource

[Michael Zanetti (mzanetti)]

As written in comment #3

"As the thumbnailer is remote process and thus won't have access to the in-binary resources files, the QQmlImageProvider could perhaps extract them from the binary, provide them to the thumbnailer and delete the temporarily extracted original file again."

[Michi Henning (michihenning)]

I really don't think this is a thumbnailer problem. The thumbnailer should know about things such as qrc virtual filesystems. It simply maps a path name to a thumbnail. That's it, and it should stay that way.

[Michi Henning (michihenning)]

Sorry, "The thumbnailer should not know about..."

[Michael Zanetti (mzanetti)]

The thumbnailer itself would not really need to know about th qrc stuff. Just the QQmlImageProvider part of the thumbnailer package. That could extract the image data from the qrc and pass it on to the thumbnailer.

Here's some code snipped that should give directions on how to do that: http://paste.ubuntu.com/15258195/

[Michi Henning (michihenning)]

This looks doable, at least in principle, but raises a few questions.

How does the thumbnailer decide *where* in the file system to write the file? Note that, if the file gets written somewhere where other applications can read it, this will potentially allow application B to see a thumbnail created by application A, provided application B knows the path name.

How do we clean up the files that are created that way? We could do it as soon as a thumbnail request for the file completes, but that would mean that we write the entire file into the file system for each thumbnail request, which is horribly expensive. It also would effectively disable caching because inode number and modification time are part of the lookup key into the cache.

If we don't clean up files immediately, when and how do we do it? One way might be to have a small persistent cache on the client side that maps qrc URLs to path names. We could then delete a temporary file when its entry falls out of the client-side cache. (There are callback hooks in persistent-cache-cpp that make this possible.) But it feels a bit hacky. For example, a crash at the wrong moment can mean that we accidentally leave a file on disk behind, so that we still would have to add a garbage collector for the temporary files. Sizing the client-side cache correctly is also problematic. (50 video files are very much larger than 50 photos.)

I'm not trying to rain on the party here. But we need to have a strategy for dealing with this before we implement something, only to find out that it's too expensive or clunky. Part of the problem seems to be that the qrc files create a virtual file system, instead of having real things in the real file system. This makes it difficult to leverage other tools that operate on files.

[James Henstridge (jamesh)]

Rather than looking at what is necessary to make files compiled into an executable visible to an external process, could we step back a bit to find out what we're trying to accomplish?

What do you think will be done differently by having "qrc:///images/menu-background.jpg" passed through the thumbnailer rather than being loaded directly by the Image component? If this is just a toy example and your real use case is a little more complex, could you describe that instead?

[Michael Zanetti (mzanetti)]

The use case is for scaling the images. Especially in games you often want to ship larger artwork so that the game looks good on a tablet. However, running that game on a phone then will be slow because of the big amount of artwork that needs to be scaled down all the time. That's when I thought the thumbnailer could be a good helper to just scale things down once and have them faster loading afterwards.

[Michael Zanetti (mzanetti)]

If you guys think this is out of scope for the thumbnailer, please let me know. I need some more flexible image cache in many places. If the thumbnailer is not the right place to do this, I will try to come up with something else for those cases.

The reason why I thought this would fit for the thumbnailer is because it already does exactly this use case for images, except it doesn't work for qrc files.

[Michi Henning (michihenning)]

Still not trying to rain on the part here... :-)

I'd like to make it work. But having images buried inside an executable is not a good idea when it comes to interoperating with tools such as the thumbnailer. Personally, I don't think the thumbnailer is a good fit for this.

Is it possible to *not* bury the images that way? If so, and they are proper files, the thumbnailer will instantly do what you want.

If all that is needed is scaling, the thumbnailer may well be overkill though. For example, you could ship different assets for different resolutions (that's the iOS approach), or you could scale the image once and keep it in your own persistent-cache-cpp instance. The API for that is child's play, so that might be viable.

For an image cache, I'd seriously suggest to look at persistent-cache-cpp. It underpins the thumbnailer, and it works really well. Please ping me if you have questions about cache coherency, dealing with updated images, etc. I'm happy to help. But, so far, it looks to me the the thumbnailer is far too heavy-weight a solution for what you need, and persistent-cache-cpp would be a better fit.

[Michael Zanetti (mzanetti)]

* Sure it is possible to not ship artwork in a qrc. However, that requires the developer to fiddle with the apparmor confined paths and afaict, there's no QML-only way to do so. Besides, as I said before, the default in our sdk templates is to put things in a qrc to ease up development.

* I agree that the persistent-cache-cpp is childplay for people like us, again there's no qml api for it so requires people to dive into C++ which apparently many refuse to do. So the way forward would be to create another QML image provider pretty much in the same way as the thumbnailer does, except scaling and caching things only. This is probably what I'd do in case we decide this out of scope for the thumbnailer.

* Shipping different sized assets, sure, possible too. But you probably won't ever be able to ship perfectly sized assets for every screen we run on, at least not when keeping the package to a reasonable size. I hoped we could be more clever in this regard than going the easy way.

In any case. I totally understand if we decide this is out of scope for the thumbnailer. Still a bit wondering why it does implement the scaling of local images use case then at all and does not just media artwork matching/fetching alone then.

[Michi Henning (michihenning)]

> Still a bit wondering why it does implement the scaling of local images use case then at all and does not just media artwork matching/fetching alone then.

From the application perspective, it wants to say "give me an image in such and such a size for this thing", where the thing could be just about anything (image, video, audio, plus others we might add in the future). Extraction of the images is not simple because we have to deal with many different media types, and applications shouldn't be burdened with that. Images can also be encoded in many different formats, and they may need rotating in some cases. And doing all of this in a way that is scalable and efficient is very much non-trivial. The thumbnailer handles all of these things transparently for applications.

Because applications tend to ask for an image in only a limited number of small sizes, it makes sense to do the scaling inside the thumbnailer because it caches the scaled images. In turn, we get much bigger bang for our disk space. Storing the scaled image also means that the Qt/QML layer doesn't have to repeatedly scale down images on the fly, which burns a lot of battery, and retrieving an already-cached thumbnail is lightning fast.

If all you need is image scaling from pre-existing image files (not other media files) or image contents, things are not hard to do in C++. We already have an Image class that does all the scaling/rotation work, and the C++ cache takes care of the caching. Both APIs are simple and small. pesistent-cache-cpp is already available as stand-alone library, and the Image class is small, one source file, maybe around 300 lines of code. It only needs gdk_pixbuf and libexif to work.

So, creating a QML stand-alone module that can do this might really be the best way to go. The persistent cache does not provide an async interface, but it would be easy to use a QThreadPool with a single thread to make it async. (Making the Image class async is probably not worth it, I suspect it'll always be fast enough.)

Architecturally, I think that's cleaner too. For one, no need to call things over DBus, install a separate service, etc. And there are no security issues to deal with because this would all run embedded in the calling application. This seems better to me than trying to shoehorn the thumbnailer into doing things it wasn't designed for.