RFE: Initial support for Wallhaven.cc

[PheniX]

Hi
I have quickly hacked the Wallbase.cc scripts to fetch wallpapers from alpha.wallhaven.cc
The patch against variety version 0.4.20 is attached

Ciao
Andrea

[PheniX (a-sterbini)]

I was too quick :-)

Now I hope the search function is fixed ... see diff attached (again, against v 0.4.20)

Ciao
Andrea

[Peter Levi (peterlevi)]

Have you forgotten to add the new files to the patch (e.g. WallhavenDownloader, AddWallhavenDialog and the GUI files from data/ui - *.ui, *.xml) ?

[PheniX (a-sterbini)]

Yes
sorry
here they are

Andrea

[Peter Levi (peterlevi)]

Thanks a lot, I'll take a look and will merge it in hopefully soon.

[Peter Levi (peterlevi)]

Please attach also the AddWallhavenDialog.py file. Thanks.

[PheniX (a-sterbini)]

Perhaps Wallhaven should be added to http://smarturl.it/varietyserveroptions ?

[PheniX (a-sterbini)]

There's been a small change in Wallhaven search url.

Here is the patch

[PheniX (a-sterbini)]

Hi Peter
today I was asking myself if it's possible to refactor a Downloader so that all specific bits (read CSS selectors) are collected in a very simple subclass.
This would help designing a generic CSS-based downloader, which could be easily-customized to other wallpaper sites just by specifying the CSS selectors required to extract the necessary info from the pages.
So, after half an hour work I got the enclosed WallhavenDownloader example, refactored to remove all CSS selectors from the code in a simpler subclass.
Perhaps the idea could be useful ...

Ciao
AndreaS

PS I haven't yet compared the downoader with the other downloaders (which would allow for further refactorization)

[PheniX (a-sterbini)]

Still testing it ...

[PheniX (a-sterbini)]

Here it
(I had some troubles learning CSS selectors' limits :-) )
(probably it would be better to use XPath selectors, shouldn't they be more powerful than CSS selectors?)

The refactored WallhavenDownloader class becomes just

class WallhavenDownloader(CSSDownloader):
    _NAME = "wallhaven"
    _SITE = "Wallhaven.cc"
    _SEARCHURL = "http://alpha.wallhaven.cc/search?q=%s&categories=111&purity=100&sorting=favorites&order=desc"
    _COUNT_CSS = 'header.listing-header h1'
    _WALL_CSS = 'figure.thumb'
    _LINK_CSS = 'figure.thumb a.preview'
    _SRC_CSS = 'img#wallpaper'
    _TAGS_CSS = 'a.tagname'
    _PURITY_CSS= 'input[name="purity"]'
    _RES_CSS = 'span.wall-res'
    _PREVIEW_CSS= 'a.preview'

[Peter Levi (peterlevi)]

PheniX, thanks for the effort again. Yes, such an abstraction is possible, but I think this level of abstraction is not general enough to be of much use. To see what I mean - try finding another website apart from Wallhaven for which this downloader would work with only modifying the CSS selectors. Basically there are usually some differences that would break this approach - search might be missing, with list of "Categories" present instead; search might be paged in different ways, and a count may or may not be present; The levels of indirection till one reaches the actual image from the search page might be different, etc. Purity, tags, etc. are entirely specific to Wallbase - other sites may not even have these concepts or they will be extracted in a very different manner (e.g. check the WallpapersNetDownloader).

Now, I totally agree that the image downloading functionality should be abstracted and made pluginnable, and this is in the queue, but most of the work is actually not on the Downloaders level - they can easily become interfaces along the lines of validate_query(), fill_queue(), download_one(), etc. Most of the work is abstracting out the GUI for adding/editing new image sources - e.g. Flickr and Wallbase currently use different GUIs to add a new image source, and while Wallbase and Wallpapers.net have something in common - that you specify a single URL (or a query in the case of Wallbase), we still want the proper specific description to what is expected.

Basically I'm putting off this refactoring for two reasons - there is at the moment quite a lot of dynamics there as I'm preparing the Beta version which has a serverside part, and everything related to image sources is subject to change because of this, so no use to go into refactoring it before the serverside is more stable. The second is that up till now there was not that much interest from other developers to start contributing to Variety, so this task was somewhat low in the priority list, but you are proof this is changing :-)

[PheniX (a-sterbini)]

Hi Peter
thanks a lot for your kind reply. I did'nt want to pressure you, the server side is definitely more important and it introduces beautiful new concepts in Variety.
I will definitely look at the other downloaders (and GUIs) as theay are now, to learn their design and play with it.
I would appreciate, if your time permits, if you could share your thoughts on the new design of the image sources.
This way I may try to focus my experiments in the right/same direction.

Ciao!
AndreaS

[Peter Levi (peterlevi)]

Andrea, sorry for the slow reply, I have some pressure at work these weeks. I have noted your question, and will get in touch to discuss the image sources/downloaders as soon as I have some more time.