Comics: add screenscraping (XPath) support as alternative to feeds

[Mark Lee]

I really, really want to get rid of the unmaintained comic applet, but in order to do so with as little complaints as possible, most, if not all, of the comics that are supported by the comic applet need to be supported by Comics! as well. Unfortunately, several of the comics do not have feeds, and require screen scraping in order to retrieve the image.

I propose that in order to keep roughly the same clean structure as the feed config files, the screen scraping configuration files should have a name, URL, and XPath to the <img/> tag containing the comic strip URL.

[Moses Palmér (mosespalmer)]

There is already support for this started.

The class feed.Feed is the abstract base class for feeds. Subclasses
need to implement the parse_file(self, file_name) method, where file
name is a local file name with a cached version of the data at the
specified URL.

Comics! decides which Feed subclass to use by reading the plug-in key of
the feed descriptor file, and uses RSSFeed as a fall-back. To implement
a new feed class, one would create a new module with two functions:
get_class(), which returns the Feed subclass that the module provides,
and matches_url(url), which returns whether the Feed is able to read
comics from a specified url.

When looking through the code, I notice that all parts are implemented,
except for saving the plug-in name in comics_add.py.

At the moment, I do not really have any spare time. If anybody would
like to start with the implementation of the different plug-ins / has a
comment on the current system, I will be able to give guidance / revise.

ons 2009-09-09 klockan 21:50 +0000 skrev Mark Lee:
> Public bug reported:
>
> I really, really want to get rid of the unmaintained comic applet, but
> in order to do so with as little complaints as possible, most, if not
> all, of the comics that are supported by the comic applet need to be
> supported by Comics! as well. Unfortunately, several of the comics do
> not have feeds, and require screen scraping in order to retrieve the
> image.
>
> I propose that in order to keep roughly the same clean structure as the
> feed config files, the screen scraping configuration files should have a
> name, URL, and XPath to the <img/> tag containing the comic strip URL.
>
> ** Affects: awn-extras
> Importance: High
> Assignee: Moses Palmér (mosespalmer)
> Status: New
>
>
> ** Tags: applet comics
>

[Gabor Karsay (gabor-karsay)]

Fix committed in rev. 1494.