&amp entity fix

[CoD]

According to all *HTML standards the & character in URLs should be replaced with its corresponding entity &

Dosage can't retrieve these URLs so I did a little change to fetchManyMatches (in util.py) to make it work. With this bugfix dosage supports all HTML compliant websites.

Just change line 23 in util.py from:

page = urlopen(url)

to:

page = urlopen(url.replace('&','&'))

[Seth Bodine (thepilgrim)]

<empty comment>

[Tristan Seligmann (mithrandi)]

I've implemented a version of this fix as a temporary workaround, but we really need to look at dealing with HTML entities properly in some fashion.

[CoD (cod-italy)]

domenica 3 gennaio 2010 ore 17:50 Tristan Seligmann ha scritto:
> I've implemented a version of this fix as a temporary workaround, but
> we really need to look at dealing with HTML entities properly in some
> fashion.

Uhm... I did it too when I created that bug.

Did you check my patch?

claudio

[Tristan Seligmann (mithrandi)]

Sorry, I must have missed something; which patch are you referring to? The only patch attached to this bug report is the trivial one that only handles &amp;

[CoD (cod-italy)]

Oh yeah.. trivial... &amp;... must be that one :-)
It was my first contribution so I didn't go deeper in the code.

The entities we should deal with, imho, are listed in "Reserved
Characters" section of this page:
http://www.w3schools.com/tags/ref_entities.asp

Maybe we could create an external file listing all the replacing rules to
be applied before any other parsing. In this way we could have all the
modules working regardless of entities and we could add more rules later
if we need to.

[Jonathan Jacobs (jjacobs)]

A more complete method has already been committed to the "bunch-of-comics" branch.

[CoD (cod-italy)]

Ok... sorry for being an annoyance ;-)