Crude duplicate checking causes some URLs to be updated several times

[Vallery Lancey]

Some but very few URLs were showing up as duplicates. This was solved by the implementation of URLs updating their entires if entries already exist. However, the issue needs cleaning up.

I suspect the problem is that a url like /about is checked against the previously downloaded example.com/about, and is seen to be different.