Librarian token checker doesn't cope with different path encodings

[William Grant]

The librarian TimeLimitedToken matcher compares encoded paths, not the actual (decoded) paths.

This causes problems for filenames containing '~', since LP sends it encoded but browsers like to play with decoding it. For example, Chromium will not send a URL containing '%7E' -- it will decode it to '~' and transmit that. Firefox will display '~' in the address bar but still send '%7E', so the link will work once but not when copied. Other characters may be affected, but common ones like '+' are preserved as '%2B'.

Is there a good reason we don't store the decoded path in the token, and check against that?

[Gary Poster (gary)]

I suspect that there is no good reason, though I don't know the new code.

[Robert Collins (lifeless)]

@wgrant, have you checked that chromium is playing games, not apache?
we previously had issues and needed to add noncanon to the proxypass
rule because apache was canonicalising the urls. Note that testing by
typing in urls by hand is not a particularly useful test, because that
invokes url heuristic code.

zope is generating non canonical urls... and they are getting stored -
fixing that should make things work much better.

Alternatively, as you propose, it should be possible to work with
strings instead of urls, but be sure to put huge XXX comments around
there, because it won't work with unicode - decoding a url only gives
the bytestring, no matter what python 3 things it should give.

I'd be inclined to stay with actual url storage and fix the zope url
generator - we should fix that anyway, and it leaves the security of
the token code easier to reason about.

-Rob