prune chokes on invalid JSON / invalid cache data

[Alexander List]

Traceback (most recent call last):
File "bin/prune", line 61, in <module>
oopstools.scripts.prune.main()
File "/srv/oops.canonical.com/python-oops-tools/src/oopstools/scripts/prune.py", line 117, in main
options.projectgroup)
File "/srv/oops.canonical.com/python-oops-tools/eggs/oops_datedir_repo-0.0.20-py2.7.egg/oops_datedir_repo/prune.py", line 71, in find_oops_references
lp_project = lp_projects[project]
File "/srv/oops.canonical.com/python-oops-tools/eggs/lazr.restfulclient-0.13.2-py2.7.egg/lazr/restfulclient/resource.py", line 965, in __getitem__
shim_resource._ensure_representation()
File "/srv/oops.canonical.com/python-oops-tools/eggs/lazr.restfulclient-0.13.2-py2.7.egg/lazr/restfulclient/resource.py", line 362, in _ensure_representation
representation = simplejson.loads(unicode(representation))
File "/srv/oops.canonical.com/python-oops-tools/eggs/simplejson-2.2.1-py2.7-linux-x86_64.egg/simplejson/__init__.py", line 413, in loads
return _default_decoder.decode(s)
File "/srv/oops.canonical.com/python-oops-tools/eggs/simplejson-2.2.1-py2.7-linux-x86_64.egg/simplejson/decoder.py", line 405, in decode
raise JSONDecodeError("Extra data", s, end, len(s))
simplejson.decoder.JSONDecodeError: Extra data: line 1 column 2995 - line 1 column 3069 (char 2995 - 3069)

We failed to identify the offending input data, but the prune job shouldn't choke on such data but rather handle it gracefully.

Emptying the cache in /home/oops_tools/.launchpadlib/api.launchpad.net/cache/ helped.

Maybe the cache could be cleaned up / invalidated on a regular basis as well.

[Colin Watson (cjwatson)]

I think this is a lazr.restfulclient bug. If a cached version of a resource is invalid, we should remove that entry from the cache and try fetching a fresh version.