AttributeError: title on Files' get_target_folders

[Paul Everitt]

The error monitor reported:

Thu Oct 27 04:56:14 2011 Exception when processing https://karl.soros.org/communities/clinics-community/files Referer: https://karl.soros.org/communities/clinics-community/wiki/front_page/ Traceback (most recent call last): File "/srv/osfkarl/production/31/eggs/karl-3.72-py2.6.egg/karl/errorlog.py", line 18, in middleware return app(environ, start_response) File "/srv/osfkarl/production/31/eggs/repoze.retry-0.9.4-py2.6.egg/repoze/retry/__init__.py", line 88, in __call__ app_iter = self.application(environ, replace_start_response) File "/srv/osfkarl/production/31/eggs/repoze.zodbconn-0.11-py2.6.egg/repoze/zodbconn/connector.py", line 21, in __call__ result = self.next_app(environ, start_response) File "/srv/osfkarl/production/31/eggs/repoze.tm2-1.0a5-py2.6.egg/repoze/tm/__init__.py", line 23, in __call__ result = self.application(environ, save_status_and_headers) File "/srv/osfkarl/production/31/eggs/repoze.who-1.0.15-py2.6.egg/repoze/who/middleware.py", line 107, in __call__ app_iter = app(environ, wrapper.wrap_start_response) File "/srv/osfkarl/production/31/eggs/repoze.urchin-0.1-py2.6.egg/repoze/urchin/__init__.py", line 20, in __call__ resp = req.get_response(self.app) File "/srv/osfkarl/production/31/eggs/WebOb-0.9.8.0_osi1-py2.6.egg/webob/request.py", line 920, in get_response application, catch_exc_info=False) File "/srv/osfkarl/production/31/eggs/WebOb-0.9.8.0_osi1-py2.6.egg/webob/request.py", line 888, in call_application app_iter = application(self.environ, start_response) File "/srv/osfkarl/production/31/eggs/repoze.bfg-1.2-py2.6.egg/repoze/bfg/router.py", line 130, in __call__ response = view_callable(context, request) File "/srv/osfkarl/production/31/eggs/repoze.bfg-1.2-py2.6.egg/repoze/bfg/configuration.py", line 1753, in _secured_view return view(context, request) File "/srv/osfkarl/production/31/eggs/repoze.bfg-1.2-py2.6.egg/repoze/bfg/configuration.py", line 1704, in _rendered_view response = view(context, request) File "/srv/osfkarl/production/31/eggs/karl-3.72-py2.6.egg/karl/content/views/files.py", line 168, in show_folder_view reverse = True, File "/srv/osfkarl/production/31/eggs/karl-3.72-py2.6.egg/karl/content/views/files.py", line 849, in get_filegrid_client_data target_folders = get_target_folders(context) File "/srv/osfkarl/production/31/eggs/karl-3.72-py2.6.egg/karl/content/views/files.py", line 1018, in get_target_folders ) for docid in docids] AttributeError: 'NoneType' object has no attribute 'title'

[Paul Everitt (paul-agendaless)]

We might need to ask Shane to scrub the data on this or write an evolve. Not sure how in the world we got something assigned without a title.

[Paul Everitt (paul-agendaless)]

Oops, meant "Ask Chris to manually scrub the production or write an evolve."

--Paul

On Oct 27, 2011, at 6:39 AM, Paul Everitt wrote:

> We might need to ask Shane to scrub the data on this or write an evolve.
> Not sure how in the world we got something assigned without a title.
>
> ** Changed in: karl3
> Importance: Undecided => Medium
>
> ** Changed in: karl3
> Milestone: None => m78
>
> ** Changed in: karl3
> Assignee: (unassigned) => Tres Seaver (tseaver)
>
> ** Changed in: karl3
> Milestone: m78 => m79
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/882502
>
> Title:
> AttributeError: title on Files' get_target_folders
>
> Status in KARL3:
> New
>
> Bug description:
> The error monitor reported:
>
> Thu Oct 27 04:56:14 2011 Exception when processing
> https://karl.soros.org/communities/clinics-community/files Referer:
> https://karl.soros.org/communities/clinics-community/wiki/front_page/
> Traceback (most recent call last): File
> "/srv/osfkarl/production/31/eggs/karl-3.72-py2.6.egg/karl/errorlog.py",
> line 18, in middleware return app(environ, start_response) File
> "/srv/osfkarl/production/31/eggs/repoze.retry-0.9.4-py2.6.egg/repoze/retry/__init__.py",
> line 88, in __call__ app_iter = self.application(environ,
> replace_start_response) File
> "/srv/osfkarl/production/31/eggs/repoze.zodbconn-0.11-py2.6.egg/repoze/zodbconn/connector.py",
> line 21, in __call__ result = self.next_app(environ, start_response)
> File
> "/srv/osfkarl/production/31/eggs/repoze.tm2-1.0a5-py2.6.egg/repoze/tm/__init__.py",
> line 23, in __call__ result = self.application(environ,
> save_status_and_headers) File
> "/srv/osfkarl/production/31/eggs/repoze.who-1.0.15-py2.6.egg/repoze/who/middleware.py",
> line 107, in __call__ app_iter = app(environ,
> wrapper.wrap_start_response) File
> "/srv/osfkarl/production/31/eggs/repoze.urchin-0.1-py2.6.egg/repoze/urchin/__init__.py",
> line 20, in __call__ resp = req.get_response(self.app) File
> "/srv/osfkarl/production/31/eggs/WebOb-0.9.8.0_osi1-py2.6.egg/webob/request.py",
> line 920, in get_response application, catch_exc_info=False) File
> "/srv/osfkarl/production/31/eggs/WebOb-0.9.8.0_osi1-py2.6.egg/webob/request.py",
> line 888, in call_application app_iter = application(self.environ,
> start_response) File
> "/srv/osfkarl/production/31/eggs/repoze.bfg-1.2-py2.6.egg/repoze/bfg/router.py",
> line 130, in __call__ response = view_callable(context, request) File
> "/srv/osfkarl/production/31/eggs/repoze.bfg-1.2-py2.6.egg/repoze/bfg/configuration.py",
> line 1753, in _secured_view return view(context, request) File
> "/srv/osfkarl/production/31/eggs/repoze.bfg-1.2-py2.6.egg/repoze/bfg/configuration.py",
> line 1704, in _rendered_view response = view(context, request) File
> "/srv/osfkarl/production/31/eggs/karl-3.72-py2.6.egg/karl/content/views/files.py",
> line 168, in show_folder_view reverse = True, File
> "/srv/osfkarl/production/31/eggs/karl-3.72-py2.6.egg/karl/content/views/files.py",
> line 849, in get_filegrid_client_data target_folders =
> get_target_folders(context) Fil...

[Tres Seaver (tseaver)]

The 'resolver' returned from the catalog search is returning 'None'
for a particular docid, which suggests we have orphan records in the
catalog. Can we manually re-run the karl.evolve.zodb.evolve11 script
and see what it prints out? Otherwise, we need to change the code
in 'karl.content.views.files.get_target_folders' to skip such items.

[Paul Everitt (paul-agendaless)]

I'm going to assign a couple of tickets to Chris for doing the evolve that Tres mentions in comment #3.

[Paul Everitt (paul-agendaless)]

Actually this is for Robert, as it is a release manager step. See comment #3.

[Chris Rossi (chris-archimedeanco)]

I've done some poking around in production and what I've found is that the specific type of corruption detected and fixed by evolve11 isn't happening here. This script looks look at all of the document map entries and makes sure all of those entries actually match up to an object in the database. I just run a variant of that script in production just to check the document map's integrity, without actually fixing anything. What I've found is that there is no problem there.

There must, then, be an index (or indexes) which have entries for documents that are no longer in the system. To check for that we'd need to look at all of the document ids in all of the indexes and see whether they correspond to a document in the system. Although in newer versions of repoze.catalog there is a uniform API for getting all of the docids in an index, we're still using a version of repoze.catalog which lacks that API, so it'll take an hour or two for me to write a script to do this check, probably.

[Robert Marianski (rmarianski)]

If it's faster, we could start by looking at what indexes the query that generates the error uses. But it sounds like it might be just as much, or just a little bit more work maybe, to just walk everything and see where the problem areas are.

[Chris Rossi (chris-archimedeanco)]

I've done some more poking around. While I did find a minor glitch in the text index that I have fixed up manually (I hope), that glitch is not sufficient to explain the exceptions in these tickets. The exceptions in these tickets appear to involve the path index and are probably related to the 'Move to' functionality of folders. Whatever condition led to the exceptions, however, seems to have gone away on its own, making the problem difficult to study. I've gone ahead and removed Balasz's temporary fix to paper over the problem, because now that I have some focus on the problem I'd like to know when it's happening. I've also written a script to check the catalog for internal consistency and am running that via cron every four hours. The results are emailed to me.

This should also let Jim potentially test on branch1 and see if he can manage to get the 'Files' tool to break by doing some file and folder moving operations.

[Paul Everitt (paul-agendaless)]

Jim, comment #8 below is from Chris, asking you to try on branch1 and see if you can break the Move Files feature. Not connected to a release, just some examination.

[Paul Everitt (paul-agendaless)]

Jim, can you test this one today (Friday) and close it out per comment #9?

[JimPGlenn (jpglenn09)]

tested branch1 could not get a failure. I moved, deleted and added files of varying sizes and extensions. All worked.

[Alexander Bittner (yb)]

Hi!

We have seen this one in our error monitor these days, too:

Tue Apr 2 14:41:13 2013 ERROR karl AttributeError: 'NoneType' object has no attribute 'title'
Details
AttributeError: 'NoneType' object has no attribute 'title'
Exception when processing https://karl.oxfam.org.uk/communities/building-livelihoods-and-strengthening-communities/files
Referer: https://karl.oxfam.org.uk/communities/building-livelihoods-and-strengthening-communities/view.html

Traceback (most recent call last):
  File "/srv/multikarl/production/15/eggs/pyramid-1.2.1-py2.6.egg/pyramid/tweens.py", line 17, in excview_tween
    response = handler(request)
  File "/srv/multikarl/production/15/eggs/pyramid_tm-0.5-py2.6.egg/pyramid_tm/__init__.py", line 107, in tm_tween
    return response
  File "/srv/multikarl/production/15/eggs/pyramid_tm-0.5-py2.6.egg/pyramid_tm/__init__.py", line 75, in __exit__
    return self._retry_or_raise(t, v, tb)
  File "/srv/multikarl/production/15/eggs/pyramid_tm-0.5-py2.6.egg/pyramid_tm/__init__.py", line 60, in _retry_or_raise
    reraise(t, v, tb) # otherwise reraise the exception
  File "/srv/multikarl/production/15/eggs/pyramid_tm-0.5-py2.6.egg/pyramid_tm/__init__.py", line 100, in tm_tween
    response = handler(request)
  File "/srv/multikarl/production/15/eggs/pyramid-1.2.1-py2.6.egg/pyramid/router.py", line 153, in handle_request
    response = view_callable(context, request)
  File "/srv/multikarl/production/15/eggs/pyramid-1.2.1-py2.6.egg/pyramid/config/views.py", line 186, in _secured_view
    return view(context, request)
  File "/srv/multikarl/production/15/eggs/pyramid-1.2.1-py2.6.egg/pyramid/config/views.py", line 292, in rendered_view
    result = view(context, request)
  File "/srv/multikarl/production/15/eggs/karl-3.106-py2.6.egg/karl/content/views/files.py", line 178, in show_folder_view
    reverse = True,
  File "/srv/multikarl/production/15/eggs/karl-3.106-py2.6.egg/karl/content/views/files.py", line 969, in get_filegrid_client_data
    target_folders = get_target_folders(context)
  File "/srv/multikarl/production/15/eggs/karl-3.106-py2.6.egg/karl/content/views/files.py", line 1276, in get_target_folders
    title = item.title
AttributeError: 'NoneType' object has no attribute 'title'

It looks quite similar, though the next-to-last line " title = item.title" looks different from your report. Both occurrences were related to the URL shown in the traceback.

Regards,
Alex