(OXFAM) error in processing atom feed for community

[Jim B. Glenn]

reported via error monitor:

ENTRY
Sun Oct 24 21:29:02 2010
Exception when processing https://karl.oxfam.org.uk/communities/dfid-funding/atom.xml
Traceback (most recent call last):
  File "/opt/karl/oxfam/3.5.4-0/src/karl/karl/errorlog.py", line 15, in middleware
    return app(environ, start_response)
  File "/opt/karl/oxfam/3.5.4-0/eggs/repoze.zodbconn-0.11-py2.5.egg/repoze/zodbconn/connector.py", line 21, in __call__
    result = self.next_app(environ, start_response)
  File "/opt/karl/oxfam/3.5.4-0/eggs/repoze.retry-0.9.4-py2.5.egg/repoze/retry/__init__.py", line 88, in __call__
    app_iter = self.application(environ, replace_start_response)
  File "/opt/karl/oxfam/3.5.4-0/eggs/repoze.tm2-1.0a5-py2.5.egg/repoze/tm/__init__.py", line 23, in __call__
    result = self.application(environ, save_status_and_headers)
  File "/opt/karl/oxfam/3.5.4-0/eggs/Paste-1.7.2-py2.5.egg/paste/registry.py", line 350, in __call__
    app_iter = self.application(environ, start_response)
  File "/opt/karl/oxfam/3.5.4-0/eggs/repoze.who-1.0.15-py2.5.egg/repoze/who/middleware.py", line 107, in __call__
    app_iter = app(environ, wrapper.wrap_start_response)
  File "/opt/karl/oxfam/3.5.4-0/eggs/repoze.browserid-0.3-py2.5.egg/repoze/browserid/middleware.py", line 127, in __call__
    return self.app(environ, start_response)
  File "/opt/karl/oxfam/3.5.4-0/eggs/repoze.bfg-1.2-py2.5.egg/repoze/bfg/router.py", line 130, in __call__
    response = view_callable(context, request)
  File "/opt/karl/oxfam/3.5.4-0/eggs/repoze.bfg-1.2-py2.5.egg/repoze/bfg/configuration.py", line 1753, in _secured_view
    return view(context, request)
  File "/opt/karl/oxfam/3.5.4-0/src/karl/karl/views/atom.py", line 195, in community_atom_view
    return CommunityAtomFeed(context, request)()
  File "/opt/karl/oxfam/3.5.4-0/src/karl/karl/views/atom.py", line 73, in __call__
    xml = render_template(self._template, view=self)
  File "/opt/karl/oxfam/3.5.4-0/eggs/repoze.bfg-1.2-py2.5.egg/repoze/bfg/chameleon_zpt.py", line 73, in render_template
    return renderer(kw, {})
  File "/opt/karl/oxfam/3.5.4-0/eggs/repoze.bfg-1.2-py2.5.egg/repoze/bfg/chameleon_zpt.py", line 46, in __call__
    result = self.template(**system)
  File "/opt/karl/oxfam/3.5.4-0/eggs/Chameleon-1.1.1-py2.5.egg/chameleon/core/template.py", line 83, in __call__
    return self.render(**kwargs)
  File "/opt/karl/oxfam/3.5.4-0/eggs/Chameleon-1.1.1-py2.5.egg/chameleon/core/template.py", line 335, in render
    return super(TemplateFile, self).render(**kwargs)
  File "/opt/karl/oxfam/3.5.4-0/eggs/Chameleon-1.1.1-py2.5.egg/chameleon/zpt/template.py", line 18, in render
    return super(PageTemplate, self).render(**kwargs)
  File "/opt/karl/oxfam/3.5.4-0/eggs/Chameleon-1.1.1-py2.5.egg/chameleon/core/template.py", line 188, in render
    return self.cook_and_render(kwargs, utils.emptydict, None, True)
  File "/opt/karl/oxfam/3.5.4-0/eggs/Chameleon-1.1.1-py2.5.egg/chameleon/core/template.py", line 315, in cook_and_render
    args, slots, macro, global_scope)
  File "/opt/karl/oxfam/3.5.4-0/eggs/Chameleon-1.1.1-py2.5.egg/chameleon/core/template.py", line 180, in cook_and_render
    return func(econtext, rcontext)
  File "<string>", line 238, in render
  File "/opt/karl/oxfam/3.5.4-0/eggs/Chameleon-1.1.1-py2.5.egg/chameleon/core/codegen.py", line 29, in lookup_attr
    return getattr(obj, key)
  File "/opt/karl/oxfam/3.5.4-0/src/karl/karl/views/atom.py", line 148, in author
    profile = profiles[self.context.creator]
  File "/opt/karl/oxfam/3.5.4-0/eggs/repoze.folder-0.4-py2.5.egg/repoze/folder/__init__.py", line 89, in __getitem__
    return self.data[name]
KeyError: u'mschmidt'

[Paul Everitt (paul-agendaless)]

Over to Tres for triage. Another KeyError on profile.

[Tres Seaver (tseaver)]

As with lp:666291, we have content still associated with a delete user.

Jim, can you please run the following to find which missing users are still
'creator' or 'modified_by'?::

  $ bin/debug
  >>> missing = set()
  >>> profiles = root['profiles']
  >>> catalog = root.catalog
  >>> for index in (catalog['creator'], catalog['modified']):
  >>> for key in index._fwd_index.keys():
  >>> if profiles.get(key) is None:
  ... missing.add(key)
  >>> for userid in sorted(missing):
  ... print userid

Please paste the output in here.

To do a fixup, we need to know what profile ID should be the new
value for 'creator' / 'modified_by'. There is some code in
'karl.utilities.rename_user.rename_user' which could be broken out
to do this, but we should figure out whether this problem is worth
the effort first (for instance, we might decide to remove the content
created by the missing user instead).

[Jim B. Glenn (jimbglenn)]

ran against OXFAM STAGING, check me:

[oxfam@karlhost01 current]$ bin/debug
Python 2.5.5 (r255:77872, Jun 7 2010, 14:10:57)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-46)] on linux2
Type "help" for more information. "root" is the karl root object.
>>> missing = set()
>>> profiles = root['profiles']
>>> catalog = root.catalog
>>> for index in (catalog['creator'], catalog['modified']):
... for key in index._fwd_index.keys():
... if profiles.get(key) is None:
... missing.add(key)
...
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/opt/python2.5//lib/python2.5/UserDict.py", line 22, in __getitem__
    raise KeyError(key)
KeyError: 'modified'

[Tres Seaver (tseaver)]

Sorry, that should be 'modified_by' instead of 'modified'.b

[Jim B. Glenn (jimbglenn)]

ran on STAGING oxfam, check me:
[oxfam@karlhost01 current]$ bin/debug
Python 2.5.5 (r255:77872, Jun 7 2010, 14:10:57)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-46)] on linux2
Type "help" for more information. "root" is the karl root object.
>>> missing = set()
>>> profiles = root['profiles']
>>> catalog = root.catalog
>>> for index in (catalog['creator'], catalog['modified_by']):
... for key in index._fwd_index.keys():
... if profiles.get(key) is None:
... missing.add(key)
...
>>> for userid in sorted(missing):
... print userid
...
None
kuhlermckeown
>>>

[Tres Seaver (tseaver)]

OK, we need to get somebody at Oxfam to decide what to do with the content
which belongs to the now-missing user, 'kuhlermckeown'::

  >>> from karl.models.interfaces import ICatalogSearch
  >>> s = ICatalogSearch(root)
  >>> count, docids, resolver = s(creator='kuhlermckeown')
  >>> for doc in [resolver(x) for x in docids]:
  ... print model_path(doc)

Iim, please post the results there and assign to whoever is the
"KARL expert" at Oxfam to make a call about the content in the results.
Once they give us a decision, assign the issue back to me for implementation.

[Jim B. Glenn (jimbglenn)]

[oxfam@karlhost01 current]$ bin/debug
Python 2.5.5 (r255:77872, Jun 7 2010, 14:10:57)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-46)] on linux2
Type "help" for more information. "root" is the karl root object.
>>> from karl.models.interfaces import ICatalogSearch
>>> s = ICatalogSearch(root)
>>> count, docids, resolver = s(creator='kuhlermckeown')
>>> for doc in [resolver(x) for x in docids]:
... print model_path(doc)
...
Traceback (most recent call last):
  File "<console>", line 2, in <module>
NameError: name 'model_path' is not defined
>>>

[Tres Seaver (tseaver)]

 >>> from repoze.bfg.traversal import model_path

[Jim B. Glenn (jimbglenn)]

[oxfam@karlhost01 current]$ bin/debug
Python 2.5.5 (r255:77872, Jun 7 2010, 14:10:57)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-46)] on linux2
Type "help" for more information. "root" is the karl root object.
>>> from karl.models.interfaces import ICatalogSearch
>>> from repoze.bfg.traversal import model_path
>>> s = ICatalogSearch(root)
>>> count, docids, resolver = s(creator='kuhlermckeown')
>>> for doc in [resolver(x) for x in docids]:
... print model_path(doc)
...
/communities/working-group-for-bmgf-loi-hiv-advocacy/files/loi-templates
/communities/working-group-for-bmgf-loi-hiv-advocacy/files/loi-templates/gh_loi_application_form.doc
/communities/working-group-for-bmgf-loi-hiv-advocacy/files/loi-templates/gh_loi_instructions.pdf
/communities/working-group-for-bmgf-loi-hiv-advocacy/files/topic-ideas-jacob-14-oct-09.doc
/communities/working-group-for-bmgf-loi-hiv-advocacy/files/background-reading
/communities/working-group-for-bmgf-loi-hiv-advocacy/files/background-reading/oi-hiv-aids-working-group-tor.pdf
/communities/working-group-for-bmgf-loi-hiv-advocacy/files/background-reading/gcol-newsletter-nov08.pdf
/communities/working-group-for-bmgf-loi-hiv-advocacy/files/meeting-notes
/communities/working-group-for-bmgf-loi-hiv-advocacy/files/meeting-notes/notes-from-group-telecon-2-november-09.doc
/communities/working-group-for-bmgf-loi-hiv-advocacy/files/background-reading/overview-ogb-gcol-oi-hiv-group-for-fundraisers.doc
/communities/working-group-for-bmgf-loi-hiv-advocacy/files/meeting-notes/update-to-fundraisers-13-11-09-ku.doc
/communities/working-group-for-bmgf-loi-hiv-advocacy/files/meeting-notes/update-to-group-13-11-09-ku.doc
/communities/working-group-for-bmgf-loi-hiv-advocacy/blog/test
/communities/working-group-for-bmgf-loi-hiv-advocacy/calendar/telecon-1-exploratory-discussion
/communities/us-foundation-donors/blog/what-would-you-like-to-know-more-about-us-foundations
/communities/working-group-for-bmgf-loi-hiv-advocacy
/communities/working-group-for-bmgf-loi-hiv-advocacy/calendar/concept-note-due
/communities/working-group-for-bmgf-loi-hiv-advocacy/files/gates-fdn-contact-report-13nov10.docx
/communities/working-group-for-bmgf-loi-hiv-advocacy/blog/test/comments/001
/communities/working-group-for-bmgf-loi-hiv-advocacy/wiki/front_page
/communities/working-group-for-bmgf-loi-hiv-advocacy/files/concept-note-template
/communities/working-group-for-bmgf-loi-hiv-advocacy/files/concept-note-template/gatesfoundation_conceptnotetemplate.doc
/communities/working-group-for-bmgf-loi-hiv-advocacy/files/meeting-notes/update-to-group-18-11-09-ku.doc
/communities/us-foundation-donors/blog/comments-about-this-community
/communities/us-foundation-donors/blog/what-would-you-like-to-know-more-about-us-foundations/comments/002

[Jim B. Glenn (jimbglenn)]

e-mailed Andy (OXFAM)
for new userid

[Jim B. Glenn (jimbglenn)]

From customer:

Hi Andy

This is Kristen Uhler-McKeown https://karl.oxfam.org.uk/profiles/kuhler-mckeown/ -- I think she was one of the people who ended up with two KARL accounts, and looks like the other one was deleted rather than deactivated (or something).

So this content can all be reassigned to her: kuhler-mckeown

Please thank Jim for picking this up!

cheers

jo

[Tres Seaver (tseaver)]

OK, this fix should be applied in production:

  $ sudo bin/debug
  >>> from karl.utilities.rename_user import re

[Tres Seaver (tseaver)]

Crap! That should be::

  >>> from kar.utilities.rename_user import rename_user
  >>> root['profiles']['kuhlermckeown'] = object()
  >>> rename_user(root, 'kuhlermckeown', 'kuhler-mckeown', merge=True)
  >>> assert 'kuhlermckeown' not in root['profiles']
  >>> import transaction; transaction.commit()

[Jim B. Glenn (jimbglenn)]

ran on staging to try it:

[oxfam@karlhost01 current]$ bin/debug
Python 2.5.5 (r255:77872, Jun 7 2010, 14:10:57)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-46)] on linux2
Type "help" for more information. "root" is the karl root object.
>>> from karl.utilities.rename_user import rename_user
>>> root['profiles']['kuhlermckeown'] = object()
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/opt/karlstaging/karl/oxfam/3.5.4-3/eggs/repoze.folder-0.4-py2.5.egg/repoze/folder/__init__.py", line 141, in __setitem__
    return self.add(name, other)
  File "/opt/karlstaging/karl/oxfam/3.5.4-3/eggs/repoze.folder-0.4-py2.5.egg/repoze/folder/__init__.py", line 165, in add
    other.__parent__ = self
AttributeError: 'object' object has no attribute '__parent__'
>>>

[Tres Seaver (tseaver)]

Hmm, OK::

  >>> from repoze.folder import Folder
  >>> from kar.utilities.rename_user import rename_user
  >>> root['profiles']['kuhlermckeown'] = Folder()
  >>> rename_user(root, 'kuhlermckeown', 'kuhler-mckeown', merge=True)
  >>> assert 'kuhlermckeown' not in root['profiles']
  >>> import transaction; transaction.commit()

[Jim B. Glenn (jimbglenn)]

ran against staging:

[oxfam@karlhost01 current]$ bin/debug
Python 2.5.5 (r255:77872, Jun 7 2010, 14:10:57)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-46)] on linux2
Type "help" for more information. "root" is the karl root object.
>>> from repoze.folder import Folder
>>> from karl.utilities.rename_user import rename_user
>>> root['profiles']['kuhlermckeown'] = Folder()
>>> rename_user(root, 'kuhlermckeown', 'kuhler-mckeown', merge=True)
>>> assert 'kuhlermckeown' not in root['profiles']
>>> import transaction; transaction.commit()
>>> quit()

ready for prod?

[Tres Seaver (tseaver)]

Yes.

[Paul Everitt (paul-agendaless)]

Jim, is this one that you can close?

[Jim B. Glenn (jimbglenn)]

Tried this on prod, got:
[oxfam@karlhost01 current]$ bin/debug
Python 2.5.5 (r255:77872, Jun 7 2010, 14:10:57)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-46)] on linux2
Type "help" for more information. "root" is the karl root object.
>>> from repoze.folder import Folder
>>> from karl.utilities.rename_user import rename_user
>>> root['profiles']['kuhlermckeown'] = Folder()
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/opt/karl/oxfam/3.5.4-2/eggs/repoze.folder-0.6.2-py2.5.egg/repoze/folder/__init__.py", line 113, in __setitem__
    return self.add(name, other)
  File "/opt/karl/oxfam/3.5.4-2/eggs/repoze.folder-0.6.2-py2.5.egg/repoze/folder/__init__.py", line 127, in add
    raise KeyError('An object named %s already exists' % name)
KeyError: u'An object named kuhlermckeown already exists'
>>> exit()

[Jim B. Glenn (jimbglenn)]

is this something we can use the "merge accounts" tool under Admin tab for?

[Chris Rossi (chris-archimedeanco)]

The "merge accounts" tell in the Admin UI is a front end for the rename_users function in the code. Since there is UI available for this task, it would be better/safer to use that rather than try to use bin/debug.

[Tres Seaver (tseaver)]

Jim, please try using the "merge accounts" UI (see Rossi's note)
to clean up this data.

[Paul Everitt (paul-agendaless)]

Jim, got a status on this one? If it's no longer an issue, please close it.

[Paul Everitt (paul-agendaless)]

Ok, I give, closing.