TypeError in tagging code when viewing a particular profile

[Chris Rossi]

This has been observed, so far, with only a single profile in OSI:

https://karl.soros.org/profiles/Farda

Traceback (most recent call last):
  File "/opt/karl3/src/karl/karl/errorpage.py", line 51, in __call__
    resp = req.get_response(self.app)
  File "/opt/karl3/eggs/WebOb-0.9.8.0_osi1-py2.5.egg/webob/request.py", line 920, in get_response
    application, catch_exc_info=False)
  File "/opt/karl3/eggs/WebOb-0.9.8.0_osi1-py2.5.egg/webob/request.py", line 888, in call_application
    app_iter = application(self.environ, start_response)
  File "/opt/karl3/eggs/repoze.urchin-0.1-py2.5.egg/repoze/urchin/__init__.py", line 20, in __call__
    resp = req.get_response(self.app)
  File "/opt/karl3/eggs/WebOb-0.9.8.0_osi1-py2.5.egg/webob/request.py", line 920, in get_response
    application, catch_exc_info=False)
  File "/opt/karl3/eggs/WebOb-0.9.8.0_osi1-py2.5.egg/webob/request.py", line 888, in call_application
    app_iter = application(self.environ, start_response)
  File "/opt/karl3/eggs/repoze.debug-0.7.1-py2.5.egg/repoze/debug/responselogger.py", line 59, in __call__
    app_iter = self.application(environ, replace_start_response)
  File "/opt/karl3/eggs/repoze.errorlog-0.7-py2.5.egg/repoze/errorlog/__init__.py", line 85, in __call__
    return self.application(environ, start_response)
  File "/opt/karl3/src/karl/karl/errorlog.py", line 15, in middleware
    return app(environ, start_response)
  File "/opt/karl3/eggs/repoze.zodbconn-0.11-py2.5.egg/repoze/zodbconn/connector.py", line 21, in __call__
    result = self.next_app(environ, start_response)
  File "/opt/karl3/eggs/repoze.retry-0.9.4-py2.5.egg/repoze/retry/__init__.py", line 88, in __call__
    app_iter = self.application(environ, replace_start_response)
  File "/opt/karl3/eggs/repoze.tm2-1.0a5-py2.5.egg/repoze/tm/__init__.py", line 23, in __call__
    result = self.application(environ, save_status_and_headers)
  File "/opt/karl3/eggs/Paste-1.7.2-py2.5.egg/paste/registry.py", line 350, in __call__
    app_iter = self.application(environ, start_response)
  File "/opt/karl3/eggs/repoze.who-1.0.15-py2.5.egg/repoze/who/middleware.py", line 107, in __call__
    app_iter = app(environ, wrapper.wrap_start_response)
  File "/opt/karl3/eggs/repoze.browserid-0.3-py2.5.egg/repoze/browserid/middleware.py", line 134, in __call__
    app_iter = self.app(environ, wrapper.wrap_start_response)
  File "/opt/karl3/eggs/repoze.bfg-1.2-py2.5.egg/repoze/bfg/router.py", line 130, in __call__
    response = view_callable(context, request)
  File "/opt/karl3/eggs/repoze.bfg-1.2-py2.5.egg/repoze/bfg/configuration.py", line 1753, in _secured_view
    return view(context, request)
  File "/opt/karl3/src/karl/karl/views/people.py", line 532, in show_profile_view
    tagbox = get_tags_client_data(context, request),
  File "/opt/karl3/src/karl/karl/views/tags.py", line 61, in get_tags_client_data
    records = tagquery.tagswithcounts,
  File "/opt/karl3/src/karl/karl/models/adapters.py", line 203, in tagswithcounts
    tagObjects = self.tags.getTagObjects(items=(self.docid,))
  File "/opt/karl3/src/karl/karl/tagging/__init__.py", line 118, in getTagObjects
    ids = self._getTagIds(items, users, tags, community)
  File "/opt/karl3/src/karl/karl/tagging/__init__.py", line 390, in _getTagIds
    res.update(bt.get(item, set()))
TypeError: expected integer key

[Paul Everitt (paul-agendaless)]

Handing some things over to Tres.

[Tres Seaver (tseaver)]

Somehow that profile had been removed from the catalog's document_map,
which broke an invariant that the tagging code (and lots of other code) relies
on. I have no idea how the profile could get into that state, but the bug would most
likely be in something like the GSA import code.

I did a manual repair in the debugger as follows:

 $ uname -a
 Linux kdiab.sixfeetup.com 2.6.18-92.1.22.el5xen #1 SMP Tue Dec 16 12:26:32 EST 2008 x86_64 x86_64 x86_64 GNU/Linux
 $ sudo /opt/karl3/bin/debug
 ...
 >>> farda = root['profiles']['Farda']
 >>> print root.catalog.document_map.address_for_docid(farda.docid)
 None
 >>> root.catalog.document_map.add('/profiles/Farda', farda.docid)
 >>> import transaction
 >>> transaction.commit()

[Chris Rossi (chris-archimedeanco)]

Related to this, I just noticed the other day that the two catalogs (root.catalog, root['people'].catalog) both end up indexing profiles and can each have different docids for the same profile. I haven't completely traced out yet under what circumstances the docids differ but since both catalogs could potentially generate a docid for a new profile and since both catalogs then store that docid as an attribute of the profile (eg, profile.docid), there seems room here for strange, subtle bugs regarding cataloging of profiles.

[Paul Everitt (paul-agendaless)]

Do we still need a separate catalog for people?

--Paul

On Jul 6, 2010, at 1:21 PM, Chris Rossi wrote:

> Related to this, I just noticed the other day that the two catalogs
> (root.catalog, root['people'].catalog) both end up indexing profiles and
> can each have different docids for the same profile. I haven't
> completely traced out yet under what circumstances the docids differ but
> since both catalogs could potentially generate a docid for a new profile
> and since both catalogs then store that docid as an attribute of the
> profile (eg, profile.docid), there seems room here for strange, subtle
> bugs regarding cataloging of profiles.
>
> --
> TypeError in tagging code when viewing a particular profile
> https://bugs.launchpad.net/bugs/595231
> You received this bug notification because you are subscribed to KARL3.
>
> Status in KARL3: Fix Committed
>
> Bug description:
> This has been observed, so far, with only a single profile in OSI:
>
> https://karl.soros.org/profiles/Farda
>
> Traceback (most recent call last):
> File "/opt/karl3/src/karl/karl/errorpage.py", line 51, in __call__
> resp = req.get_response(self.app)
> File "/opt/karl3/eggs/WebOb-0.9.8.0_osi1-py2.5.egg/webob/request.py", line 920, in get_response
> application, catch_exc_info=False)
> File "/opt/karl3/eggs/WebOb-0.9.8.0_osi1-py2.5.egg/webob/request.py", line 888, in call_application
> app_iter = application(self.environ, start_response)
> File "/opt/karl3/eggs/repoze.urchin-0.1-py2.5.egg/repoze/urchin/__init__.py", line 20, in __call__
> resp = req.get_response(self.app)
> File "/opt/karl3/eggs/WebOb-0.9.8.0_osi1-py2.5.egg/webob/request.py", line 920, in get_response
> application, catch_exc_info=False)
> File "/opt/karl3/eggs/WebOb-0.9.8.0_osi1-py2.5.egg/webob/request.py", line 888, in call_application
> app_iter = application(self.environ, start_response)
> File "/opt/karl3/eggs/repoze.debug-0.7.1-py2.5.egg/repoze/debug/responselogger.py", line 59, in __call__
> app_iter = self.application(environ, replace_start_response)
> File "/opt/karl3/eggs/repoze.errorlog-0.7-py2.5.egg/repoze/errorlog/__init__.py", line 85, in __call__
> return self.application(environ, start_response)
> File "/opt/karl3/src/karl/karl/errorlog.py", line 15, in middleware
> return app(environ, start_response)
> File "/opt/karl3/eggs/repoze.zodbconn-0.11-py2.5.egg/repoze/zodbconn/connector.py", line 21, in __call__
> result = self.next_app(environ, start_response)
> File "/opt/karl3/eggs/repoze.retry-0.9.4-py2.5.egg/repoze/retry/__init__.py", line 88, in __call__
> app_iter = self.application(environ, replace_start_response)
> File "/opt/karl3/eggs/repoze.tm2-1.0a5-py2.5.egg/repoze/tm/__init__.py", line 23, in __call__
> result = self.application(environ, save_status_and_headers)
> File "/opt/karl3/eggs/Paste-1.7.2-py2.5.egg/paste/registry.py", line 350, in __call__
> app_iter = self.application(environ, start_response)
> File "/opt/karl3/eggs/repoze.who-1.0.15-py2.5.egg/repoze/who/middleware.py", line 107, in __call__
> app_iter = app(environ, wrapper.wrap_start_response)
> File "/opt/karl3/eggs/repoze.browserid-0.3-py2.5.egg/re...

[Chris Rossi (chris-archimedeanco)]

Probably not. I think people catalog still has some indexes that aren't in
the main catalog but i can't think of any reason why they couldn't be moved
to the main catalog. It would eliminate some moving parts. We could also
check to see which indexes are actually being used in reports, as that may
have changed when Shane reimplemented.

On Tue, Jul 6, 2010 at 1:36 PM, Paul Everitt <email address hidden> wrote:

>
> Do we still need a separate catalog for people?
>
> --Paul
>
> On Jul 6, 2010, at 1:21 PM, Chris Rossi wrote:
>
> > Related to this, I just noticed the other day that the two catalogs
> > (root.catalog, root['people'].catalog) both end up indexing profiles and
> > can each have different docids for the same profile. I haven't
> > completely traced out yet under what circumstances the docids differ but
> > since both catalogs could potentially generate a docid for a new profile
> > and since both catalogs then store that docid as an attribute of the
> > profile (eg, profile.docid), there seems room here for strange, subtle
> > bugs regarding cataloging of profiles.
> >
> > --
> > TypeError in tagging code when viewing a particular profile
> > https://bugs.launchpad.net/bugs/595231
> > You received this bug notification because you are subscribed to KARL3.
> >
> > Status in KARL3: Fix Committed
> >
> > Bug description:
> > This has been observed, so far, with only a single profile in OSI:
> >
> > https://karl.soros.org/profiles/Farda
> >
> > Traceback (most recent call last):
> > File "/opt/karl3/src/karl/karl/errorpage.py", line 51, in __call__
> > resp = req.get_response(self.app)
> > File "/opt/karl3/eggs/WebOb-0.9.8.0_osi1-py2.5.egg/webob/request.py",
> line 920, in get_response
> > application, catch_exc_info=False)
> > File "/opt/karl3/eggs/WebOb-0.9.8.0_osi1-py2.5.egg/webob/request.py",
> line 888, in call_application
> > app_iter = application(self.environ, start_response)
> > File
> "/opt/karl3/eggs/repoze.urchin-0.1-py2.5.egg/repoze/urchin/__init__.py",
> line 20, in __call__
> > resp = req.get_response(self.app)
> > File "/opt/karl3/eggs/WebOb-0.9.8.0_osi1-py2.5.egg/webob/request.py",
> line 920, in get_response
> > application, catch_exc_info=False)
> > File "/opt/karl3/eggs/WebOb-0.9.8.0_osi1-py2.5.egg/webob/request.py",
> line 888, in call_application
> > app_iter = application(self.environ, start_response)
> > File
> "/opt/karl3/eggs/repoze.debug-0.7.1-py2.5.egg/repoze/debug/responselogger.py",
> line 59, in __call__
> > app_iter = self.application(environ, replace_start_response)
> > File
> "/opt/karl3/eggs/repoze.errorlog-0.7-py2.5.egg/repoze/errorlog/__init__.py",
> line 85, in __call__
> > return self.application(environ, start_response)
> > File "/opt/karl3/src/karl/karl/errorlog.py", line 15, in middleware
> > return app(environ, start_response)
> > File
> "/opt/karl3/eggs/repoze.zodbconn-0.11-py2.5.egg/repoze/zodbconn/connector.py",
> line 21, in __call__
> > result = self.next_app(environ, start_response)
> > File
> "/opt/karl3/eggs/repoze.retry-0.9.4-py2.5.egg/repoze/retry/__init__.py",
> line 88, in __call__
> > app_iter = self.application(environ, replace_start...

[Tres Seaver (tseaver)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Paul Everitt wrote:
> Do we still need a separate catalog for people?

If so, then the code which adds profiles to that catalog (form
controller, cico, whatver) should be passing in the already-generated
docid when adding the path to its document map, or at least ensuring
that the secondary catalog's docid does not replace the one from the
main catalog.

BTW, there are a couple of dozen other profiles which have the same
non-presence in the root catalog::

- ------------------------- %< ------------------------
$ sudo /opt/karl3/bin/debug
Python 2.5.4 (r254:67916, Apr 17 2009, 13:39:45)
[GCC 4.1.2 20071124 (Red Hat 4.1.2-42)] on linux2
Type "help" for more information. "root" is the karl root object.
>>> admap = root.catalog.document_map.address_to_docid
>>> for p_id in root['profiles'].keys():
... if '/profiles/%s' % p_id not in admap:
... print p_id
...
Ahmad Saleem
George Por
Nadja(UEP2006)
Noloyiso Mchunu
Stefi(TASZ)
Ulan Shabynov
ajohnson@asapnys
<email address hidden>
<email address hidden>
chandra kirana
<email address hidden>
<email address hidden>
<email address hidden>
iris alexe
<email address hidden>
<email address hidden>
<email address hidden>
<email address hidden>
<email address hidden>
<email address hidden>
renéwalkens
<email address hidden>
smorrison@asapnys
théodulezilien
<email address hidden>
- ------------------------- %< ------------------------

I can fix these the same way I fixed the one for Farda, but we should be
sure we know that we have removed the root cause.

Tres.
- --
===================================================================
Tres Seaver [phone] <email address hidden>
Palladion Software "Excellence by Design" http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkwzcXwACgkQ+gerLs4ltQ5DzgCfWdGkbGsL5m8JiKOT2/766uL1
IDQAnR927qeoq1nxoJBrc+1ObHxIlmQC
=AGQW
-----END PGP SIGNATURE-----

[Chris Rossi (chris-archimedeanco)]

I think the reason we have two catalogs is because originally, karl.peopledir was an add-on package and so it made sense for it to use its own catalog. Now that peopledir functionality is part of core Karl, the only reasons to maintain a separate catalog would be performance reasons--I'm not aware of any real performance gains keeping them separate, however. If we can agree that there is no reason not to merge the catalogs, that would eliminate some moving parts and make this a somewhat easier problem to solve. All profiles could then be reindexed after the merge, clearing up the above inconsistencies.

[Tres Seaver (tseaver)]

All but two of the broken profiles where created 2009-05-28 (during the
KARL2 -> KARL3 migration).

The two exceptions were both created in October 2009: Paul suspects that
they still pre-date the changeover from the OSI-only karl.peopledir code
to the core stuff.

I plan to fix those profiles in the database, and mark this task as done.
We can reopen the investigation if the issue resurfaces for any newly-
created / updated profiles.

[Paul Everitt (paul-agendaless)]

Got done in M43.

[JimPGlenn (jpglenn09)]

I still get this error on OSI staging for /profiles/Farda

[JimPGlenn (jpglenn09)]

had JimB redownload prod data to staging and is now fixed.

[JimPGlenn (jpglenn09)]

fixed