No url for <Message at ...> when trying to access bug messages through the API

[Eleanor Berger]

Using launchpadlib, trying:

    list(launchpad.bugs[1].messages)

results in:

HTTPError() HTTP Error 500: Internal Server Error
No url for <Message at 0x7e614d0> because <Message at 0x7e614d0> broke the chain.

Traceback (most recent call last):
  File "/srv/staging.launchpad.net/staging/launchpad/lib/zope/publisher/publish.py", line 133, in publish
    result = publication.callObject(request, obj)
  File "/srv/staging.launchpad.net/staging/launchpad/utilities/../lib/canonical/launchpad/webapp/publication.py", line 331, in callObject
    return mapply(ob, request.getPositionalArguments(), request)
  File "/srv/staging.launchpad.net/staging/launchpad/lib/zope/publisher/publish.py", line 108, in mapply
    return debug_call(obj, args)
  File "/srv/staging.launchpad.net/staging/launchpad/lib/zope/publisher/publish.py", line 114, in debug_call
    return obj(*args)
  File "/srv/staging.launchpad.net/staging/launchpad/utilities/../lib/canonical/lazr/rest/resource.py", line 521, in __call__
    result = self.do_GET()
  File "/srv/staging.launchpad.net/staging/launchpad/utilities/../lib/canonical/lazr/rest/resource.py", line 1006, in do_GET
    return simplejson.dumps(result, cls=ResourceJSONEncoder)
  File "/var/lib/python-support/python2.4/simplejson/__init__.py", line 215, in dumps
    separators=separators, encoding=encoding,
  File "/var/lib/python-support/python2.4/simplejson/encoder.py", line 352, in encode
    chunks = list(self.iterencode(o))
  File "/var/lib/python-support/python2.4/simplejson/encoder.py", line 297, in _iterencode
    for chunk in self._iterencode_dict(o, markers):
  File "/var/lib/python-support/python2.4/simplejson/encoder.py", line 263, in _iterencode_dict
    for chunk in self._iterencode(value, markers):
  File "/var/lib/python-support/python2.4/simplejson/encoder.py", line 294, in _iterencode
    for chunk in self._iterencode_list(o, markers):
  File "/var/lib/python-support/python2.4/simplejson/encoder.py", line 192, in _iterencode_list
    for chunk in self._iterencode(value, markers):
  File "/var/lib/python-support/python2.4/simplejson/encoder.py", line 305, in _iterencode
    for chunk in self._iterencode_default(o, markers):
  File "/var/lib/python-support/python2.4/simplejson/encoder.py", line 311, in _iterencode_default
    newobj = self.default(o)
  File "/srv/staging.launchpad.net/staging/launchpad/utilities/../lib/canonical/lazr/rest/resource.py", line 111, in default
    return IJSONPublishable(obj).toDataForJSON()
  File "/srv/staging.launchpad.net/staging/launchpad/utilities/../lib/canonical/lazr/rest/resource.py", line 614, in toDataForJSON
    repr_name, repr_value = self._unmarshallField(name, field)
  File "/srv/staging.launchpad.net/staging/launchpad/utilities/../lib/canonical/lazr/rest/resource.py", line 758, in _unmarshallField
    repr_value = marshaller.unmarshall(self.entry, value)
  File "/srv/staging.launchpad.net/staging/launchpad/utilities/../lib/canonical/lazr/rest/marshallers.py", line 477, in unmarshall
    repr_value = canonical_url(value)
  File "/srv/staging.launchpad.net/staging/launchpad/utilities/../lib/canonical/launchpad/webapp/publisher.py", line 370, in canonical_url
    urlparts = [urldata.path
  File "/srv/staging.launchpad.net/staging/launchpad/utilities/../lib/canonical/launchpad/webapp/publisher.py", line 325, in canonical_urldata_iterator
    raise NoCanonicalUrl(obj, current_object)
NoCanonicalUrl: No url for <Message at 0x7e614d0> because <Message at 0x7e614d0> broke the chain.

OOPSes related: OOPS-1012EA222, OOPS-1012EB156, OOPS-1012EB157, OOPS-1012EB158, OOPS-1012EB159, OOPS-1012EB160, OOPS-1012EB162, OOPS-1012EC148, OOPS-1012EC149, OOPS-1012ED155

[Eleanor Berger (intellectronica)]

This only seems to affect a small number of bugs (I still don't know what the cause is).

[Eleanor Berger (intellectronica)]

Here are some bugs for which there are messages affected by this problem:

          1
  17216
  22336
  40246
  59695
  65609
  69931
108091
129469
146706

[Eleanor Berger (intellectronica)]

Interestingly, this doesn't happen when I try to access those messages not through the API, which makes it quite a lot harder to debug.

[Eleanor Berger (intellectronica)]

https://bugs.edge.launchpad.net/ubuntu/+source/mobile-basic-flash/+bug/129469/comments/1 is a message that triggers this, for example

[Eleanor Berger (intellectronica)]

It looks like the reason this is happening is that when the canonical_url for the message is looked up, it doesn't appear to have any bugs associated with it.

How that happens is a mystery to me. After all, the reason that message was returned is precisely because it is associated with a bug.

[Eleanor Berger (intellectronica)]

Looking at the DB confirms that there's no connection between the offending messages and any bug, so the remaining question is why they are being returned at all.

[Diogo Matsubara (matsubara)]

Ursula, please keep an eye on the users that generated OOPSes similar to this one. If it's Tom Berger or Markus Korn, it means they're investigating the issue. Bring up new OOPSes if they are caused by other users.

Meanwhile Tom will continue diagnosing this issue.

Thanks.

[Björn Tillenius (bjornt)]

Is this still an issue? The last time I saw an oops like this was October 8.

[Eleanor Berger (intellectronica)]

This is still an issue (I can trigger it reliably) but its scope is very limited. It looks like it's the result of bad data, so it's worth investigating further, cleaning the DB and finding out why it happens.

[Eleanor Berger (intellectronica)]

lowering importance since this is not a very common problem

[Ursula Junque (ursinha)]

I can reproduce it consistently, with bug 297169.
It OOPSes on <message at https://api.staging.launchpad.net/beta/ubuntu/+source/mozvoikko/+bug/297169/comments/15>, all other 21 comments work perfectly.

lp.load('https://api.staging.launchpad.net/beta/ubuntu/+source/mozvoikko/+bug/297169/comments/15') is enough for it to blow as HTTP Error 500: Internal Server Error.

[Eleanor Berger (intellectronica)]

This is now happening much more frequently

[Eleanor Berger (intellectronica)]

~leonardr theorizes that all offending messages are messages that were submitted by email. so far it seems that the data confirms this.

[Leonard Richardson (leonardr)]

Here are some specific messages that have problems from the list of bad bugs Tom posted:

https://bugs.staging.launchpad.net/acpi/+bug/22336/comments/233
https://bugs.edge.launchpad.net/ubuntu/+source/glib2.0/+bug/40246/comments/3
https://bugs.staging.launchpad.net/dell/+bug/59695/comments/168
https://bugs.staging.launchpad.net/ubuntu/+source/firefox/+bug/65609/comments/37
https://bugs.edge.launchpad.net/ubuntu/+source/firefox/+bug/69931/comments/12
https://bugs.edge.launchpad.net/ubuntu/+source/mobile-basic-flash/+bug/129469/comments/1
https://bugs.edge.launchpad.net/ubuntu/+source/xserver-xorg-video-nv/+bug/146706/comments/12
https://bugs.staging.launchpad.net/beta/ubuntu/+source/hplip/+bug/108091/comments/7

All these messages have one thing in common: they were created from email messages. Either they have the form of email messages (with greeting and signoff) or they have quoting and "on [date], [person] wrote...". Not all messages created from emails cause problems, but some do.

[James Westby (james-w)]

On Tue, 2009-01-13 at 17:48 +0000, Leonard D. Richardson wrote:
> Here are some specific messages that have problems from the list of bad
> bugs Tom posted:
>
> https://bugs.staging.launchpad.net/acpi/+bug/22336/comments/233
> https://bugs.edge.launchpad.net/ubuntu/+source/glib2.0/+bug/40246/comments/3
> https://bugs.staging.launchpad.net/dell/+bug/59695/comments/168
> https://bugs.staging.launchpad.net/ubuntu/+source/firefox/+bug/65609/comments/37
> https://bugs.edge.launchpad.net/ubuntu/+source/firefox/+bug/69931/comments/12
> https://bugs.edge.launchpad.net/ubuntu/+source/mobile-basic-flash/+bug/129469/comments/1
> https://bugs.edge.launchpad.net/ubuntu/+source/xserver-xorg-video-nv/+bug/146706/comments/12
> https://bugs.staging.launchpad.net/beta/ubuntu/+source/hplip/+bug/108091/comments/7
>
> All these messages have one thing in common: they were created from
> email messages. Either they have the form of email messages (with
> greeting and signoff) or they have quoting and "on [date], [person]
> wrote...". Not all messages created from emails cause problems, but some
> do.
>

The error message to me suggests some problem with threading. Perhaps
the messages were sent without an "In-Reply-To:" header?

Thanks,

James

[Björn Tillenius (bjornt)]

On Tue, Jan 13, 2009 at 05:48:43PM -0000, Leonard D. Richardson wrote:
> All these messages have one thing in common: they were created from
> email messages.

I found another thing they have in common. They all have a datecreated
(in the DB) that is lacking the decimal part. I.e., usually messages
have a datecreated that looks like this:

    2007-04-04 11:39:32.173281

The messages that break have a datecreated that look like this:

    2007-04-04 12:35:47

Hopefully you'll be able to reproduce this bug locally, if you set a
message's datecreated manually to the latter form.

[Eleanor Berger (intellectronica)]

I can't reproduce this locally by shortening datecreated, so perhaps that's not it.

[Eleanor Berger (intellectronica)]

It seems that the offending messages have a parent which is not one of the messages for the bug. I'm not sure how that might be related.

[Gavin Panella (allenap)]

Although this is the earlier bug, progress has been recorded on bug #394097 so I have marked this one as a duplicate.