Launchpad can't sync with the PIDA Trac since it installed the LP plugin

[Graham Binns]

For some reason, checkwatches is OOPSing when trying to set the Launchpad bug ID for bugs on the PIDA Trac:

OOPS-1167CCW1637, OOPS-1167CCW1638, OOPS-1167CCW1639, OOPS-1167CCW1640, OOPS-1167CCW1641, OOPS-1167CCW1642, OOPS-1167CCW1643, OOPS-1167CCW1644

This doesn't appear to be a problem in Launchpad but I can't at present reproduce it in a test instance of Trac.

The traceback from the error is as follows:

Traceback (most recent call last):
  Module canonical.launchpad.scripts.checkwatches, line 668, in updateBugWatches
    self.linkLaunchpadBug(remotesystem, bug_watch)
  Module canonical.launchpad.scripts.checkwatches, line 897, in linkLaunchpadBug
    remotesystem.setLaunchpadBugId(
  Module canonical.launchpad.components.externalbugtracker.trac, line 275, in decorator
    return func(self, *args, **kwargs)
  Module canonical.launchpad.components.externalbugtracker.trac, line 483, in setLaunchpadBugId
    timestamp = self._server.launchpad.set_launchpad_bug(
  Module xmlrpclib, line 1096, in __call__
    return self.__send(self.__name, args)
  Module xmlrpclib, line 1383, in __request
    verbose=self.__verbose
  Module canonical.launchpad.components.externalbugtracker.xmlrpc, line 98, in request
    response = self._parse_response(self.opener.open(request), None)
  Module xmlrpclib, line 1286, in _parse_response
    return u.close()
  Module xmlrpclib, line 744, in close
    raise Fault(**self._stack[0])
Fault: <Fault 2: "'int argument required' while executing 'launchpad.set_launchpad_bug()'">

Which is an error raised by the remote server.

[Graham Binns (gmb)]

This needs further investigation; I'll start on it in the morning.

[Graham Binns (gmb)]

I've added a Trac-Launchpad task for this since that's what appears to be raising the error; I'll leave the Malone status as Incomplete for now until the whys and wherefores are worked out.

[Tim Hatch (timhatch)]

AFAIK that error in Python normally only happens in string formatting, i.e.:

>>> "%d" % "a"
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: int argument required

I can reproduce the error by passing in the bug id as a string, i.e.

>>> server.launchpad.set_launchpad_bug("1", 123)
*** Fault: <Fault 2: "'int argument required' while executing 'launchpad.set_launchpad_bug()'">

Do your OOPSes save locals with types to know if that's the cause?

[Graham Binns (gmb)]

> AFAIK that error in Python normally only happens in string formatting,
> i.e.:
>
>>>> "%d" % "a"
> Traceback (most recent call last):
>  File "<stdin>", line 1, in <module>
> TypeError: int argument required

Agreed.

> I can reproduce the error by passing in the bug id as a string, i.e.
>
>>>> server.launchpad.set_launchpad_bug("1", 123)
> *** Fault: <Fault 2: "'int argument required' while executing 'launchpad.set_launchpad_bug()'">
>

What's odd is that at this point I haven't seen any other instances of this.

> Do your OOPSes save locals with types to know if that's the cause?

Irritatingly, not in the level of detail that's needed

However, what's confusing here is that I'm testing using the same code
that production Launchpad uses for other bug trackers. I haven't seen
any other instance of the error in the log so far, but I'll go back
and check to make sure it's not something common.

[Graham Binns (gmb)]

This has happened before, on the Pidgin Trac. (http://developer.pidgin.im).

What's puzzling me is that the code *is* passing an int to the remote system over XML-RPC; I've checked.

However, I'm just wondering - could this be something to do with the fix for bug 301676 that we issued a while back? IIRC the Pidgin Trac is running that (the latest) version of trac-launchpad, as is this one. The bug trackers running older versions of the plugin don't seem to have this problem.

[Tim Hatch (timhatch)]

My intuition is that it's a db-backend issue, and some searching confirms that MySQLdb has caused that specific exception before - http://mail.python.org/pipermail/python-list/2008-June/666728.html.

I don't see any tickets with launchpad_bug set on PIDA, so I don't think this has ever worked there. Although I setLaunchpadBugId in the traceback, can you confirm that other calls like bug_info do succeed (or have in the past)? I'm setting up my test instance to work on MySQL and will get back to you tonight (PDT).

[Tim Hatch (timhatch)]

I can't reproduce any better with the mysql backend. Seems to not be
the issue. Various validations are happening in the xml-rpc system
itself, such as long ints being passed in so that's not it. When
bugid (that is, the Trac bug) is a string it does produce the issue.

The current behavior is slightly incorrect (talking about tracext/
launchpad/rpc.py:252-254 here), and I see that although it's linking
to the correct bug in the linkified custom field, the one displayed in
the ticket comment is incorrect. I've pushed a fix for this, which
will fix the issue only if Malone(?) is actually passing the Trac bug
id as a string. I'm holding out on making a new release until we know
whether there's some way that's possible (Graham's previous comment
suggests it's not).

If we know the server admins, I'd like to hear whether anything
obvious shows up in the Trac log. There's also a little more I can
check from my end.

[ronny (ronny-pfannschmidt-deactivatedaccount)]

It indeed didn't work before, as it is a new installation.

Feel free to ask about any extra details you need to know on our setup.

Some extra infos i already told on irc:

* our egg cache was corruted, this caused some sporadic exceptions, its fixed now

[Tim Hatch (timhatch)]

I replied last night over email, but I think Launchpad ate my reply...

I can't reproduce any better with the mysql backend. Seems to not be the issue. Various validations are happening in the xml-rpc system itself, such as long ints being passed in so that's not it. When bugid (that is, the Trac bug) is a string it does produce the issue.

The current behavior is slightly incorrect (talking about tracext/launchpad/rpc.py:252-254 here), and I see that although it's linking to the correct bug in the linkified custom field, the one displayed in the ticket comment is incorrect. I've pushed a fix for this, which will fix the issue only if Malone is actually passing the Trac bug id as a string. I'm holding out on making a new release until we know whether there's some way that's possible (Graham's previous comment suggests it's not).

Ronny, can you check the log to see if an exception like this one is showing up? Specifically is it line 252 in the last stack frame?

2009-03-13 21:50:21,894 Trac[web_ui] ERROR: Traceback (most recent call last):
  File "/Users/tim/code/trac/xmlrpcplugin/tracrpc/web_ui.py", line 60, in process_request
  File "/Users/tim/code/trac/xmlrpcplugin/tracrpc/api.py", line 85, in __call__
    return default
  File "/Users/tim/code/trac/trac-launchpad/tracext/launchpad/rpc.py", line 252, in set_launchpad_bug
TypeError: int argument required

[ronny (ronny-pfannschmidt-deactivatedaccount)]

the only errors we have in the log are from the borked egg cache

there are no other errors since that

[Tim Hatch (timhatch)]

I have some time to help in US/Pacific evenings if you have any more info. Can you repro on a test instance, or have any further on the mysql backend suspicions?

[Graham Binns (gmb)]

On Wednesday, 12 January 2011, Tim Hatch <email address hidden> wrote:
> I have some time to help in US/Pacific evenings if you have any more
> info. Can you repro on a test instance, or have any further on the
> mysql backend suspicions?

I haven't looked any further into this for a while. I'll try to find some free time in the next week or so to look at this again.