Launchpad and mailman are out of sync

[Curtis Hovey]

Lists states (and probably their subscribers) are out of sync.

This list has been waiting to be created for a few hours
    https://edge.launchpad.net/~registry/+mailinglist

These list have been waiting for several days to be created
    https://launchpad.net/~meadl-devel/+mailinglist
    https://launchpad.net/~tagin-devel/+mailinglist

mbarnett reported that he saw a list waiting to be deactivated.

emails are being delivered, so this looks like a communication issue between lp and mailman. xmlrpc is ise to talk, so mm may not be polling, ot xmlrpc is not providing a list of actions to do, or lp is not providing the actions to do.

[Michael Barnett (mbarnett)]

The mailing list for ~ubuntu-br.am is still "waiting to be deactivated" on lp.

-Michael

[Curtis Hovey (sinzui)]

List creation works fine on staging. Its code is the same as lpnet, but the data two weeks old.

[Curtis Hovey (sinzui)]

I found a questions from last week that verifies my suspicion that list memberships are not in sync:
    ayatana list is not getting new members.

[Nicolas Delvaux (malizor)]

Same problem here:
The "lp-l10n-fr-community" list "will be available within a few minutes" and the "lp-l10n-fr" one "is being deactivated"... since more than 2 days.

[Curtis Hovey (sinzui)]

I think the root cause of this bug is bug #576212. We may have a user with a name that blows the xmlrpc server to smithereens.

[Curtis Hovey (sinzui)]

https://pastebin.canonical.com/32109/ indicates that syncing broke on Wednesday.

[Curtis Hovey (sinzui)]

The leading candiates for the one TB I have seen implies
    proxy = xmlrpclib.ServerProxy(mm_cfg.XMLRPC_URL)
    ^ the config is wrong and we are not really talking to a server

There could be routing or port issue too.

[Curtis Hovey (sinzui)]

I think an admin can verify that mailman's instance of lib/mailman/Mailman/mm_cfg.py points the the real xmlrpc server. It would also be good to verify that requests can be made to the xmlrpc server on the correct port.

[Curtis Hovey (sinzui)]

A restarting mailman:
May 10 22:00:59 2010 mailmanctl(23937): No child with pid: 30085
May 10 22:00:59 2010 mailmanctl(23937): [Errno 3] No such process
May 10 22:00:59 2010 mailmanctl(23937): Stale pid file removed.
May 10 22:01:11 2010 mailmanctl(24050): PID unreadable in: /srv/lists.launchpad.net/var/mailman/data/master-qrunner.pid
May 10 22:01:11 2010 mailmanctl(24050): [Errno 2] No such file or directory: '/srv/lists.launchpad.net/var/mailman/data/master-qrunner.pid'
May 10 22:01:11 2010 mailmanctl(24050): Is qrunner even running?
May 10 22:01:19 2010 qrunner(24069): Cannot import runner module Mailman.Queue.XMLRPCRunner
May 10 22:01:19 2010 qrunner(24069): No module named psycopg2

^ Note that Mailman.Queue.XMLRPCRunner *is* mailman lp integration. This implies syncing cannot be done.

[Barry Warsaw (barry)]

This looks like a build problem. Certainly an import error would cause XMLRPC runner process to exit.

[Curtis Hovey (sinzui)]

The cowboy patch made after the last rollout broken the mail env; mailman could not import psychopg2. An alternate patch was applied and mailman was restarted. The all the lists waiting to be created and users to be added were synced.