Mailing list stopped working after the rollout today (ImportError)

[Elliot Murphy]

tonight our private mailing list stopped working, I'm attaching a bounce with full headers.

[Barry Warsaw (barry)]

Not only private mailing lists. It looks like all mailing lists are broken.

[Barry Warsaw (barry)]

The failure message gives me no clue as to what was broken, but I think it's upstream of Launchpad. I.e. something in the mail server?

[Steve McInerney (spm)]

Have cowboy reverted part of revno 7200.

Was totally screwing mailman's ability to get the right lib path and hence totally brokenness broke out.

[20/11/08 18:00:53] <elmo> - launchpad_top = os.path.dirname(os.path.dirname(canonical.__file__))
[20/11/08 18:00:57] <elmo> + launchpad_top = os.path.dirname(HERE)
[20/11/08 18:00:57] <elmo> barry: ^-- r7200
[20/11/08 18:01:01] <elmo> i.e. came in with this rollout

Was creating paths in lib/mailman/Mailman/mm_cfg.py
sys.path.insert(0, '/srv/lists.launchpad.net/production/launchpad-rev-7316/lib/canonical/launchpad/mailman')

instead of what should be there of:
sys.path.insert(0, '/srv/lists.launchpad.net/production/launchpad-rev-7316/lib')

[Barry Warsaw (barry)]

Curtis, let's figure out the right fix for this and rc it in if Kiko gives the okay.

[Curtis Hovey (sinzui)]

I wonder if staging would have caught this?

[Curtis Hovey (sinzui)]

I agree that the correct fix is to use revert `launchpad_top`.

How do we test this to verify it can be released. Emails are not getting through to staging mailman. I saw the bounce in the staging inbox. We verified that the site templates worked by running a script that used the staging version mailman to generate the email message. This kind of problem really needs to be tested via an incoming email.

[Barry Warsaw (barry)]

We need to get IS to allow emails to staging mailing lists. I think we would have seen this bug if that test was possible.

[Curtis Hovey (sinzui)]

This break happened because of the fix for bug 268936 [list moderation cancel URLs do not work]

[Curtis Hovey (sinzui)]

This is still critical since the fix was cowboyed into production.

[Curtis Hovey (sinzui)]

The original test for bug 268936 [list moderation cancel URLs do not work] was to send an email to <email address hidden> to veryify we got the correct response. This is the same test we want to run to verify this big is fixed.

What do we have to do to allow incoming emails to <email address hidden>?

[Curtis Hovey (sinzui)]

Fixed in RF 7319.

[Curtis Hovey (sinzui)]

Fix released in 2.1.11. r7322.