Runaway Launchpad bug dups create "Unsubscribe Me Too!" hell

[Paul Sladen]

If a bug report (eg. bug #429322) has 2000 duplicates then any email message generated in relation to one of the duplicates will send an email to 2000 subscribers.

If an automated tool (eg. apport retracer) is working though those 2000 duplicates then four million emails have been sent and at least one percent of the subscribers have complained, requesting to be "unsubscribed!!!" by replying to the messages from LP, and thus, generated even more email messages sent to 2000 people.

To each of those "unsubscribe me!!!" messages, helpful people have replied, giving instructions for unsubscribing (going to the bug report, to clicking to /+subscribe), but because of bug #471974 all of these instructions result in Launchpad "timeout" pages and no action.

Ideally, something *manually* needs doing (stopping the emails, or unsubscribing people), and *soon*; There have been 923 messages so far and there is likely to be at least another 1100 to come from the apport tracer backlog---of course it's not possible to know the exact numbers as the pages in question are timing out.

[Bagua (bagua1978)]

I'm getting crazy about these emails?
Please unsubscribe me!!!!!!!!!

[Graham Binns (gmb)]

2009/11/13 Deryck Hodge <email address hidden>:
> ** Changed in: malone
>       Status: New => Triaged
>
> ** Changed in: malone
>   Importance: Undecided => High

Probably the simplest way to fix this in the short term, though
unsubtle, would be to nuke the dupe subscriptions. Longer term we
should investigate:

 1. Whether people should be subscribed from dupes
 2. "Ignore" subscriptions, whereby a BugSubscription record is
created but it essentially remains dormant to all updates save those
on its parent bug.

[Paul Sladen (sladen)]

Any ideas?

There have ten (10) apport messages plus two (2) "Stop e-mailing me now!!!!!!!!" new messages arrive from Launchpad since Graham's previous comment.

[Graham Binns (gmb)]

2009/11/14 Paul Sladen <email address hidden>:
> Any ideas?

Yes; I've mailed the internal Launchpad list about a couple of them
and we'll comment here when we chose a solution (also note that it's
EOW for all the LP devs now and not everyone is immediately available
to deal with this).

The ideas I've had so far are:

 1. Nuke the subscriptions (nasty, database muckaboutery).
 2. Make the bug notifications script not send out emails to
subscribers of bugs that are duplicates about changes to the dupe'd
bug.

These are both short term and not particularly pretty. Longer term we
should either have a way to let users opt-in to having their
subscription to a dupe channel notifications about the duplicated bug
or just not send out the notifications to subscribers-from-dupes at
all.

[Deryck Hodge (deryck)]

There's a plan in place to temporarily delete the subscriptions until the
problems can be fixed. Tom and I will do this as quickly as we can, but
it's the weekend and people are out of pocket with sprinting and UDS .

So this isn't going to happen as quickly as it could during the week, but
we'll get to it as soon as possible and fix things so this doesn't happen
again.

On Nov 13, 2009 6:40 PM, "Paul Sladen" <email address hidden> wrote:

Any ideas?

There have ten (10) apport messages plus two (2) "Stop e-mailing me
now!!!!!!!!" new messages arrive from Launchpad since Graham's previous
comment.

-- Runaway Launchpad bug dups create "Unsubscribe Me Too!" hell
https://bugs.launchpad.net/bugs/48...
Status in Launchpad Bugs: Triaged

Bug description: If a bug report (eg. bug #429322) has 2000 duplicates then
any email message gener...

[Paul Sladen (sladen)]

My hunch would for a one-off kludge in the outgoing mail script to silently grep out and drop any outgoing messages containing the string

  "*** This bug is a duplicate of bug 429322 ***"

That would provide breathing-space to investigate tackling the root causes (bug #204980).

[Eleanor Berger (intellectronica)]

As a temporary measure, we've disabled all subscriptions to the bug. Once we have a more complete solution we can restore them and allow users to subscribe themselves using the UI if they wish.

[Paul Sladen (sladen)]

Tom: are you sure? I've received probably 100 more today, most recently 19 minutes ago---that is definitely less than 19 hours ago.

[Paul Sladen (sladen)]

02:26 < intellectronica> sladen: subscriptions to duplicates nuked. this will continue to be a problem as new dupes and subscriptions are created all the time, but i hope it will make things manageable until we come up with a longer-term solution tomorrow

(which has the handy side-effect of allowing bug #429322 not timeout!).

[Paul Sladen (sladen)]

Closing; no more emails have arrived. Bug #204980 and bug #429322 remain.