Launchpad should batch attachment notification emails

[Marco Rodrigues]

Binary package hint: apport

Hi!

The apport-collect currently add one attachment per comment, and it's a bad idea. Why not just add one comments with all the attachments inside like the apport-gtk does ?

Thanks

---

Proposed solution from one of the duplicate bugs:

* If there are a lot of updates in a short period of time, only send 1 email

[Martin Pitt (pitti)]

I'm just using launchpadlib.addAttachment(); unfortunately there is nothing that I can do about bug mail on the launchpadlib client side. But I agree that Launchpad bug mail for followup comments/attachments should be condensed into one mail, similar to what happens with initial bug filing.

[komputes (komputes)]

I find that Launchpad should wait until all attachments from apport have been sent in before sending an email notification (notifying about all attachments). Currently (in some cases) Launchpad sends one email per apport attachment creating a lot of pollution in the users email inbox.

[Gabe Gorelick (gabegorelick)]

This really makes apport-collect almost unusable for me (apport-collect on a linux bug generate 26 (!) separate comments/attachments, most of which don't get threaded together in gmail). The real solution is obviously for Launchpad to just allow multiple attachments per comment, which solves the problem of multiple attachments also cluttering the bug report. Until that is implemented (see bug 82652), a good solution would be for Launchpad to just merge the emails notifying you of an attachment right after another attachment. This is already done when a comment follows right after a bug status change, for example.

[Graham Binns (gmb)]

The problem is simply that Launchpad treats each new comment as a boundary by which to split up notification emails (so if you make several changes to a bug and add a comment the changes and the comment will all arrive in the same notification, whereas if you comment first and then make the changes you'll get separate emails).

Now, from the difficulty-of-implementation point of view I think it's actually simpler (and more sensible) to support the multiple-attachments-per-comment use case via addAttachment() (and eventually via the UI) than it is to change the way Launchpad handles the notification boundaries. That said, we bundle up notifications for comments imported from remote bugs, though I can't remember off the top of my head how that works. It may be that we can re-use that.

I'm marking this as Triaged, Low on the basis that we should probably add the multiple upload functionality rather than fix this bug.

[Leo (leorolla)]

Indeed they are not threaded in gmail...

e-mails are already condensated when you first file the bug.
e-mails are already condensated when you commend and change the status right away.
Why not going radical and waiting until a 5 minutes are gone?

Whatever a user does waits in a sandbox and after a lapse of time the changes become permanent (like in the wiki pages) and after that just one neat e-mail is sent. Bugs are not evolving that fast so that we couldn't possibly wait for 5 minutes.

[Josh Brown (joshbrown)]

Having a set time to wait before changes are applied seems like a bad idea. Perhaps we could merge the status changer with the add comment box, so that no changes are made until the submit button is pressed.

[Leo (leorolla)]

At least set a time to wait until mails are fired?

[Leo (leorolla)]

Usually there are many satus changes... there can be changes in the title and description as well... together with a comment and even a postscriptum... then add 28 attachements to it. What's the point of sending more than 1 e-mail ?

[Josh Brown (joshbrown)]

I think this bug needs a more ambiguous title considering the duplicate bugs.

[Josh Brown (joshbrown)]

Graham, this bug concerns more than just attachment emails - it is also about changing tags, summary, importance, etc.

[Graham Binns (gmb)]

Josh, the original report was about the problem with notifications generated when apport uploads attachments to Launchpad (the original summary was ambiguous; I'm looking at the description here). That's specifically about attachment notifications.

We're about to start work on a big push to fix a lot of bugs in subscriptions and notifications code. We'll use this bug to track the fact that attachment notifications should be batched. We will doubtless look at batching further, but fixing this specific bug is a nice first step.

[Josh Brown (joshbrown)]

Graham: In that case, should I report a new bug with a more ambiguous summary?

[Leo (leorolla)]

Is it simpler to detect and batch attachments than just replacing on-the-fly notifications by 5-minute-interval digests?

[Graham Binns (gmb)]

On 4 June 2010 20:10, Leo <email address hidden> wrote:
> Is it simpler to detect and batch attachments than just replacing on-
> the-fly notifications by 5-minute-interval digests?
>

At this point, yes.

The bug notification code (which can be found in
lib/canonical/launchpad/mailnotification.py [1] for anyone who wants
to take a look) is pretty ugly and is in sore need of refactoring. As
part of that work we may look at how batching works and whether we
need to do something different, but that's not the first thing on our
list of priorities. This bug is causing a lot of noise, so it's better
to fix this sooner rather than later.

 [1] http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/files/head:/lib/canonical/launchpad/mailnotification.py

[Deryck Hodge (deryck)]

Don't we batch in 5 minute (or some such) intervals now? Except for comments and attachments, of course.

I thought that was the point of this bug, to enable batching for attachments.

[Leo (leorolla)]

Well, when there are many consecutive comments I have the impression that I
get many consecutive e-mails? Is this wrong impression?

The idea is that many comments sent in the same block of time don't result
in many e-mails.

I rarely see something batched. The most common is a status change following
or followed by a comment, which result in only one e-mail for both actions.

[Graham Binns (gmb)]

On 6 June 2010 01:46, Deryck Hodge <email address hidden> wrote:
> Don't we batch in 5 minute (or some such) intervals now?  Except for
> comments and attachments, of course.
>
> I thought that was the point of this bug, to enable batching for
> attachments.

We batch in five minute intervals but comments mark the boundaries
between batches, so if you make a bunch of changes and add a comment
and then someone else makes a bunch of changes you'll get two separate
notifications. Attachments cause problems because they each get their
own comment, so we send out one notification per attachment. Otherwise
our batching is quite sane at the moment.

We *might* be able to move to five-minute digests when we've fixed all
the Launchpad-fakes-email-addresses bugs. The easy fix to some of
those bugs is to start to use <$<email address hidden>> as the From
address for notifications and if we do that we may be able to change
notification emails to look like this:

 From: Bug $FOO <$<email address hidden>>
 ...
 ** Graham Binns (gmb) made the following changes
   * In Ubuntu:
     * status Incomplete -> Confirmed
   * Comment added:
     This new digest thing is the bomb isn't it?

 ** Deryck Hodge (deryck) made the following changes:
   * In Malone
     * status Incomplete -> Fix Released
   * Comment added:
     Yeah, it's not bad.

But I'm not yet convinced that there would be a massive reduction in
bug mail traffic if we did that, or at least not a reduction directly
related to the batching of attachment notifications.

[Deryck Hodge (deryck)]

Thanks for the detailed explanation, Graham. That meshes with my understanding of what we do, too. That is a nice example of the advantage to sending mail from the bug rather than the person, but I think it should be a separate concern and we should consider this bug fixed when we batch attachments from the same person. That's an easy fix and and easy win.

We should open a bug about batching comments themselves, assuming we can find a nice way to do that. I would have thought that bug existed, but I cannot find one. Otherwise, as Graham says, we do this genrally okay now. It's just not obvious to people because comments are common, frequent, and can break batching of other changes in unexpected ways.

Cheers,
deryck

[Graham Binns (gmb)]

Marking this as Triaged again since we won't be coming back to it until after the new subscription UI work is done.

[Gary Poster (gary)]

Graham investigated the status of aggregating attachments in notification emails.

Launchpad already aggregates attachments that are from the same person and for the same bug. Therefore, this bug, as described in its title and in the goals set out by Deryck and Graham, is fixed. I've marked it fix released.

However, we only allow each notification to contain a single comment. Apport includes comments with each attachment. Therefore, we might be able to solve the original complaint that led this bug to be filed by allowing notifications to aggregate comments (in additions to metadata and attachments).

I filed this as bug 768997.

[Brian Murray (brian-murray)]

Gary does this mean that if apport were not to actually comment and only add attachments this would reduce the quantity of emails?

For example in https://bugs.launchpad.net/ubuntu/+source/xserver-xorg-video-intel/+bug/750259/comments/7 - we can see apport adds a message of 'apport information'. If we were to remove that from apport would all the attachments being added be sent in one email?

[Gary Poster (gary)]

@Brian: yes, that's exactly what Graham reported.

[Brian Murray (brian-murray)]

Well the api requires a comment when adding an attachment so it doesn't seem that there is a way to modify apport to prevent multiple email messages.

[Robert Collins (lifeless)]

What if you have an empty comment, or does it reject that? If it
rejects it its probably a bug.

[Graham Binns (gmb)]

On 27 April 2011 19:09, Brian Murray <email address hidden> wrote:
> Well the api requires a comment when adding an attachment so it doesn't
> seem that there is a way to modify apport to prevent multiple email
> messages.

Ah, see, here's the problem:

Internally `comment` can be either a string or an IMessage, which is
how we can theoretically have several attachments linked to one
comment. In the declaration for IBug, we see:

    @operation_parameters(
        data=Bytes(constraint=attachment_size_constraint),
        comment=Text(), filename=TextLine(), is_patch=Bool(),
        content_type=TextLine(), description=Text())
    @export_factory_operation(IBugAttachment, [])
    def addAttachment(owner, data, comment, filename, is_patch=False,
                      content_type=None, description=None):

So, the problem is that the API won't let you pass a message in to
addAttachment(). If we want to change that we should probably track it
elsewhere.

[Robert Collins (lifeless)]

Wouldn't hangling '' as the commentin the same way the 'add
attachment' web form does be sane? Then the web form could be
ajaxified too :)