"No X-Launchpad-Original-To header was present in email" when processing mail

[Ursula Junque]

As seen on OOPS-1838REPORTIFSEEN1393, process-mail is failing to find a launchpad header.

There's no traceback or exception raised, only the following logged as the request variables:
    * <oops-message-0>: No X-Launchpad-Original-To header was present in email: http://launchpadlibrarian.net/62112187/8a7e5b20-1e5d-11e0-8fd4-001e0bc3957e.txt
    * path: /srv/launchpad.net/production/launchpad/cronscripts/process-mail.py
    * script_name: process-mail

[Ursula Junque (ursinha)]

Marked Critical as per zero oops policy.

[Gary Poster (gary)]

Example OOPS is from spam. A reasonable behavior would be to ignore it somehow, if we can come up with a reasonable way to recognize the situation.

The OOPS tool reports 19917 OOPSes like this in the past week. However, the messages vary widely. Here's another OOPS that is in the same group:

<oops-message-87>: Found 2 competing templates with translation domain 'messages': "openerp-messages in OpenObject Web Client trunk"; "viewcalendar-messages in OpenObject Web Client trunk".
path: /srv/staging.launchpad.net/staging/launchpad/cronscripts/rosetta-approve-imports.py
script_name: translations-import-queue-gardener

That appears to be from another script entirely.

Here's another one I found, with the same signature as the one originally reported:

<oops-message-6>: No X-Launchpad-Original-To header was present in email: http://launchpadlibrarian.net/62145638/80d1f7fc-1ebd-11e0-bde1-001e0bc3957e.txt
path: /srv/launchpad.net/production/launchpad/cronscripts/process-mail.py
script_name: process-mail

That's another piece of spam.

We probably should go log trolling to make sure that all of a reasonable sample of the recorded "No X-Launchpad-Original-To header was present in email" errors are spam. If they are, the goal of this bug is to simply make the situation not cause an OOPS. We might also want to check with the LOSAs to make sure that this is not indicative of some spam hole that we need to close. Finally, we should file a bug with the OOPS tool to divide up these OOPSes better

[Gary Poster (gary)]

As of a few hours ago, loganberry's logs on devpad showed 14426 of these since the start of 2011 (via a "wc -l" of the output of "grep -Hn 'No X-Launchpad-Original-To header was present in email:' 2011-01-*/*" on devpad:/srv/launchpad.net-logs/scripts/loganberry). Of the 14426 urls librarian urls, 11754 of them are 404s, and I'm not sure why.

Of the 2974 urls that resolve, the last 2098 are all available consecutively on the librarian; I'm assuming the interspersed 404s early on in the listhave to do with garbage collection.

More and more spam appears in these results. However, they are occasionally interspersed with real messages. Here are some of the most recent legitimate messages, in reverse order:

http://launchpadlibrarian.net/62127365/1735fb4c-1e84-11e0-90d7-001e0bc3957e.txt
http://launchpadlibrarian.net/62020697/8a0d5234-1ce3-11e0-a48a-001e0bc3957e.txt
http://launchpadlibrarian.net/62017096/fc6da084-1cd6-11e0-82e5-001e0bc3957e.txt
http://launchpadlibrarian.net/61929017/ed7c93d6-1b47-11e0-9a77-001e0bc3957e.txt
http://launchpadlibrarian.net/61867640/4cefe132-1a86-11e0-9c67-001e0bc3957e.txt
http://launchpadlibrarian.net/61867218/0a3aa396-1a85-11e0-b658-001e0bc3957e.txt

Those six are the non-spam matches starting sometime 7 January through today (with the most recent, at the top, from 12 January).

The next steps are to see if there's anything that seems to link those six (and maybe the many, many spam messages), and take that to the LOSAs to see if they can use that to identify a problem.

[Gary Poster (gary)]

diagnostic.txt is the guts of what I used to analyze, so I or others can refer back to this for convenience later.

[Gary Poster (gary)]

I sent the raw text of the six good emails above, as well as that of a few spam emails, to RT 43394, asking that they look at the emails to see if they suggest an avenue of getting to Launchpad without receiving the X-Launchpad-Original-To header. Assigning to LOSAs, at least until they come back with a reply asking for more work on our side.

[Gary Poster (gary)]

Per the RT, SAs identified some mail server misconfiguration and addressed it, AIUI.

The last instance of this is two days ago, from Fri, 21 Jan 2011 12:18:05 +0000 (http://launchpadlibrarian.net/62571148/916041d6-2558-11e0-b038-001e0bc3957e.txt). Based on that, the bug might have been squashed, but I'm not entirely sure yet, since that is later than I understand that they fixed the configuration earlier than that time.

[Gary Poster (gary)]

The good news: as I reported in the previous message, the mail server configuration does seem to have been a problem, and as far as we can tell both on the SA side and in the Launchpad logs, *all* emails now have the header inserted.

The bad news: we still do have OOPSes like this. I saw 58 from Friday in the logs, for instance. Brad Marshall noticed that the representative message I sent him was spam, and had malformed headers (namely, an extra newline in the middle of the headers). This is probably confusing our log code, causing us to complain. I confirmed that all 58 have the header, and the ones I looked at manually all had the same problem Brad described.

For this last problem, then, I recommend that, if we don't find the X-Launchpad-Original-To header in the proper place, we look in the body for it. If we find it in the body, we should silently ignore the problem, without an OOPS, perhaps merely logging that we are regarding it as spam.

If we do *not* find it in the body, we should continue to generate an OOPS, because this caught a legitimate configuration error.

[Martin Pool (mbp)]

> For this last problem, then, I recommend that, if we don't find the X-Launchpad-Original-To header in the proper place, we look in the body for it. If we find it in the body, we should silently ignore the problem, without an OOPS, perhaps merely logging that we are regarding it as spam. If we do *not* find it in the body, we should continue to generate an OOPS, because this caught a legitimate configuration error.

Gary, if I understand correctly, all incoming mail will get an x-l-original-to added before it's processed. This should happen for both spam and non-spam messages. If that header is missing, there is definitely a configuration problem.

Having the string "X-Launchpad-Original-To" in the body of the message isn't a reliable indication that the message is spam. It occurs in many of the messages about this bug.

I'm not sure how you can have an extra newline in the middle of rfc2822 headers, and unfortunately all the example files seem to have expired. If spam is being sent in a weird format that confuses our parser maybe we can detect that directly.

hth

[Launchpad QA Bot (lpqabot)]

Fixed in stable r12566 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/12566>.