digest message improperly handles boundary

[Arik]

When I generate a digest message using MIME encapsulation (outside of VM) I notice that VM seems to require an extra line new line in order to properly display the "from" and "subject" headers (both in summary and presentation) of digest sub-messages. In fact I think that it is seeking past the header block entirely. RFC2046 shows a digest part having a newline before the headers, but this doesn't appear to be required (?).

I have attached an example digest, created from an NPR news feed. I have taken the liberty of inserting the VM-required new-line for the *first message only*. Visiting the digest temporary folder one will notice a normal first message and then subsequent messages with subject "" and sender VM (some sort of default).

I will look into this some more, unless I am just wrong about the standard...

[Arik (akwm)]

This works in opposite too. When I forward more than one message, VM does a very nice thing and makes a digest of the marked messages. The problem is, it inserts a new line after the border of each digest part:

--56HD+tjpq5
Content-Type: multipart/digest; boundary="coV+5jrhX5"
Content-Transfer-Encoding: 7bit

This is a digest, 2 messages, MIME encapsulation.

--coV+5jrhX5

MIME-Version: 1.0
Content-Type: multipart/alternative;
 boundary="----=_NextPart_000_0017_01CB3494.494390C0"

If the message is encoded in non-ascii (iso-2022-jp in this case) then another client (gmail here) might not know that and show garbled text, if not then it will simply show the headers (sort of unpleasent).

[Arik (akwm)]

I just noticed that VM also used this code to display MIME encapsulated forwarded messages, this resulted in an infinite loop of forwarded message buttons since the headers were included and had the forwarded message content description. I offer this patch to handle the issue, but it may need to be looked into some more. In fact, I'm not even sure the headers need to be inserted for the vm-mime-types-match = "message", but rather only for vm-mime-types-match "multipart/digest"

[Uday Reddy (reddyuday)]

I cant reproduce this infinite loop. What exactly do I need to do to get it?

Can you add the fix to the original branch, especially because this may not be the last of it?

Cheers,
Uday

[Arik (akwm)]

 > I cant reproduce this infinite loop. What exactly do I need to do to
 > get it?

Forward yourself a message (only one) via VM (VM uses MIME headers to
indicate message is forwarded and includes all original headers). When
you select that message there should be a button:

 forwarded message [Press RETURN to display message]

Which will open a virtual folder containing the forwarded message. That
message will be the same button, since the mime headers have:

 Content-Description: forwarded message

and are included. Let me know if you do not see this.

 >
 > Can you add the fix to the original branch, especially because this may
 > not be the last of it?

Okay.

Thanks,
~Arik

[Uday Reddy (reddyuday)]

Under the default settings, the forwarded message is simply displayed. There is no button.

If I add "message" to vm-mime-internal-type-exceptions, then I get a button that says "save to file" rather than "display message".

If I hit D enough to make it display buttons only, then the button says "display message". I don't know why it does different things at different times. There seems to be inconsistent coding here.

Anyway, once I click the "display message" button, I get a digest view, but the actual message in this view is displayed as a button!

So, I am running into various bugs, but not an infinite loop.

[Arik (akwm)]

Interesting. I had the following:

(setq vm-auto-displayed-mime-content-types '("text/plain" "multipart"))
(setq vm-mime-internal-content-type-exceptions '("text/html"))

and was getting infinite recursion on trying to display. Adding
"message" to the content types just simply displays it. I think I like
that better. I posted the patch in the old branch "mime-digest-fix",
we'll have to figure out what other cases use this code, since it's
not only digest type!

~Arik

[Uday Reddy (reddyuday)]

Ok, I looked at what is happening.

According to the RFC 2046, sec.5.1, para 2:

Each part of a multipart/X is supposed to have a bunch of MIME headers, followed by a blank line, and then the body of the MIME part.

For multipart/digest, the body is in turn a message, which should have message headers, followed by a blank line, and then the body of the message.

The NPR digest that you have attached is not satisfying these criteria. The blank line following the MIME headers is missing. It is ok for the MIME headers to be missing. VM applies the defaults then. But the blank line should not be missing.

VM code has been fine. It is unable to parse the NPR digest because it is invalid. I am afraid we have to revert all your fixes.

If we want to allow this kind of thing, then the proper thing to do would be to alter the MIME parsing part of VM to allow for such deviations. If you can find some clear criteria whereby VM can guess that the sender is sending bad MIME, then we should discuss them and change VM accordingly.

[Arik (akwm)]

Hi Uday,

I don't read the rfc the same way. I see that the boundary line is
proceeded by a "body part" (no mention of required new line) which is
in turn a message. This message can have headers but it is permitted
not to as well, but I don't read it as having two sets of headers
(MIME and message headers). It says a body part that starts with a
blank line is allowed but it looks to me like this should then be
interpreted as having all default values [for header values].

Thanks,
~Arik

Uday Reddy writes:
 > Ok, I looked at what is happening.
 >
 > According to the RFC 2046, sec.5.1, para 2:
 >
 > Each part of a multipart/X is supposed to have a bunch of MIME headers,
 > followed by a blank line, and then the body of the MIME part.
 >
 > For multipart/digest, the body is in turn a message, which should have
 > message headers, followed by a blank line, and then the body of the
 > message.
 >
 > The NPR digest that you have attached is not satisfying these criteria.
 > The blank line following the MIME headers is missing. It is ok for the
 > MIME headers to be missing. VM applies the defaults then. But the
 > blank line should not be missing.
 >
 > VM code has been fine. It is unable to parse the NPR digest because it
 > is invalid. I am afraid we have to revert all your fixes.
 >
 > If we want to allow this kind of thing, then the proper thing to do
 > would be to alter the MIME parsing part of VM to allow for such
 > deviations. If you can find some clear criteria whereby VM can guess
 > that the sender is sending bad MIME, then we should discuss them and
 > change VM accordingly.
 >
 >
 >
 > ** Changed in: vm
 > Milestone: 8.1.93a => None
 >
 > --
 > digest message improperly handles boundary
 > https://bugs.launchpad.net/bugs/612111
 > You received this bug notification because you are a direct subscriber
 > of the bug.
 >
 > Status in VM (View Mail) for Emacs: In Progress
 >
 > Bug description:
 > When I generate a digest message using MIME encapsulation (outside of VM) I notice that VM seems to require an extra line new line in order to properly display the "from" and "subject" headers (both in summary and presentation) of digest sub-messages. In fact I think that it is seeking past the header block entirely. RFC2046 shows a digest part having a newline before the headers, but this doesn't appear to be required (?).
 >
 > I have attached an example digest, created from an NPR news feed. I have taken the liberty of inserting the VM-required new-line for the *first message only*. Visiting the digest temporary folder one will notice a normal first message and then subsequent messages with subject "" and sender VM (some sort of default).
 >
 > I will look into this some more, unless I am just wrong about the standard...
 >
 > To unsubscribe from this bug, go to:
 > https://bugs.launchpad.net/vm/+bug/612111/+subscribe

--
Arik W. Mitschang

[Uday Reddy (reddyuday)]

Arik writes:

> Hi Uday,
>
> I don't read the rfc the same way. I see that the boundary line is
> proceeded by a "body part" (no mention of required new line) which is
> in turn a message.

The part we are concerned with section 5.1:

    After its boundary delimiter line, each body part then consists of
    a header area, a blank line, and a body area. Thus a body part is
    similar to an <RFC 822 message> in syntax, but different in meaning.

"body part" here is each part of a multipart. Note the three
components of the body part. "header area" and "body area" could
possibly be empty, but not the "blank line".

The reference to "RFC 822 message" is unfortunate. It is being used
as an analogy for the whole "body part". Since the "body area" could
in turn be a real RFC 822 message, this can be confusing. So, I have
put the RFC 822 message in angle brackets. You can complete ignore
what is inside the angle brackets. It doesn't affect anything.

    A body part is an entity and hence is NOT to be interpreted as
    actually being an <RFC 822 message>. To begin with, NO header fields
    are actually required in body parts. A body part that starts with
    a blank line, therefore, is allowed and is a body part for which
    all default values are to be assumed. In such a case, the absence
    of a Content-Type header usually indicates that the corresponding
    body has a content-type of "text/plain; charset=US-ASCII".

> This message can have headers but it is permitted not to as well,
> but I don't read it as having two sets of headers (MIME and message
> headers). It says a body part that starts with a blank line is
> allowed but it looks to me like this should then be interpreted as
> having all default values [for header values].

Note that this stuff is written at the beginning of 5.1, describing
all kinds of multiparts, not only 'multipart/message' things. For
instance, it could be multipart/alternative where the "body parts" are
text/plain or text/html or whatever else. So, there need be no
messages here. They are *like* messages in that they have a header
area, blank line and body area.

In our case, the body area is an actual message. So, it will have its
own message headers, blank line and message body.

Hope that clarifies the confusion.

What the VM parser would do is

- skip the beginning boundary line
- parse all lines until a blank line as MIME headers of a part
- skip the blank line
- parse all the stuff till the next boundary as the body of the part.

And, then if the "body" is a message, display it as a message (for
which a separate message header and message body are needed).

Cheers,
Uday

[Arik (akwm)]

Okay. gmail is getting this wrong then for digest messages, I will
file a bug report.

~Arik

[Uday Reddy (reddyuday)]

Arik writes:

> Okay. gmail is getting this wrong then for digest messages, I will
> file a bug report.

Gees, I didn't realize it was gmail-generated. Thought NPR was
sending it through some custom program.

I can imagine that many mail clients do this. Since all the MIME
information for messages is contained in the message header itself,
the messages don't need a separate MIME header. So, almost invariably
each message has to start with a blank line. I can imagine many a
programmer thinking it is silly and skipping it.

So, I don't mind if you want to change VM to accommodate this. But,
please do it in the MIME parsing stage. Just trying to display
something sensible when the internal information is wrong won't work
very well.

Cheers,
Uday

[Arik (akwm)]

Uday Reddy writes:
 > Arik writes:
 >
 > > Okay. gmail is getting this wrong then for digest messages, I will
 > > file a bug report.
 >
 > Gees, I didn't realize it was gmail-generated. Thought NPR was
 > sending it through some custom program.

No, the NPR feed I generated, the problem with gmail is if you forward
multiple messages (uses digest) then gmail doesn't recognize the
encoding properly. This is probably what led me to interpret the rfc
that way. But now, after doing some tests, it looks like gmail's
problem is that for digest messages it does not properly assume that
the content-type is "message/rfc822" like it should for
"multipart/digest" when it is not specified.

I gathered this by sending utf encoded messages to gmail in the normal
VM way (no mime-header part) and then I explicitly added the
"message/rfc822" header at which point gmail displayed the messages
properly.

[Uday Reddy (reddyuday)]

Arik writes:

> I gathered this by sending utf encoded messages to gmail in the normal
> VM way (no mime-header part) and then I explicitly added the
> "message/rfc822" header at which point gmail displayed the messages
> properly.

Then, do you want to make VM generate the message/rfc822 header?
There is no harm in generating it.

Uday

[Arik (akwm)]

 > > I gathered this by sending utf encoded messages to gmail in the normal
 > > VM way (no mime-header part) and then I explicitly added the
 > > "message/rfc822" header at which point gmail displayed the messages
 > > properly.
 >
 > Then, do you want to make VM generate the message/rfc822 header?
 > There is no harm in generating it.

This might be the safest bet, just to cover all bases. I think this
only needs to be done in the vm-digest.el where the boundary is
inserted.

~Arik