example of problem when forwarding non regular-text mail

[mere user]

try forwarding this to gmail, for example. it wont open properly there...

X-Mozilla-Status: 1001
X-Mozilla-Status2: 00000000
Content-Transfer-Encoding: quoted-printable
X-Mailer: N/A
Content-Type: text/html; charset="us-ascii"
X-Source-IP: yy.rc.tt.xx.xx [140.99]
X-CT-RefId: str=0001.0A020202.4D75CE79.00C0,ss=1,fgs=0
X-Spam-Score: 0.00%
Return-Path: <email address hidden>
Received-SPF: SoftFail (HUB00.vvv.xx.xx: domain of transitioning
 <email address hidden> discourages use of 10.99 as
 permitted sender)
MIME-Version: 1.0
From: <email address hidden>
To: <email address hidden>
Subject: Your Extended Service Plan Details
Date: Tue, 8 Mar 2011 06:34:52 +0000

<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
><p>Dear Valued Customer,</p><p>Thank you for your recent Extended Service
Plan purchase at Yy Style. Your decision to protect your investment is a
wise one that will ensure long-term peace of mind and the assurance of
continued enjoyment from your product purchase.<p><p><a href=3D"http://www.=
xx">Click
here</a> for your official Terms and Conditions of your service agreement.=
=20
This document can be saved or printed for your permanent records and
contains all coverage details and instructions should you need service. If
you need service or have any questions surrounding your Extended Service
Plan, please call the toll free number printed on the first page.<p><p>We
look forward to serving you.<p><p>Yy Extended Service Plans<p><p>NOTE:=20
Please do not reply to this message. This e-mail was sent from an
auto-notification system that cannot accept incoming e-mail.<p>

[Uday Reddy (reddyuday)]

Note that you are now supposed to use capital `Z' to forward messages as MIME attachments. Earlier `z' used to do that. (That was a stop-gap solution really.)

If you use emacs-w3m for html, then `z' will forward a plain text version of html. That is working for gmail.

Cheers,
Uday

[mere user (emacs-user)]

hi Uday, when I use capital Z to forward to gmail, I get (in gmail) an
option to "view as text", which is not a pretty sight, or to download,
which does lead to a properly formatted html file. But I would have
expected to just see the forwarded html message in gmail without
having to download it first. do you see something else when
forwarding this specific message to gmail? best, Eli

On Fri, Mar 11, 2011 at 2:27 AM, Uday Reddy <email address hidden> wrote:
> Note that you are now supposed to use capital `Z' to forward messages as
> MIME attachments.  Earlier `z' used to do that.  (That was a stop-gap
> solution really.)
>
> If you use emacs-w3m for html, then `z' will forward a plain text
> version of html.  That is working for gmail.
>
> Cheers,
> Uday
>
> --
>
> Bug description:
>  try forwarding this to gmail, for example.  it wont open properly
>  there...
>
>  X-Mozilla-Status: 1001
>  X-Mozilla-Status2: 00000000
>  Content-Transfer-Encoding: quoted-printable
>  X-Mailer: N/A
>  Content-Type: text/html; charset="us-ascii"
>  X-Source-IP: yy.rc.tt.xx.xx [140.99]
>  X-CT-RefId: str=0001.0A020202.4D75CE79.00C0,ss=1,fgs=0
>  X-Spam-Score: 0.00%
>  Return-Path: <email address hidden>
>  Received-SPF: SoftFail (HUB00.vvv.xx.xx: domain of transitioning
>   <email address hidden> discourages use of 10.99 as
>   permitted sender)
>  MIME-Version: 1.0
>  From: <email address hidden>
>  To: <email address hidden>
>  Subject: Your Extended Service Plan Details
>  Date: Tue, 8 Mar 2011 06:34:52 +0000
>
>  <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
>  ><p>Dear Valued Customer,</p><p>Thank you for your recent Extended Service
>  Plan purchase at Yy Style.  Your decision to protect your investment is a
>  wise one that will ensure long-term peace of mind and the assurance of
>  continued enjoyment from your product purchase.<p><p><a href=3D"http://www.=
>  xx">Click
>  here</a>  for your official Terms and Conditions of your service agreement.=
>  =20
>  This document can be saved or printed for your permanent records and
>  contains all coverage details and instructions should you need service.  If
>  you need service or have any questions surrounding your Extended Service
>  Plan, please call the toll free number printed on the first page.<p><p>We
>  look forward to serving you.<p><p>Yy Extended Service Plans<p><p>NOTE:=20
>  Please do not reply to this message.  This e-mail was sent from an
>  auto-notification system that cannot accept incoming e-mail.<p>
>
> To unsubscribe from this bug, go to:
> https://bugs.launchpad.net/vm/+bug/732943/+subscribe
>

[Uday Reddy (reddyuday)]

Yes, it looks fine for me. What is your value of vm-forwarding-digest-type?

[mere user (emacs-user)]

mime

On Sat, Mar 19, 2011 at 9:43 PM, Uday Reddy <email address hidden> wrote:
> Yes, it looks fine for me.  What is your value of vm-forwarding-digest-
> type?
>
> --
> You received this bug notification because you are a direct subscriber
> of the bug.
> https://bugs.launchpad.net/bugs/732943
>
> Title:
>  example of problem when forwarding non regular-text mail
>
> Status in VM (View Mail) for Emacs:
>  New
>
> Bug description:
>  try forwarding this to gmail, for example.  it wont open properly
>  there...
>
>  X-Mozilla-Status: 1001
>  X-Mozilla-Status2: 00000000
>  Content-Transfer-Encoding: quoted-printable
>  X-Mailer: N/A
>  Content-Type: text/html; charset="us-ascii"
>  X-Source-IP: yy.rc.tt.xx.xx [140.99]
>  X-CT-RefId: str=0001.0A020202.4D75CE79.00C0,ss=1,fgs=0
>  X-Spam-Score: 0.00%
>  Return-Path: <email address hidden>
>  Received-SPF: SoftFail (HUB00.vvv.xx.xx: domain of transitioning
>   <email address hidden> discourages use of 10.99 as
>   permitted sender)
>  MIME-Version: 1.0
>  From: <email address hidden>
>  To: <email address hidden>
>  Subject: Your Extended Service Plan Details
>  Date: Tue, 8 Mar 2011 06:34:52 +0000
>
>  <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
>  ><p>Dear Valued Customer,</p><p>Thank you for your recent Extended Service
>  Plan purchase at Yy Style.  Your decision to protect your investment is a
>  wise one that will ensure long-term peace of mind and the assurance of
>  continued enjoyment from your product purchase.<p><p><a href=3D"http://www.=
>  xx">Click
>  here</a>  for your official Terms and Conditions of your service agreement.=
>  =20
>  This document can be saved or printed for your permanent records and
>  contains all coverage details and instructions should you need service.  If
>  you need service or have any questions surrounding your Extended Service
>  Plan, please call the toll free number printed on the first page.<p><p>We
>  look forward to serving you.<p><p>Yy Extended Service Plans<p><p>NOTE:=20
>  Please do not reply to this message.  This e-mail was sent from an
>  auto-notification system that cannot accept incoming e-mail.<p>
>
> To unsubscribe from this bug, go to:
> https://bugs.launchpad.net/vm/+bug/732943/+subscribe
>

[Uday Reddy (reddyuday)]

I have forwarded your message to your gmail address. Does it look ok?

And, do you want to forward your copy to <email address hidden>? I
will see what is wrong with it.

Cheers,
Uday

[mere user (emacs-user)]

On Sat, Mar 19, 2011 at 10:06 PM, Uday S Reddy <email address hidden> wrote:
> I have forwarded your message to your gmail address.  Does it look ok?

it did look very good

> And, do you want to forward your copy to <email address hidden>?  I
> will see what is wrong with it.

just did

> Cheers,
> Uday
>

[Uday Reddy (reddyuday)]

Your forward came as an "attachment", i.e., it had content-disposition
marked as "attachment" and a filename associated with it. That is why
you can't view it on the web in gmail. See the attachment below as I
received it (using D to view the raw form).

If you normally use 'Z', the content-disposition will be
"unspecified". Do you know how you got the attachment disposition?

If you click the right mouse button (or the Mac equivalent of it) on
the attachment button, you will get a menu where you can change the
content disposition. Please try it and see.

Cheers,
Uday

[Uday Reddy (reddyuday)]

If the content-disposition is unspecified some SMTP servers make it "attachment". So, make it inline instead.

[mere user (emacs-user)]

thanks!

On Fri, Mar 25, 2011 at 7:49 AM, Uday Reddy <email address hidden> wrote:
> ** Changed in: vm
>       Status: Triaged => Fix Committed

[Tim Cross (tcross)]

Just got bitten by this one. I have gotten into the habit of forwarding meeting invites (sent as an icalendar invite from Outlook) to gmail so that it can be added to my calendar. Previously, hitting 'z' worked fine and I could go to gmail and click on Add To Calendar. However, now, hitting 'z' only gives a (largely unreadable) dump of the icalendar file. Gmail does not recognize it as an icalendar file (because of no MIME indicator I guess). Sending with 'Z' seems to work.

While I can understand (parti8ally) the issue, this does not strike me as a good solution. Perhaps the code needs to be a little smarter. If your forwarding on a message that has a MIME part that (perhaps) is not HTML or plain text, maybe 'z' should default to sending it as a mime message? The problem with the current solution is that you have to remember to use Z and to some extent, you need to know what the recipient is likely to be using to read the message and decide if plain text or mime is better.

[Uday Reddy (reddyuday)]

Tim Cross writes:

> While I can understand (parti8ally) the issue, this does not strike me
> as a good solution. Perhaps the code needs to be a little smarter. If
> your forwarding on a message that has a MIME part that (perhaps) is not
> HTML or plain text, maybe 'z' should default to sending it as a mime
> message?

Hi Tim, the idea of `z' is to produce a human-readable message as
close to plain-text as possible. (This has always been the case from
the very beginning of VM, but the meaning of "as close as possible to
plain-text" has changed as VM developed.)

The idea of `Z' is to produce a faithful forward that can be processed
by programs at the other end. (Earlier it used to be `@', which still
exists.)

If this doesn't suit you, you can always change your key bindings of
course...

Cheers,
Uday

[John Hein (xpqheqdvq4)]

FWIW, I have been used to 'z' sending mime message/rfc822 for a long time now, (and I've always preferred that over the outlook-ish version of forwarding). I don't have a huge problem with the behavior change, but it should be mentioned prominently in the notes for the next release. I'll just have to get into the habit of using 'Z' instead of 'z'.

However, the recent forwarding behavior change seems to have broken vm-forward-message-all-headers. I always want that to be an encapsulated rfc822 message (although a plain text version _might_ be okay if it actually showed all the headers). Now that vm-forward-message-all-headers points to the new plain text version of vm-forward-message, the "full headers" part no longer works (rev 1181 at least).

(maybe that should be opened as a new bug)

General comment about what the default for 'z' should be ...
I've never seen any problems with sending encapsulated rfc822 to people. It seems to me that 'z' should default to that (mime) since that's what it's been for so long (I've never had the expectation that 'z' would try to convert a message to plain text). Are there a large number of real cases where sending mime causes problems for recipients these days? Maybe 'Z' should be the plain text version since that's new behavior for mesages forwarded with vm (at least the vm I've been using for 15 years, as far as I can recall)... unless mime causes lots of problems for more than a tiny percentage of recipients.

[Tim Cross (tcross)]

Uday,

I think John's comments summed up things better then my attempt. However,
note that there are two issues here for me. I can get use to the new
behavior or rebind the key. However, there is also an issue with the MIME
part VM is selecting to send as plain text which may need improvements as
well.

I still feel this change is sub-optimal. The message which highlighted this
for me was an invitation sent in icalendar format. If I want to forward this
on to someone else, I have to make a decision. Will their mail client
support the icalendar format? If it does, then I should send it as a MIME
type so that their client can processes it easily. However, if they don't,
then that is likely to be a problem for them and I should send it as plain
text using 'z' I now need to think about this and make a decision on which
to use, 'z' or 'Z'. Previously, this was not necessary. To make matters
worse, the part of the message being forwarded as plain text is incorrect
IMO (see below).

From memory, one of the reasons for this change was because some people
noted (including me BTW) that if you forward a message with no additional
text added in the presentation buffer, for some recipients, it looked like
an empty message (I guess they don't notice/see the MIME attachment 'button'
(or whatever their client uses). If this is the main reason for the change,
then perhaps a better solution would be for VM to check before sending
whether any additional text has been added to the presentation buffer and if
there is none, add a standard "MIME message attached" bit of text (or
something similar).

The second issue I've observed is that VM doesn't seem to be using the most
appropriate MIME type to convert and forward. As an example, consider the
icalendar invite that triggered noticing these changes for me. This message
has three MIME parts consisting of

Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64

Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: base64

Content-Type: text/calendar; charset="utf-8"; method=REQUEST
Content-Transfer-Encoding: base64

I have VM configured so that it displays the text/calendar part as this then
allows me to use icalendar-import-buffer to add the invite to my diary file.
However, using 'z' results in a text representation of the icalendar format
being forwarded as text/plain when I think it should be forwarding the
text/plain part. Apart from this format not being very human readable
(especially compared to the text/plain or text/html parts), because it has a
mime type of text/plain, clients don't recognise it as an icalendar format
message and won't import it or translate it to something more readable for
humans. The end result is a very human unfriendly message that is difficult
to read (see example below).

My suggestion would be to have the 'z' function use the closest plain text
mime type in the message being forwarded i.e. instead of using
text/calendar, use the text/plain type if available. VM should ignore what
the user's MIME display setting is i.e. just because I choose to display
text/calendar, this should not also imply this is the type to use in 'z'
forwarding.

To clari...

[Uday Reddy (reddyuday)]

John Hein writes:

> Maybe 'Z' should be the plain text version
> since that's new behavior for mesages forwarded with vm (at least
> the vm I've been using for 15 years, as far as I can
> recall)... unless mime causes lots of problems for more than a tiny
> percentage of recipients.

Right. I went and checked. You were close.

`vm-forwarding-digest-type' was changed to "mime" in Jan 1997. Before
that, I think it used to be "rfc934". But I had set it to nil for
myself in those days. So I don't have a good recollection of the
event.

I think what you guys are missing is the use of this variable for
vm-forward-message. I will add it back and then you should be fine.

Cheers,
Uday

[Uday Reddy (reddyuday)]

Tim Cross writes:

> Uday,
>
> I think John's comments summed up things better then my attempt. However,
> note that there are two issues here for me. I can get use to the new
> behavior or rebind the key. However, there is also an issue with the MIME
> part VM is selecting to send as plain text which may need improvements as
> well.

As I mentioned in my response to John, it does look like
`vm-forwarding-digest-type' should be respected by `vm-forward-message'.

As for the calendar issues, can you open a new bug report and attach the
message that you would like to forward?

(There are some anomalies with the `text/calendar' MIME type because it is
inconsistent. text types are meant to be displayed to the user but
`text/calendar' is meant to be processed by a program. It should have really
been named `applicationc/calendar'.)

As for Google producing bad `text/plain' content, please do complain to them
if there is a way to do so!

Cheers,
Uday

[John Hein (xpqheqdvq4)]

Uday Reddy wrote at 06:49 -0000 on Apr 13, 2011:
 > `vm-forwarding-digest-type' was changed to "mime" in Jan 1997.
....
 > I think what you guys are missing is the use of this variable for
 > vm-forward-message. I will add it back and then you should be fine.

That makes sense. Thanks.

[Uday Reddy (reddyuday)]

Ok, I have put this on my to-do list for the 8.2.0 release. Meanwhile,
please feel free to rebind `z' to vm-forward-message-encapsulated for your
own use.

Cheers,
Uday

[Tim Cross (tcross)]

On Wed, Apr 13, 2011 at 5:49 PM, Uday Reddy <email address hidden> wrote:

> Tim Cross writes:
>
> > Uday,
> >
> > I think John's comments summed up things better then my attempt. However,
> > note that there are two issues here for me. I can get use to the new
> > behavior or rebind the key. However, there is also an issue with the MIME
> > part VM is selecting to send as plain text which may need improvements as
> > well.
>
> As I mentioned in my response to John, it does look like
> `vm-forwarding-digest-type' should be respected by `vm-forward-message'.
>
> OK, makes sense and certainly worth trying.

> As for the calendar issues, can you open a new bug report and attach the
> message that you would like to forward?
>
> Yep, will do.

> (There are some anomalies with the `text/calendar' MIME type because it is
> inconsistent. text types are meant to be displayed to the user but
> `text/calendar' is meant to be processed by a program. It should have
> really
> been named `applicationc/calendar'.)
>

I guess it depends on how you interpret the mime types. The content is just
'text' and it could be argued that using text/calendar means that clients
can, if they want, include a formatter for that mime type that could improve
the content presentation or they can just display 'as is'. In general,
application/* types are more likely to have binary or non-printable content
and would need some additional processing to display.

>
> As for Google producing bad `text/plain' content, please do complain to
> them
> if there is a way to do so!
>
> Google is working correctly based on what we send it. The issue is that we
are forwarding the text/calendar part as text/plain.

> Cheers,
> Uday
>
>
> ** Visibility changed to: Public
>
> --
> You received this bug notification because you are a direct subscriber
> of the bug.
> https://bugs.launchpad.net/bugs/732943
>
> Title:
> example of problem when forwarding non regular-text mail
>
> Status in VM (View Mail) for Emacs:
> Fix Committed
>
> Bug description:
> try forwarding this to gmail, for example. it wont open properly
> there...
>
> X-Mozilla-Status: 1001
> X-Mozilla-Status2: 00000000
> Content-Transfer-Encoding: quoted-printable
> X-Mailer: N/A
> Content-Type: text/html; charset="us-ascii"
> X-Source-IP: yy.rc.tt.xx.xx [140.99]
> X-CT-RefId: str=0001.0A020202.4D75CE79.00C0,ss=1,fgs=0
> X-Spam-Score: 0.00%
> Return-Path: <email address hidden>
> Received-SPF: SoftFail (HUB00.vvv.xx.xx: domain of transitioning
> <email address hidden> discourages use of 10.99 as
> permitted sender)
> MIME-Version: 1.0
> From: <email address hidden>
> To: <email address hidden>
> Subject: Your Extended Service Plan Details
> Date: Tue, 8 Mar 2011 06:34:52 +0000
>
> <meta http-equiv=3D"Content-Type" content=3D"text/html;
> charset=3Dus-ascii"=
> ><p>Dear Valued Customer,</p><p>Thank you for your recent Extended Service
> Plan purchase at Yy Style. Your decision to protect your investment is a
> wise one that will ensure long-term peace of mind and the assurance of
> continued enjoyment from your product purchase.<p><p><a href=3D"
> http://www.=
> xx">Click
> here</a> for your official Terms and Conditions of your service
> ...

[Uday Reddy (reddyuday)]

John Hein writes:

> However, the recent forwarding behavior change seems to have broken vm-
> forward-message-all-headers. I always want that to be an encapsulated
> rfc822 message (although a plain text version _might_ be okay if it
> actually showed all the headers).

Dear John, When I use MIME forwarding, I get all the headers. Do you really
need vm-forward-message-all-headers in this case?

Cheers,
Uday

[John Hein (xpqheqdvq4)]

I'm not sure I understand exactly what you're asking.

'Z' (vm-forward-message-encapsulated) in the trunk (r1187) does what I want when I want all the headers.

'z' _used to_ encapsulate the forwarded message as message/rfc822, but not with all headers - just some "interesting" ones (as defined by the vm-forwarded-headers list).

I would say that 80% of the time I want the latter behavior. 20% of the time the former. Maybe 90/10.

I almost never want a plain-textified inline include as a forward. vm-forward-message-all-headers does this now (again, r1187). It _used to_ do the same thing vm-forward-message-encapsulated does now, if I recall correctly.

What I was thinking was that 'z' do the same it's always done (encapsulated mime message/rfc822 with selected headers). 'Z' could do the new inline include thing (normal outlook style forwarding) - some people might like to use that. Perhaps 'C-u z' could do message/rfc822 with "all headers".

Do I need it to be called "vm-forward-message-all-headers". No, I don't care about the name. But it's nice to have both the reduced headers and full headers forwarding as functional options.

Sorry if I wasn't sure what you were asking. I hope my longer explanation clarifies my thinking and explains my usage.

[Uday Reddy (reddyuday)]

John Hein writes:

> 'z' _used to_ encapsulate the forwarded message as message/rfc822, but
> not with all headers - just some "interesting" ones (as defined by the
> vm-forwarded-headers list).
>
> I would say that 80% of the time I want the latter behavior. 20% of the
> time the former. Maybe 90/10.

Ok, I see that I changed this in revision 971, in October 2010. Since then,
MIME-encapsulated forwarding, irrespective which key-binding is used, has
been forwarding all headers.

The rationale is two-fold. If people make mistakes in the forwarded
headers, the MIME might be broken and the recipients won't be able to
process the encapsulated message. Secondly, there is no harm in sending all
the headers because the recipients are expected to process the
MIME-encapsulated message and use their own formatting to view the headers.

So, if you use MIME-forwarding, you don't need
vm-forward-message-all-headers. You get all the headers in the normal
forward.

Cheers,
Uday

[John Hein (xpqheqdvq4)]

Uday Reddy wrote in comment 21:
 > The rationale is two-fold. If people make mistakes in the forwarded
 > headers, the MIME might be broken and the recipients won't be able to
 > process the encapsulated message. Secondly, there is no harm in sending all
 > the headers because the recipients are expected to process the
 > MIME-encapsulated message and use their own formatting to view the headers.

re: "no harm in sending all the headers". Often I don't want to forward on all the headers because I don't want to expose internal header information. My only recourse now (for encapsulated message/rfc822 forwarding) is to edit the message before forwarding which is klunky (and a [tiny] step backward in vm functionality).

So I think there are two different independent behavior knobs for forwarding that we're talking about:
 (1) inline vs. mime encapsulated attachment
 (2) all or partial headers

The combination I would probably never use: inline + full headers.
The next combinations in order of what would be used most (for me):

 - mime + partial (most of the time)

 - mime + all (for the occasional circumstance where I specifically want to include all the header noise)

 - inline + partial (rarely)

Having mime + all be the default is fine. Often people don't forward me enough (typically outlook users that only show names and not email addresses in forwarded messages), so if mime + all is the default, that's okay with me. But keeping a way to easily do mime + partial (like 'z' has done for a long time now) would be much appreciated.

inline forwarding is new for vm. I'll take a moment here to lobby for _not_ making it the default (even for a forward of a single message). Reading "threaded" followup conversations in outlook style (top post on top of 20 messages that have already been sent) is difficult and people never prune content. If you're going to forward email conversations, a mime digest is the better way to go (especially for recipients who use a decent mail reader that get help show proper threading - like vm!) - yes, vm still does the digest when forwarding multiple messages, but I'd like to see mime be the default for a single message as well. Discouraging that kind of top-post forwarding makes the world a better place ;) Plus, it's more consistent with what vm has been doing for the last ('z' => mime, not inline) decade+.

[Uday Reddy (reddyuday)]

John Hein writes:

> re: "no harm in sending all the headers". Often I don't want to forward
> on all the headers because I don't want to expose internal header
> information. My only recourse now (for encapsulated message/rfc822
> forwarding) is to edit the message before forwarding which is klunky
> (and a [tiny] step backward in vm functionality).

Ok, that is a good argument. I have added new variables for controlling
plain-text forwarding. So, I am reverting the original variables to the old
behaviour.

Changes committed in revision 1199. Please test it out.

Cheers,
Uday