Email-in blog entry scrambles formatting

[Jim B. Glenn]

After today's update, I sent an e-mail into a community with a bulleted list and with the phrase "is_own_profile".

The bulleted list was stripped and was mashed together as one paragraph with no breaks.
The own in "is_own_profile" turned into italics.

Is e-mail in getting converted into markdown formatting?

[Paul Everitt (paul-agendaless)]

Jim, is there a reason to believe TinyMCE had anything to do with this? If the text was composed in email, then TinyMCE isn't used. I'll edit the title.

[Chris Rossi (chris-archimedeanco)]

Well, email in is not getting converted to markdown. It is being treated as markdown. Markdown2 is used to convert plain text emails into html blog entries. If it were not used, you would have no paragraph breaks, among other things. We slip markdown in surreptitiously to make content that is mailed in magically appear properly formatted. In the vast majority of cases, no one notices, because it does the intuitive right thing. Everyone would notice, though, if we took it out. In some cases, because it is, technically speaking, a markup language, someone enters some text that it interprets (or misinterprets) in some unintended way.

There is not, as far as I can tell, an easy fix for this. We had discussed allowing html email via mailin, which would eliminate the need for using something like markdown in the cases where we get an html copy. The technical challenge was then to find or build something to scrub the incoming html clean of anything which might be a security threat.

Another option might be to build our own text to html converter. Something as simple as converting newline characters to <br/> tags, for example, would probably get us a long way.

I would recommend considering any work along these lines new feature work. I don't think this is a good candidate for an M47 bug, since there's not really a bug, per se, to fix. The problem is just that markdown doesn't always do what you'd expect. Any move to something different, though, will require some careful consideration.

[Paul Everitt (paul-agendaless)]

Sorry, I misunderstood the bug. I thought Jim was saying that something which used to work, changed behavior. Jim, is that correct?

If not, then I agree with Chris's assessment. The design of this approach goes back to a late 2006 discussion with Jonathan Hooper at OSI. We can put it back on the table, but it will have to go through OSI as a new feature on the product backlog, because the change will be significant.

Once I hear back from Jim regarding whether this is a regression, I'll update the status.

--Paul

On Sep 8, 2010, at 9:00 AM, Chris Rossi wrote:

> Well, email in is not getting converted to markdown. It is being
> treated as markdown. Markdown2 is used to convert plain text emails
> into html blog entries. If it were not used, you would have no
> paragraph breaks, among other things. We slip markdown in
> surreptitiously to make content that is mailed in magically appear
> properly formatted. In the vast majority of cases, no one notices,
> because it does the intuitive right thing. Everyone would notice,
> though, if we took it out. In some cases, because it is, technically
> speaking, a markup language, someone enters some text that it interprets
> (or misinterprets) in some unintended way.
>
> There is not, as far as I can tell, an easy fix for this. We had
> discussed allowing html email via mailin, which would eliminate the need
> for using something like markdown in the cases where we get an html
> copy. The technical challenge was then to find or build something to
> scrub the incoming html clean of anything which might be a security
> threat.
>
> Another option might be to build our own text to html converter.
> Something as simple as converting newline characters to <br/> tags, for
> example, would probably get us a long way.
>
> I would recommend considering any work along these lines new feature
> work. I don't think this is a good candidate for an M47 bug, since
> there's not really a bug, per se, to fix. The problem is just that
> markdown doesn't always do what you'd expect. Any move to something
> different, though, will require some careful consideration.
>
> --
> Email-in blog entry scrambles formatting
> https://bugs.launchpad.net/bugs/632754
> You received this bug notification because you are subscribed to KARL3.
>
> Status in KARL3: New
>
> Bug description:
> After today's update, I sent an e-mail into a community with a bulleted list and with the phrase "is_own_profile".
>
> The bulleted list was stripped and was mashed together as one paragraph with no breaks.
> The own in "is_own_profile" turned into italics.
>
> Is e-mail in getting converted into markdown formatting?
>
>

[Jim B. Glenn (jimbglenn)]

I believe that we used to be able to e-mail in bullet list and it show up as bullet list in blog.

It is now stripped out when it shows up in blog.
-Jim

On Sep 8, 2010, at 9:15 AM, Paul Everitt wrote:

>
> Sorry, I misunderstood the bug. I thought Jim was saying that something which used to work, changed behavior. Jim, is that correct?
>
> If not, then I agree with Chris's assessment. The design of this
> approach goes back to a late 2006 discussion with Jonathan Hooper at
> OSI. We can put it back on the table, but it will have to go through
> OSI as a new feature on the product backlog, because the change will be
> significant.
>
> Once I hear back from Jim regarding whether this is a regression, I'll
> update the status.
>
> --Paul
>
> On Sep 8, 2010, at 9:00 AM, Chris Rossi wrote:
>
>> Well, email in is not getting converted to markdown. It is being
>> treated as markdown. Markdown2 is used to convert plain text emails
>> into html blog entries. If it were not used, you would have no
>> paragraph breaks, among other things. We slip markdown in
>> surreptitiously to make content that is mailed in magically appear
>> properly formatted. In the vast majority of cases, no one notices,
>> because it does the intuitive right thing. Everyone would notice,
>> though, if we took it out. In some cases, because it is, technically
>> speaking, a markup language, someone enters some text that it interprets
>> (or misinterprets) in some unintended way.
>>
>> There is not, as far as I can tell, an easy fix for this. We had
>> discussed allowing html email via mailin, which would eliminate the need
>> for using something like markdown in the cases where we get an html
>> copy. The technical challenge was then to find or build something to
>> scrub the incoming html clean of anything which might be a security
>> threat.
>>
>> Another option might be to build our own text to html converter.
>> Something as simple as converting newline characters to <br/> tags, for
>> example, would probably get us a long way.
>>
>> I would recommend considering any work along these lines new feature
>> work. I don't think this is a good candidate for an M47 bug, since
>> there's not really a bug, per se, to fix. The problem is just that
>> markdown doesn't always do what you'd expect. Any move to something
>> different, though, will require some careful consideration.
>>
>> --
>> Email-in blog entry scrambles formatting
>> https://bugs.launchpad.net/bugs/632754
>> You received this bug notification because you are subscribed to KARL3.
>>
>> Status in KARL3: New
>>
>> Bug description:
>> After today's update, I sent an e-mail into a community with a bulleted list and with the phrase "is_own_profile".
>>
>> The bulleted list was stripped and was mashed together as one paragraph with no breaks.
>> The own in "is_own_profile" turned into italics.
>>
>> Is e-mail in getting converted into markdown formatting?
>>
>>
>
> --
> Email-in blog entry scrambles formatting
> https://bugs.launchpad.net/bugs/632754
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in KARL3: New
>
> Bug description:
...

[Chris Rossi (chris-archimedeanco)]

Hi Jim,

Really, nothing has changed in this department in a long time. Can you find an instance of a bullet list that was created via mail in? If you can, could you attach (or send to me privately) the full raw emails for both the instance where the bullet list didn't convert correctly and the instance where it did? I suspect that if it has worked before and didn't this time, it would be because you happened to hit the format the markdown recognized as a bullet list in the first email and not in the second.

See here for a reference: http://daringfireball.net/projects/markdown/syntax#list

[Chris Rossi (chris-archimedeanco)]

After discussion with Jim B on irc, we have decided this is probably not a regression. Paul will bring up a new feature using the normal channels for that.