Thunderbird places incorrect newlines when composing a reply using "reply above quote"

[Brian Norris]

When I compose a reply to any message, the newlines placed around the quote are incorrectly formatted. I am using "reply above the quote", and I only get a single newline above the quote, but there's two newlines below the quote. This means I always have to click around and press "Enter" and "Delete" a few times to reformat every email reply I compose.

I think this is a regression moving to Thunderbird 13.x. I see the issue on both Ubuntu 11.10 and 12.04.

I will try to add some screenshots to demonstrate.

ProblemType: Bug
DistroRelease: Ubuntu 11.10
Package: thunderbird 13.0.1+build1-0ubuntu0.11.10.1
ProcVersionSignature: Ubuntu 3.0.0-19.33-generic 3.0.27
Uname: Linux 3.0.0-19-generic x86_64
NonfreeKernelModules: nvidia
AddonCompatCheckDisabled: False
AlsaVersion: Advanced Linux Sound Architecture Driver Version 1.0.24.
ApportVersion: 1.23-0ubuntu4
Architecture: amd64
ArecordDevices:
 **** List of CAPTURE Hardware Devices ****
 card 0: PCH [HDA Intel PCH], device 0: ALC892 Analog [ALC892 Analog]
   Subdevices: 1/1
   Subdevice #0: subdevice #0
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC1: brian 1767 F.... pulseaudio
 /dev/snd/controlC0: brian 1767 F.... pulseaudio
BuildID: 20120615164636
CRDA: Error: [Errno 2] No such file or directory
Card0.Amixer.info:
 Card hw:0 'PCH'/'HDA Intel PCH at 0xfa620000 irq 60'
   Mixer name : 'Realtek ALC892'
   Components : 'HDA:10ec0892,10438469,00100302'
   Controls : 35
   Simple ctrls : 20
Card1.Amixer.info:
 Card hw:1 'NVidia'/'HDA NVidia at 0xfa080000 irq 17'
   Mixer name : 'Nvidia GPU 15 HDMI/DP'
   Components : 'HDA:10de0015,10de0101,00100100'
   Controls : 16
   Simple ctrls : 4
Channel: release
Date: Fri Jul 6 09:41:05 2012
ForcedLayersAccel: False
IfupdownConfig:
 auto lo
 iface lo inet loopback
InstallationMedia: Ubuntu 11.04 "Natty Narwhal" - Release amd64 (20110427.1)
IpRoute:
 default via 192.168.1.1 dev eth0 proto static
 169.254.0.0/16 dev eth0 scope link metric 1000
 192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.80 metric 1
IwConfig:
 lo no wireless extensions.

 eth0 no wireless extensions.
PrefSources:
 /etc/thunderbird/syspref.js
 prefs.js
 user.js
 [Profile]/extensions/{a62ef8ec-5fdc-40c2-873c-223b8a6925cc}/defaults/preferences/preferences.js
 [Profile]/extensions/{e2fda1a4-762b-4020-b5ad-a41df1933103}/defaults/preferences/lightning.js
Prefs:
 places.database.lastMaintenance: 1336507414 (prefs.js)
 extensions.lastAppVersion: "13.0.1" (prefs.js)
 network.cookie.prefsMigrated: true (prefs.js)
 places.history.expiration.transient_optimal_database_size: 167772160 (prefs.js)
 places.history.expiration.transient_current_max_pages: 104858 (prefs.js)
Profiles: Profile0 (Default) - LastVersion=13.0.1/20120615164636 (In use)
RelatedPackageVersions:
 gnome-shell 3.2.2.1-0ubuntu0.1
 rhythmbox-plugins 2.90.1~20110908-0ubuntu1.3
 icedtea-plugin 1.1.3-1ubuntu1.1
 totem-mozilla 3.0.1-0ubuntu7.1
RfKill:
 0: hci0: Bluetooth
  Soft blocked: no
  Hard blocked: no
RunningIncompatibleAddons: False
SourcePackage: thunderbird
UpgradeStatus: Upgraded to oneiric on 2011-11-08 (241 days ago)
WifiSyslog:
 Jul 6 09:25:52 brian-ubuntu NetworkManager[966]: <info> kernel firmware directory '/lib/firmware' changed
 Jul 6 09:26:34 brian-ubuntu NetworkManager[966]: <info> kernel firmware directory '/lib/firmware' changed
 Jul 6 09:32:16 brian-ubuntu kernel: [589127.615257] gnome-shell[1757]: segfault at 10 ip 00007f605935f0c4 sp 00007fffeedf12f8 error 4 in libnvidia-glcore.so.280.13[7f6058386000+15f4000]
 Jul 6 09:32:48 brian-ubuntu kernel: [589160.172707] SysRq : HELP : loglevel(0-9) reBoot Crash terminate-all-tasks(E) memory-full-oom-kill(F) kill-all-tasks(I) thaw-filesystems(J) saK show-backtrace-all-active-cpus(L) show-memory-usage(M) nice-all-RT-tasks(N) powerOff show-registers(P) show-all-timers(Q) unRaw Sync show-task-states(T) Unmount show-blocked-tasks(W) dump-ftrace-buffer(Z)
 Jul 6 09:32:49 brian-ubuntu kernel: [589161.171764] SysRq : HELP : loglevel(0-9) reBoot Crash terminate-all-tasks(E) memory-full-oom-kill(F) kill-all-tasks(I) thaw-filesystems(J) saK show-backtrace-all-active-cpus(L) show-memory-usage(M) nice-all-RT-tasks(N) powerOff show-registers(P) show-all-timers(Q) unRaw Sync show-task-states(T) Unmount show-blocked-tasks(W) dump-ftrace-buffer(Z)
dmi.bios.date: 11/22/2011
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 2103
dmi.board.asset.tag: To be filled by O.E.M.
dmi.board.name: P8P67 PRO
dmi.board.vendor: ASUSTeK COMPUTER INC.
dmi.board.version: Rev 1.xx
dmi.chassis.asset.tag: Asset-1234567890
dmi.chassis.type: 3
dmi.chassis.vendor: Chassis Manufacture
dmi.chassis.version: Chassis Version
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr2103:bd11/22/2011:svnSystemmanufacturer:pnSystemProductName:pvrSystemVersion:rvnASUSTeKCOMPUTERINC.:rnP8P67PRO:rvrRev1.xx:cvnChassisManufacture:ct3:cvrChassisVersion:
dmi.product.name: System Product Name
dmi.product.version: System Version
dmi.sys.vendor: System manufacturer

[Jbg-w (jbg-w)]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.142 Safari/535.19

Steps to reproduce:

Reply to a message with the "reply above" option set

Actual results:

TB used to include at least one blank line before the reply section, now it jams the reply right up against the first line of the reply area.

Expected results:

Leave at least one blank line, or make the amount of space configurable.

[Rsx11m-pub (rsx11m-pub)]

*** Bug 765826 has been marked as a duplicate of this bug. ***

[Rsx11m-pub (rsx11m-pub)]

Looks like a regression in the 13.0 HTML editor, possibly introduced with the new filelink feature. Steps to reproduce:

1. Compose in HTML mode
2. Don't have a signature defined
3. Set "reply above quote"

Clicking reply inserts two blank lines above the reply header in 10.0.5 ESR and places the cursor to the first one. On 14.0a2, there is only a single blank line above the <div class="moz-cite-prefix"> node.

[Rsx11m-pub (rsx11m-pub)]

Plain-text composition doesn't appear to be affected.

[hashemmasoud (wraithlike)]

*** Bug 768292 has been marked as a duplicate of this bug. ***

[Rsx11m-pub (rsx11m-pub)]

*** Bug 767752 has been marked as a duplicate of this bug. ***

[Brian Norris (computersforpeace)]

Adding another screenshot. This one has some text entered into the reply to show the issue more clearly.

[Nixon049 (nixon049)]

This occurs even if a signature is defined, so long as signature is placed below the quote.

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:13.0) Gecko/20120614 Thunderbird/13.0.1 ID:20120614161456

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[D. Mark Simms (dmsimms)]

I think I'm having the same problem. In previous versions of Thunderbird, when I would reply to a message (quoting the previous message), there would be two blank lines above the start of the quote, with the cursor on the top line. Now there is just one blank line, which makes the whole message feel cramped and makes me press Enter a few times before I put the cursor back at the beginning and start typing. Minor I guess, but considering it's worked perfect for the previous 12 versions of Thunderbird, I consider it a regression!

[D. Mark Simms (dmsimms)]

I got around this by creating a user.js file (see https://developer.mozilla.org/en/Thunderbird/Thunderbird_Configuration_Files#user.js) and putting in the following text:

// Modify quote header //
user_pref("mailnews.reply_header_ondate", "\nOn %s");

I tried using the Config Editor to change the mailnews.reply_header_ondate but it wouldn't take the \n as a newline.

[Brian Norris (computersforpeace)]

Thanks for the workaround. It mostly works for me, but it doesn't take care of removing the extra newlines added at the end of emails.

I agree that this is a regression, of course.

[Jake1138 (jake1138)]

I noticed this recently too. I am running 13.0.1. I hope this gets fixed soon. It's really annoying.

[Alice0775 (alice0775)]

Regression window
Good:
http://hg.mozilla.org/comm-central/rev/3ad9c88900d6
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:13.0) Gecko/20120313 Thunderbird/13.0a1 ID:20120313030025
Bad:
http://hg.mozilla.org/comm-central/rev/1593bfcee4fb
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:14.0) Gecko/20120314 Thunderbird/14.0a1 ID:20120314030025
Pushlog:
http://hg.mozilla.org/comm-central/pushloghtml?fromchange=3ad9c88900d6&tochange=1593bfcee4fb

Suspected : Bug 735380

[Brian Norris (computersforpeace)]

I can confirm that this bug still exists in 15.0. I'm using Ubuntu 12.04.

This issue has been reported on the Ubuntu launchpad tracker:
https://bugs.launchpad.net/ubuntu/+source/thunderbird/+bug/1021813

[Thorokol (thorokol)]

I am using 15.0.1 and I've been waiting for many versions for this to be fixed. I've been using Thunderbird since the very beginning and this is very annoying as it only got introduced into TB a few versions ago. Please fix this.

[Toban57 (toban57)]

I'm running Thunderbird 17.0, portable on XP, and this is STILL an issue. Does not look like they are working on this bug too hard. Very annoying!

[Toban57 (toban57)]

Sorry about the multiple posts. Not sure how to delete the two duplicates.

[Antoine-mechelynck-gmail (antoine-mechelynck-gmail)]

(In reply to Toby Banner from comment #14)
> Sorry about the multiple posts. Not sure how to delete the two duplicates.

You can't. Just be more careful next time. :-)

[Brian Norris (computersforpeace)]

Created attachment 687497
Path: adds back a newline for HTML replies

Since there's been no real help from Mozilla, I'm posting my own fix. This fixes the worst annoyance -- the missing newline at the start of HTML, reply-on-top mail -- but it still leaves a seemingly-extraneous newline below the quote. Please at least give this a look.

[Brian Norris (computersforpeace)]

Bump! Mike, can you take a look at this? Or shall I find a different reviewer?

[Mike Conley (mconley)]

Created attachment 728622
Test fix

This test needs to be modified in order to accommodate the new patch.

[Mike Conley (mconley)]

Comment on attachment 687497
Path: adds back a newline for HTML replies

Review of attachment 687497:
-----------------------------------------------------------------

This appears to fix the described problem, although it breaks one of the Filelink tests (which I've added a patch to address). So I like this!

I think we'll need to request review from the SM folks, since this is a mailnews change.

[Mike Conley (mconley)]

Comment on attachment 728622
Test fix

We've added an extra line break, so we have to modify one of the tests to accommodate it. Blake, you originally reviewed these tests...does this look OK to you?

[Johnvorhaus (johnvorhaus)]

Is there any fix available as an ADD-IN or a SETTING CHANGE? I'm lost in code and am look for something even a STOOPID person can master. Thanks.

[Brian Norris (computersforpeace)]

(In reply to johnvorhaus from comment #21)
> Is there any fix available as an ADD-IN or a SETTING CHANGE? I'm lost in
> code and am look for something even a STOOPID person can master. Thanks.

Hi John (forgive me if that is not your name),

This fix is seemingly making progress toward being included in an upcoming Thunderbird release, and so it will not need an add-on or other workaround. Please be patient, and hopefully this regression will be squashed!

If, however, you want an immediate solution, I did find a workaround that sorta works (it fixes HTML mail but adds an *extra* newline for plain text mail) here:

https://bugs.launchpad.net/ubuntu/+source/thunderbird/+bug/1021813/comments/5

[Brian Norris (computersforpeace)]

Just a ping: any responses from Neil or Blake on the bugfix + test case?

[Bwinton-a (bwinton-a)]

Comment on attachment 728622
Test fix

Yeah, this change seems okay to me, assuming it makes the test pass…

Thanks,
Blake.

[Brian Norris (computersforpeace)]

Ping: Neil, can you review the bugfix patch? If not, what's the next step?

[EricDHH (ericdhh)]

This is 13.04amd64 with thunderbird 17.0.6

Quoting is impossible, the complete text is often demolished by thunderbird, ctrl-z is the best friend to find a way to have proper quoting. This is a feature that the simple claws-mail handle at a glance. Please repair the quoting in thunderbird, regression is right.

[Brian Norris (computersforpeace)]

Hi ErircDHH,

This bug report is specifically for one problem: that Thunderbird places extra newlines on certain types of messages. Having "the complete text ... demolished by thunderbird" sounds like a different issue. In that case, you should file a new bug report with detailed information.

Thanks,
Brian

[Josiah-l (josiah-l)]

Neil seems pretty busy.

IanN, do you have a few to review this? (Looks like this is your area)

[Pf4public (pf4public)]

Any news on this?

Tried user.js with
// Modify quote header //
 user_pref("mailnews.reply_header_ondate", "\nOn %s");

but it does not help
TB version 24.1.1

[Louiserbaldwin (louiserbaldwin)]

Small patch to fix major annoyance is just waiting review for the past year. Come on, ladies and gentlemen. How can we find someone to review this?

[Josiah-l (josiah-l)]

Neil, could you please review this quickly? It's a two liner.

[Pf4public (pf4public)]

Obviously, it is way too difficult to fix this :(

[Brian Norris (computersforpeace)]

Time for an annual checkup here. Anyone listening? Can the provided patch be applied?

[Josiah-l (josiah-l)]

If I can't get Neil to look at this later today then I think Mike or myself could probably just review it.

[Brian Norris (computersforpeace)]

(In reply to Josiah Bruner [:JosiahOne] from comment #32)
> If I can't get Neil to look at this later today then I think Mike or myself
> could probably just review it.

It's past "later today" on all parts of the globe ;) Can you or Mike take a look please?

FWIW, I retested against the latest, and everything patches and runs as expected.

(Now that I'm revisiting the history, it looks like Mike already added his "review+", which I think means my patch was officially "reviewed." But then he pushed it back to Neil, who hasn't responded. Can we just finish this off already?)

[Neil-httl (neil-httl)]

Comment on attachment 687497
Path: adds back a newline for HTML replies

Sorry for dropping the ball on this.

Editor is a bit strange when it comes to breaks, in particular you need a break in an empty block so that you can put the caret there, so it creates a bogus break to achieve this.

However when you insert a break (e.g. by pressing Enter in an empty Compose window) this bogus break gets deleted, and the net result is that nothing happens, which is unhelpful.

Inserting a break by pressing Shift+Enter does force the insert of a break without deleting the bogus break, but there's no direct function to achieve this, and the one API point we have behaves differently in plain text compose.

So I think the best solution here is to move this extra break to the beginning of the if (reply_on_top == 1) block, with a suitable comment to the effect that the HTML editor eats the initial break.

(On an unrelated note, the code to detect a signature is over-engineered - any signature is already in aSignature at this point.)

[Brian Norris (computersforpeace)]

(In reply to <email address hidden> from comment #34)
> So I think the best solution here is to move this extra break to the
> beginning of the if (reply_on_top == 1) block, with a suitable comment to
> the effect that the HTML editor eats the initial break.

So you'd like essentially the same patch, just moved around and with an extra comments? I'll respin, retest, and flag you for review shortly.

[Brian Norris (computersforpeace)]

Created attachment 8451310
Patch: adds back a newline for HTML replies (version 2)

Adding version 2 patch. Essentially the same, refactorerd to address comments. Please review, Neil.

[Patrick Cloke (clokep)]

https://hg.mozilla.org/comm-central/rev/e1e1955cce9c
https://hg.mozilla.org/comm-central/rev/64e49d15dff7

[Brian Norris (computersforpeace)]

My patch was finally merged. I think it should show up in Thunderbird 36 (?). Let's see if they'll backport it to a version we're actually using...

[Michal Stanke (mstanke)]

May I ask, will be this bug fixed in next 31.x, or waiting for the next major release?

[Rsx11m-pub (rsx11m-pub)]

Comment on attachment 8451310
Patch: adds back a newline for HTML replies (version 2)

Let's find out, this should be simple enough for a branch landing.
Patch seems to apply to comm-esr31 by visual inspection.

[Approval Request Comment]
Regression caused by (bug #): bug 735380
User impact if declined: blank-space regression in replies for certain cases
Testing completed (on c-c, etc.): sufficient baking on all channels
Risk to taking this patch (and alternatives if risky): appears low

[Rsx11m-pub (rsx11m-pub)]

Comment on attachment 728622
Test fix

[Approval Request Comment]
Test adjustments implied by attachment 728622.

[Rsx11m-pub (rsx11m-pub)]

(on the other hand, we have 31.5 and 31.6 left, then comes 38.0 already...)

[Rkent-3 (rkent-3)]

Comment on attachment 8451310
Patch: adds back a newline for HTML replies (version 2)

Pushed https://hg.mozilla.org/releases/comm-esr31/rev/8dda16e711da

[Rkent-3 (rkent-3)]

Comment on attachment 728622
Test fix

Pushed https://hg.mozilla.org/releases/comm-esr31/rev/0b5a99cc7cc3

[5gmol1 (5gmol1)]

Just wanted to thank those that kept at this. Seemed so simple, but obviously not. Much appreciated.

[Paul White (paulw2u)]

Upstream bug showing "RESOLVED FIXED" on 2014-11-07
Target release: Thunderbird 36.0
Tested ok using Ubuntu 18.04 and Thunderbird 60.2.1
Closing by marking "Fix Released"