[Upstream] Calc mis-sorts left to right (sort columns)

[Wolf Rogner]

1) lsb_release -rd
Description: Ubuntu 12.10
Release: 12.10

2) apt-cache policy libreoffice-calc
libreoffice-calc:
  Installed: 1:3.6.2~rc2-0ubuntu4
  Candidate: 1:3.6.2~rc2-0ubuntu4
  Version table:
 *** 1:3.6.2~rc2-0ubuntu4 0
        900 http://archive.ubuntu.com/ubuntu/ quantal-updates/main i386 Packages
        100 /var/lib/dpkg/status
     1:3.6.2~rc2-0ubuntu3 0
        500 http://archive.ubuntu.com/ubuntu/ quantal/main i386 Packages

3) What is expected to happen in Calc via a terminal:
cd ~/Desktop && wget https://bugs.launchpad.net/ubuntu/+source/libreoffice/+bug/1082449/+attachment/3442313/+files/test.xls && localc --nologo test.xls

highlight columns B through G -> Data -> Sort... -> Options tab under Direction click radio button Let to right (sort columns) -> uncheck checkbox Range contains row labels -> tab Sort Criteria -> drop down Sort key 1 click Row 1 -> radio buton Ascending -> button OK and the order is sorted as noted in the file Expected.

4) What happens instead is it sorts as noted in the file Result.

+ Same problem reproducible in:
Microsoft Windows Vista Business x86
6.0.6002 Service Pack 2 Build 6002
Version 3.6.3.2 (Build ID: 58f22d5)

but issue has WORKAROUND if you run sort twice consecutively.

WORKAROUND: Use Gnumeric.

apt-cache policy gnumeric
gnumeric:
  Installed: 1.10.17-1.1ubuntu1
  Candidate: 1.10.17-1.1ubuntu1
  Version table:
 *** 1.10.17-1.1ubuntu1 0
        500 http://archive.ubuntu.com/ubuntu/ quantal/universe i386 Packages
        100 /var/lib/dpkg/status

WORKAROUND: Use Excel via WINE.
Microsoft Office Professional Plus 2010 Excel Version 14.0.6023.1000 (32-bit)

apt-cache policy wine1.5
wine1.5:
  Installed: 1.5.17-0ubuntu4
  Candidate: 1.5.17-0ubuntu4
  Version table:
 *** 1.5.17-0ubuntu4 0
        500 http://ppa.launchpad.net/ubuntu-wine/ppa/ubuntu/ quantal/main i386 Packages
        100 /var/lib/dpkg/status

ProblemType: Bug
DistroRelease: Ubuntu 12.10
Package: libreoffice-calc 1:3.6.2~rc2-0ubuntu4
ProcVersionSignature: Ubuntu 3.5.0-19.30-generic 3.5.7
Uname: Linux 3.5.0-19-generic x86_64
NonfreeKernelModules: nvidia wl
ApportVersion: 2.6.1-0ubuntu6
Architecture: amd64
Date: Fri Nov 23 17:47:55 2012
InstallationDate: Installed on 2011-10-16 (404 days ago)
InstallationMedia: Ubuntu 11.10 "Oneiric Ocelot" - Release amd64+mac (20111012)
MarkForUpload: True
SourcePackage: libreoffice
UpgradeStatus: Upgraded to quantal on 2012-10-19 (34 days ago)

[Wolf Rogner (war-rsb)]

Sorry, the report lost its formating but the testfile includes all expected and real results.

[Wolf Rogner (war-rsb)]

Just verified this with calc 3.5.4.2 (from 12.04):

The results are correct there.

This error occurs in 3.6.2.2 (from 12.10)

[Wolf Rogner (war-rsb)]

Just make it clear: This is a regression

[penalvch (penalvch)]

Wolf Rogner, thank you for taking the time to report this bug and helping to make Ubuntu better. The issue you are reporting is an upstream one and it would be nice if somebody having it could send the bug to the developers of the software by following the instructions verbatim at http://wiki.documentfoundation.org/BugReport . If you have done so, please tell us the number of the upstream bug (or the link), so we can add a bugwatch that will inform us about the status. Thanks in advance.

Helpful bug reporting tips:
https://help.ubuntu.com/community/ReportingBugs

[Wolf Rogner (war-rsb)]

Created attachment 70488
Testfile to demonstrate behaviour

Problem description:
Calc sorts columns the wrong way

Steps to reproduce:
There is an error in sorting columns in calc.

How to test:
Create a spread sheet similar to this

   C A D B F E
R1 a d
R2 b e
R3 c f

Mark the columns labled C, A...E
Sort the columns ascending

Expected behavior:

   A B C D E F
R1 a d
R2 b e
R3 c f

Current behavior:

   C A D B F E
R1 c f
R2 a d
R3 b e

Platform (if different from the browser):
Macbook Pro 6,2, Ubuntu 12.10

Browser: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:17.0) Gecko/17.0 Firefox/17.0

[Wolf Rogner (war-rsb)]

I filed the bug with Libreoffice

https://www.libreoffice.org/bugzilla/show_bug.cgi?id=57465

[Wolf Rogner (war-rsb)]

as for the suggested workaround: double sort

This does not work under Ubuntu 12.10 / LibreOffice 3.6.2.2

sorry

Excel is not a valid workaround as the purpose of using open solutions is to NOT use proprietary systems.
Gnumeric might me a workaround for some (but not for me).

[Wolf Rogner (war-rsb)]

This does not happen in Calc 3.5.4.2

[Davian818 (davian818)]

Confirmed in 4.0-master, though the result is ADCBFE.

[Markus Mohrhard (moggi)]

Can you please test with 3.6.4.1 or 3.6.4.2 because they should be fixed there.

[Wolf Rogner (war-rsb)]

@Markus Mohrhard:
I'd love to whenever the update is downstreamed into Ubuntu repositories.

Q: How can such a regression happen? Any ideas or explanations?

[Wolf Rogner (war-rsb)]

This should be fixed in 3.6.4.2.

[Markus Mohrhard (moggi)]

(In reply to comment #4)
> @Markus Mohrhard:
> I'd love to whenever the update is downstreamed into Ubuntu repositories.
>
> Q: How can such a regression happen? Any ideas or explanations?

Calc has about 1 million lines of own source code + several more million lines of shared source code. And how much time did you spend yourself testing master to check for regressions.

Marking as fixed as I think it is fixed in 3.6.4.1 and therefore in the 3.6.4 release. Reopen if it is not fixed there.

[penalvch (penalvch)]

Wolf Rogner, thank you for reporting this upstream, and your comments. Regarding your comments https://bugs.launchpad.net/ubuntu/+source/libreoffice/+bug/1082449/comments/8 :
>"as for the suggested workaround: double sort This does not work under Ubuntu 12.10 / LibreOffice 3.6.2.2 sorry"

I also tried the Windows/3.6.3.2 work around in Ubuntu before editing the Bug Description, and noticed it did not work. This is why I did not create a third, separate "WORKAROUND:..." entry in the Bug Description.

>"Excel is not a valid workaround as the purpose of using open solutions is to NOT use proprietary systems."

For the purposes of satisfying the work around criteria noted at https://wiki.ubuntu.com/Bugs/Importance :
"Bugs that have easy work-arounds"

it is clear Excel would not be considered this, due to Excel not being provided by Ubuntu. This is why I marked this report Medium, instead of Low. The WORKAROUNDs in the Bug Description provide informational options one may potentially use while remaining booted in Ubuntu, irrespective of software license or cost. As well, noting this information provides ammo for getting this addressed sooner, as LibreOffice intends compatibility with Office. This would fit the criteria of expectation compatibility.

"Gnumeric might me a workaround for some (but not for me)."

Ok. Given Gnumeric is FOSS, providing this potentially gives developers code to base a patch off of, and as you noted, a work around for others until the patch lands in Ubuntu.

Thank you for your understanding.

[penalvch (penalvch)]

Reproducible in:
Microsoft Windows Vista Business x86
6.0.6002 Service Pack 2 Build 6002
Version 3.6.4.1 (Build ID: a9a0717)

[Wolf Rogner (war-rsb)]

(In reply to comment #5)
> (In reply to comment #4)
> > @Markus Mohrhard:
> > I'd love to whenever the update is downstreamed into Ubuntu repositories.
> >
> > Q: How can such a regression happen? Any ideas or explanations?
>
> Calc has about 1 million lines of own source code + several more million
> lines of shared source code. And how much time did you spend yourself
> testing master to check for regressions.
>
> Marking as fixed as I think it is fixed in 3.6.4.1 and therefore in the
> 3.6.4 release. Reopen if it is not fixed there.

First I am pretty sure that not all of the 1 million lines deal with sorting. Second, in my development projects we used test suites to prevent from regression and third there are statistical methods to identify areas of unstable code based on previous bug reports.

Assuming that all this is in place here as well, I was wondering what could be the cause of those regressions. If you had an idea I could direct some research effort into how this can be improved. Maybe we can discuss this on a different channel (mail?)

[Markus Mohrhard (moggi)]

(In reply to comment #7)
> (In reply to comment #5)
> > (In reply to comment #4)
> > > @Markus Mohrhard:
> > > I'd love to whenever the update is downstreamed into Ubuntu repositories.
> > >
> > > Q: How can such a regression happen? Any ideas or explanations?
> >
> > Calc has about 1 million lines of own source code + several more million
> > lines of shared source code. And how much time did you spend yourself
> > testing master to check for regressions.
> >
> > Marking as fixed as I think it is fixed in 3.6.4.1 and therefore in the
> > 3.6.4 release. Reopen if it is not fixed there.
>
> First I am pretty sure that not all of the 1 million lines deal with
> sorting. Second, in my development projects we used test suites to prevent
> from regression and third there are statistical methods to identify areas of
> unstable code based on previous bug reports.

We have highly coupled code with to less test code. Sadly our code is so old and written by so many people that even simple tasks like sorting affect more places than it should. For example sorting is highly integrated into calc core in ScTable so that a change to fix another bug can introduce a new one in parts that nobody expects.

We are writing new tests for fixed bugs but since we are mainly only three developers doing all the work on calc this means that there are thousand of untested cases. However we are always happy if someone helps us out with extending the test cases which is actually not that difficult.

>
> Assuming that all this is in place here as well, I was wondering what could
> be the cause of those regressions. If you had an idea I could direct some
> research effort into how this can be improved. Maybe we can discuss this on
> a different channel (mail?)

We were removing some limitations for sorting in 3.6 and therefore had to change some internal data structure and change the UI code. This was done by a volunteer and sadly not enough tests were written at the time. I would be very glad if you would be intersted in helping with that.

[Wolf Rogner (war-rsb)]

Hello Markus

I tried the same sorting on a version 3.6.4.3 downloaded from the web site and installed on a plain machine.

The sorting does not work here either.

Colums are sorted: ADCBFE

I did not have the chance to get a look into the sources. So I'm afraid I'm of little help here.

[Reynoldsp (reynoldsp)]

I was trying to sort columns in ascending/descending order by the row of totals at the bottom of the columns. I'm not sure what order LibreOffice 3.6.4.3 is sorting but it's certainly not numerical.

OpenOffice.org works fine for this. I currently have 3.3.0 installed because of an unrelated bug in 3.4.1.

This issue apparently goes back at least as far as LibreOffice 3.6.0.2. - see http://ask.libreoffice.org/en/question/5898/libreoffice-calc-sorts-numerical-data-as-text/

LibreOffice sorts vertically just fine.

I'm on Windows 7 Home Premium (64 bit).

[Wolf Rogner (war-rsb)]

I'v tried LOdev 4.0.

The same problem applies there as well. However, resorting immediately after the first sort produces the correct results.

[Libreoffice-bugs (libreoffice-bugs)]

Markus Mohrhard committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=7af886d7784b321e6cda2a870b514607e355b112

reset one of the sort containers before refilling, fdo#57465

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.

[Libreoffice-bugs (libreoffice-bugs)]

Markus Mohrhard committed a patch related to this issue.
It has been pushed to "libreoffice-4-0":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=baa9a119dabfc5e5dbb5c174cebf8c24298bad67&h=libreoffice-4-0

reset one of the sort containers before refilling, fdo#57465

It will be available in LibreOffice 4.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.

[Libreoffice-bugs (libreoffice-bugs)]

Markus Mohrhard committed a patch related to this issue.
It has been pushed to "libreoffice-3-6":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=c3f7589fb540072f3a2434f6de99fe2472dad3b1&h=libreoffice-3-6

reset one of the sort containers before refilling, fdo#57465

It will be available in LibreOffice 3.6.6.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.

[Libreoffice-bugs (libreoffice-bugs)]

Markus Mohrhard committed a patch related to this issue.
It has been pushed to "libreoffice-3-6-5":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=238f6797e47a5029ad8bd5df3dfc36bbeff6532c&h=libreoffice-3-6-5

reset one of the sort containers before refilling, fdo#57465

It will be available already in LibreOffice 3.6.5.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.

[Launchpad Janitor (janitor)]

This bug was fixed in the package libreoffice - 1:4.0.0~beta2-0ubuntu2

---------------
libreoffice (1:4.0.0~beta2-0ubuntu2) raring; urgency=low

  * Build using the internal libmspub for a first build.
 -- Matthias Klose <email address hidden> Tue, 19 Feb 2013 00:05:26 +0100

[Jmadero-dev (jmadero-dev)]

As part of regular FDO cleanup we are checking the version of regressions to see if it's the oldest version that displays the incorrect behavior.

In this case I have been able to verify the bug existence in
Version 3.6.0.0.beta1 (Build ID: 1f1cdd)

Changing version to reflect this

[Jmadero-dev (jmadero-dev)]

*** Bug 57797 has been marked as a duplicate of this bug. ***

[Libreoffice-commits (libreoffice-commits)]

Xisco Fauli committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/commit/fd745457be6371794ca8178631ecb3523c9474cd

tdf#57465: sc: Add UItest

It will be available in 7.2.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.