libdbi / libdbd-sqlite3 returns incorrect values for several data types

Bug #673307 reported by SatPhil on 2010-11-10
62
This bug affects 10 people
Affects Status Importance Assigned to Milestone
GnuCash
Fix Released
High
libdbi (Debian)
Fix Released
Unknown
libdbi (Ubuntu)
High
Clint Byrum
Maverick
High
Clint Byrum
libdbi-drivers (Fedora)
Fix Released
Medium

Bug Description

This is related to
- Gnome's bugzilla bug 611936 https://bugzilla.gnome.org/show_bug.cgi?id=611936
- https://bugzilla.redhat.com/show_bug.cgi?id=629964 in Fedora's bugtracker

It seems that the 8.3 version of libdbd-sqlite3 delivered with Ubuntu 10.10 i386 is complied with optimisation -ffast-math which causes all the numeric values in gnucash 2.3.15 (svn r19751) with SQLite backend to be retrieved as zero.

This problem only started after I upgraded from Ubuntu 10.04 to 10.10.

ProblemType: Bug
DistroRelease: Ubuntu 10.10
Package: libdbd-sqlite3 0.8.3-1-0ubuntu1
ProcVersionSignature: Ubuntu 2.6.35-22.35-generic 2.6.35.4
Uname: Linux 2.6.35-22-generic i686
Architecture: i386
Date: Wed Nov 10 11:40:17 2010
InstallationMedia: Ubuntu 10.04 LTS "Lucid Lynx" - Release i386 (20100427.1)
ProcEnviron:
 LANG=en_AU.utf8
 SHELL=/bin/bash
SourcePackage: libdbi-drivers

== SRU Information ==

=== IMPACT ===

This bug prevents any programs that use floating point numbers in a libdbi accessed database from functioning properly on i386. This includes GNUcash, a widely used piece of accounting software.

=== DEV FIX ===

This was fixed by turning off fast-math optimizations in libdbi.

=== PATCH ===

See linked merge proposal against maverick

=== TEST CASE: ===

This only affects the i386 architecture, and so much be run on an i386 machine/chroot/vm.

1. sudo apt-get install libdbd-sqlite3 -- note this should install libdbi0 as well
2. sudo apt-get build-dep libdbi-drivers
3. apt-get source libdbi-drivers
4. cd into libdbi-drivers-$version
5. run 'debian/rules build'
6. run 'make check'
7. Accept the default answer for libdbidriver directory of /usr/lib/dbd
8. type in 'sqlite3' to answer 'test which driver?'
9. Accept the default answer for database directory and database name
10. Tests will run with "SUCCESS", but the test for double floats will look like this:

the_double: in:1.7976931348623157E+307 out:0.000000e+00<<

11. Upgrade libdbi0 to the fixed package, then rerun steps 6 - 10, the test should now look like this:

the_double: in:1.7976931348623157E+307 out:1.797693e+307<<

=== REGRESSION POTENTIAL ===

This is a change only in compile flags, and so the only regression potential is that it will slow down all floating point operations in libdbi. Given that the alternative is the wrong answer, slower is actually better in this case.

Related branches

Description of problem:
Problems with GnuCash sqlite losing transactions data.
See GnuCash bug 611936 and gnucash-devel thread "Bug 611936 and the 2.4.0 release" for a discussion of the bug.

How reproducible:
Only occurs with Fedora build of libdbi.

Steps to Reproduce:
1. Install libdbi and libdbi-drivers using YUM
2. Obtain latest GnuCash svn, build and install.
3. Save to sqlite, close GnuCash, re-open saved data.

Actual results:
Loss of data as described in thread and GnuCash bug.

Additional info:
Removing rpm libraries and compiling from source using "make debug" cures the problem. Apparently this does not occur with other other library distributions.

Apologies for referring to GnuCash mailing list and bugzilla but all the info is already there.

Um ... could you provide URLs, rather than expecting me to know where to find this stuff? Assume I've never heard of GnuCash.

I suppose there's probably a version of GnuCash available in Fedora. Does the problem happen with that, rather than a custom build?

Oh, sure sorry.

https://bugzilla.gnome.org/show_bug.cgi?id=611936
And the start of the thread is at;
https://lists.gnucash.org/pipermail/gnucash-devel/2010-August/029141.html

It is rather a long thread however.

This applies to the upcoming release of GnuCash 2.4, in development at 2.3.14 at the moment.
The FC12 package is for Gnucash version 2.2, gnucash-2.2.9-4.fc12.i686 and the sqlite database is not part of that package. I can't comment on later Fedora releases I'm afraid.

Hm, so -fno-fast-math makes it go away? Seems a bit odd, but I'm more than willing to nuke any "optimization" option that upstream is unwilling to include in their debug builds.

I tried this on my Fedora 13 x86_64 box, using the gnucash that comes with F13 (gnucash-2.3.13-1.fc13.x86_64), and didn't see a problem. So either I'm testing wrongly, 2.3.13 doesn't have the problem, or it only occurs on 32-bit builds. Any thoughts which is the most likely path to pursue?

... and the answer is that it's 32-bit-specific, or at least I can reproduce the data corruption on i686 F-13 and not x86_64. I don't see the other warnings that were mentioned in the upstream thread, though.

Getting rid of -ffast-math does appear to fix it. I'll spin new RPMs with that fix shortly. Please test.

libdbi-drivers-0.8.3-6.fc13,libdbi-0.8.3-4.fc13 has been submitted as an update for Fedora 13.
https://admin.fedoraproject.org/updates/libdbi-drivers-0.8.3-6.fc13,libdbi-0.8.3-4.fc13

libdbi-drivers-0.8.3-6.fc12,libdbi-0.8.3-4.fc12 has been submitted as an update for Fedora 12.
https://admin.fedoraproject.org/updates/libdbi-drivers-0.8.3-6.fc12,libdbi-0.8.3-4.fc12

libdbi-drivers-0.8.3-6.fc14,libdbi-0.8.3-4.fc14 has been submitted as an update for Fedora 14.
https://admin.fedoraproject.org/updates/libdbi-drivers-0.8.3-6.fc14,libdbi-0.8.3-4.fc14

(In reply to comment #5)
> ... and the answer is that it's 32-bit-specific, or at least I can reproduce
> the data corruption on i686 F-13 and not x86_64. I don't see the other
> warnings that were mentioned in the upstream thread, though.
The other warnings I was getting was (I think) due to my GnuCash SVN tree getting corrupted, somethind I'd done elsewhere. It went away when I check out a fresh SVN tree.
>
> Getting rid of -ffast-math does appear to fix it. I'll spin new RPMs with that
> fix shortly. Please test.
Will do.

Tom,

Thank you for your incredibly swift response. I have tested the updated packages on Fedora 13 i386 and can confirm they fix this bug.

Is there anything I can do to speed up the process of getting them included in the official updates ?

(In reply to comment #10)
> Is there anything I can do to speed up the process of getting them included in
> the official updates ?

Once the packages are pushed to updates-testing, you can add +1 karma to them (add a "works for me" comment to their bodhi page, which will have a link here once the push occurs).

This works for me on FC12 using GnuCash. Thanks Tom for very quick work.

libdbi-drivers-0.8.3-6.fc14, libdbi-0.8.3-4.fc14 has been pushed to the Fedora 14 testing repository. If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with
 su -c 'yum --enablerepo=updates-testing update libdbi-drivers libdbi'. You can provide feedback for this update here: https://admin.fedoraproject.org/updates/libdbi-drivers-0.8.3-6.fc14,libdbi-0.8.3-4.fc14

Will this be pushed to FC13 as well ? Or should I add a worksforme comment on the FC14 admin page ? I have only tested it in FC13 so far.

Yes, it's in the queue for F13 and F12 too. What with the holiday weekend, I wasn't expecting to see *any* of those pushes happen before today ;-). Have patience, it should happen in the next day or two.

libdbi-drivers-0.8.3-6.fc14, libdbi-0.8.3-4.fc14 has been pushed to the Fedora 14 stable repository. If problems still persist, please make note of it in this bug report.

libdbi-drivers-0.8.3-6.fc12, libdbi-0.8.3-4.fc12 has been pushed to the Fedora 12 stable repository. If problems still persist, please make note of it in this bug report.

libdbi-drivers-0.8.3-6.fc13, libdbi-0.8.3-4.fc13 has been pushed to the Fedora 13 stable repository. If problems still persist, please make note of it in this bug report.

SatPhil (satphil) wrote :
SatPhil (satphil) wrote :

Error in /tmp/gnucash.trace:

* 22:22:23 CRIT <gnc.engine> xaccSplitSetValue: assertion `gnc_numeric_check(amt) == GNC_ERROR_OK' failed
* 22:22:23 CRIT <gnc.engine> xaccSplitSetAmount: assertion `gnc_numeric_check(amt) == GNC_ERROR_OK' failed

Jeff (jeffsf) wrote :

libdbd is VERY broken as distributed for 10.10.

in libdbi-drivers-0.8.3-1/tests make and run test_dbi

Not the differences in output between Ubuntu and FreeBSD. In Ubuntu, there are clear failures with:
* longlong
* ulonglong
* double
* datetime
* datetime_tz

jeff@fx:~$ diff -u test_dbi.output*
--- test_dbi.output 2011-01-01 10:42:03.707828037 -0800
+++ test_dbi.output.freebsd 2011-01-01 10:43:55.856451151 -0800
@@ -2,15 +2,15 @@
 Driver information:
 -------------------
  Name: sqlite3
- Filename: /usr/lib/dbd/libdbdsqlite3.so
+ Filename: /usr/local/lib/dbd/libdbdsqlite3.so
  Desc: SQLite3 database support (using libsqlite3)
  Maintainer: Markus Hoenicka <email address hidden>
  URL: http://libdbi-drivers.sourceforge.net
  Version: dbd_sqlite3 v0.8.3-1
- Compiled: Aug 7 2010
+ Compiled: Jan 1 2011

 Successfully connected!
- Using database engine version 30702 (numeric) and 3.7.2 (string)
+ Using database engine version 30704 (numeric) and 3.7.4 (string)

 Test 1: List databases:
  Got result, try to access rows
@@ -47,10 +47,10 @@
 the_ushort: in:32767 out:32767<<
 the_long: in:-2147483648 out:-2147483648<<
 the_ulong: in:2147483647 out:2147483647<<
-the_longlong: in:-9223372036854775807 out:0<<
-the_ulonglong: in:9223372036854775807 out:0<<
+the_longlong: in:-9223372036854775807 out:-9223372036854775807<<
+the_ulonglong: in:9223372036854775807 out:9223372036854775807<<
 the_float: in:3.402823466E+38 out:3.402823e+38<<
-the_double: in:1.7976931348623157E+307 out:0.000000e+00<<
+the_double: in:1.7976931348623157E+307 out:1.797693e+307<<
 the_driver_string: in:'Can 'we' "quote" this properly?' out:'Can 'we' "quote" this properly?'<<
 the_quoted_string: in:'Can 'we' "quote" this properly?' out:'Can 'we' "quote" this properly?'<<
 the_quoted_string_copy: in:'Can 'we' "quote" this properly?' out:'Can 'we' "quote" this properly?'<<
@@ -58,11 +58,11 @@
 the_escaped_string_copy: in:'Can 'we' "escape" this properly?' out:'Can 'we' "escape" this properly?'<<
 the_empty_string: out:''<<
 the_null_string: out:'(null)'
-the_datetime: in:'2001-12-31 23:59:59' out:2001-12-31 0:0:0
-the_datetime_tz: in:'2001-12-31 23:59:59 -10:00' out:2001-12-31 0:0:0
+the_datetime: in:'2001-12-31 23:59:59' out:2001-12-31 23:59:59
+the_datetime_tz: in:'2001-12-31 23:59:59 -10:00' out:2002-1-1 9:59:59
 the_date: in:'2001-12-31' out:2001-12-31
-the_time: in:'23:59:59' out:0:0:0
-the_time_tz: in:'23:59:59-10:00' out:0:0:0
+the_time: in:'23:59:59' out:23:59:59
+the_time_tz: in:'23:59:59-10:00' out:9:59:59
 the_quoted_binary: in: 65-66-0-67-39-68- out: 65-66-0-67-39-68-<<
 the_escaped_binary_copy: in: 65-66-0-67-39-68- out: 65-66-0-67-39-68-<<

Jeff (jeffsf) wrote :

Compiling libdbi and libdbd-sqlite3 without the -ffast-math optimization seems to resolve the issue.

Compiling libdbd-sqlite3 alone without the -ffast-math optimization did not seem to resolve the issue.

libdbi could not be built on the local machine, even after apt-get build-dep libdbi, without use of the ./configure --disable-docs

summary: - gnucash 2.3 with sqlite retrieves all numbers as zero
+ libdbi / libdbd-sqlite3 returns incorrect values for several data types
Jeff (jeffsf) on 2011-01-01
Changed in libdbi-drivers (Ubuntu):
status: New → Confirmed
Clint Byrum (clint-fewbar) wrote :

Setting Importance to High. I think this only affects maverick, as the build in natty runs the full test suite for mysql, pgsql, and sqlite3 and does not report this failure.

Changed in libdbi-drivers (Ubuntu):
importance: Undecided → High
tags: added: regression-release
Changed in libdbi-drivers (Ubuntu Maverick):
status: New → Confirmed
importance: Undecided → High
status: Confirmed → Triaged
Clint Byrum (clint-fewbar) wrote :

Oh so it appears the test suite actually doesn't "fail", its just letting these incorrect numbers through. Crazy.

The Fedora fix seems to be the way to go, will push that out ASAP. I actually think this may affect natty too.

Changed in libdbi-drivers (Ubuntu):
status: Confirmed → In Progress
assignee: nobody → Clint Byrum (clint-fewbar)
Clint Byrum (clint-fewbar) wrote :

This is actually a bug in libdbi, not libdbi-drivers. Redirecting as such.

affects: libdbi-drivers (Ubuntu) → libdbi (Ubuntu)
Changed in libdbi (Debian):
status: Unknown → New
Changed in gnucash:
importance: Unknown → High
status: Unknown → Confirmed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package libdbi - 0.8.3-0ubuntu2

---------------
libdbi (0.8.3-0ubuntu2) natty; urgency=low

  * configure.in, configure: patched CFLAGS to pass -fno-fast-math to
    prevent floating point problems on i386 (LP: #673307)
 -- Clint Byrum <email address hidden> Tue, 01 Feb 2011 14:43:43 +0100

Changed in libdbi (Ubuntu):
status: In Progress → Fix Released

Clint, thanks for finding the time to implement the fix, it is much
appreciated.

Given the problem is a bit of a showstopper (all your numbers and dates come
back as zero), will this fix be available as a patch for Ubuntu 10.10
Maverick or do I need to use a tool like Prevu to backport it myself?

On Wed, Feb 2, 2011 at 12:50 AM, Launchpad Bug Tracker <
<email address hidden>> wrote:

> This bug was fixed in the package libdbi - 0.8.3-0ubuntu2
>
> ---------------
> libdbi (0.8.3-0ubuntu2) natty; urgency=low
>
> * configure.in, configure: patched CFLAGS to pass -fno-fast-math to
> prevent floating point problems on i386 (LP: #673307)
> -- Clint Byrum <email address hidden> Tue, 01 Feb 2011 14:43:43 +0100
>
> ** Changed in: libdbi (Ubuntu)
> Status: In Progress => Fix Released
>
> --
> You received this bug notification because you are a direct subscriber
> of the bug.
> https://bugs.launchpad.net/bugs/673307
>
> Title:
> libdbi / libdbd-sqlite3 returns incorrect values for several data
> types
>

Clint Byrum (clint-fewbar) wrote :

SatPhil, yes there is definitely a plan to fix this soon in maverick. You might see that it has a separate status for maverick which is "triaged", meaning it is acknowledged for stable release update.

On Feb 5, 2011, at 5:45 PM, SatPhil <email address hidden> wrote:

> Clint, thanks for finding the time to implement the fix, it is much
> appreciated.
>
> Given the problem is a bit of a showstopper (all your numbers and dates come
> back as zero), will this fix be available as a patch for Ubuntu 10.10
> Maverick or do I need to use a tool like Prevu to backport it myself?
>
> On Wed, Feb 2, 2011 at 12:50 AM, Launchpad Bug Tracker <
> <email address hidden>> wrote:
>
>> This bug was fixed in the package libdbi - 0.8.3-0ubuntu2
>>
>> ---------------
>> libdbi (0.8.3-0ubuntu2) natty; urgency=low
>>
>> * configure.in, configure: patched CFLAGS to pass -fno-fast-math to
>> prevent floating point problems on i386 (LP: #673307)
>> -- Clint Byrum <email address hidden> Tue, 01 Feb 2011 14:43:43 +0100
>>
>> ** Changed in: libdbi (Ubuntu)
>> Status: In Progress => Fix Released
>>
>> --
>> You received this bug notification because you are a direct subscriber
>> of the bug.
>> https://bugs.launchpad.net/bugs/673307
>>
>> Title:
>> libdbi / libdbd-sqlite3 returns incorrect values for several data
>> types
>>
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/673307
>
> Title:
> libdbi / libdbd-sqlite3 returns incorrect values for several data
> types
>
> Status in GnuCash - Finance manager:
> Confirmed
> Status in “libdbi” package in Ubuntu:
> Fix Released
> Status in “libdbi” source package in Maverick:
> Triaged
> Status in “libdbi” package in Debian:
> New
> Status in “libdbi-drivers” package in Fedora:
> Unknown
>
> Bug description:
> This is related to
> - Gnome's bugzilla bug 611936 https://bugzilla.gnome.org/show_bug.cgi?id=611936
> - https://bugzilla.redhat.com/show_bug.cgi?id=629964 in Fedora's bugtracker
>
> It seems that the 8.3 version of libdbd-sqlite3 delivered with Ubuntu
> 10.10 i386 is complied with optimisation -ffast-math which causes all
> the numeric values in gnucash 2.3.15 (svn r19751) with SQLite backend
> to be retrieved as zero.
>
> This problem only started after I upgraded from Ubuntu 10.04 to 10.10.
>
> ProblemType: Bug
> DistroRelease: Ubuntu 10.10
> Package: libdbd-sqlite3 0.8.3-1-0ubuntu1
> ProcVersionSignature: Ubuntu 2.6.35-22.35-generic 2.6.35.4
> Uname: Linux 2.6.35-22-generic i686
> Architecture: i386
> Date: Wed Nov 10 11:40:17 2010
> InstallationMedia: Ubuntu 10.04 LTS "Lucid Lynx" - Release i386 (20100427.1)
> ProcEnviron:
> LANG=en_AU.utf8
> SHELL=/bin/bash
> SourcePackage: libdbi-drivers
>
>

Changed in gnucash:
status: Confirmed → Fix Released
description: updated
Clint Byrum (clint-fewbar) wrote :

For those interested, I've submitted the fix for merge, it should be looked at fairly soon by a sponsor who can upload it to maverick-proposed. There are limited resources to verify SRU's, so it will help immensely if multiple members of the community can verify the fix when it hits proposed and post those verification reports here once the 'verification-needed' tag has been added.

Changed in libdbi (Ubuntu Maverick):
assignee: nobody → Clint Byrum (clint-fewbar)
status: Triaged → In Progress
SatPhil (satphil) wrote :

Thanks again for working on this, Clint. Will do my best to verify once the
word is out.

Would the gcc folks be interested in this case of FASTMATH optimisation
breaking functionality? Or would it be more likely to be dodgy programming
on the part of libdbi? How do you suggest we proceed?

Benjamin Drung (bdrung) wrote :

Uploaded libdbi 0.8.3-0ubuntu1.1 to maverick-proposed.

Accepted libdbi into maverick-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Changed in libdbi (Ubuntu Maverick):
status: In Progress → Fix Committed
tags: added: verification-needed

I can confirm this fixes the bug.

I upgraded from libdbi0_0.8.3-0ubuntu1 to libdbi0_0.8.3-0ubuntu1.1 on Ubuntu
10.10 i386 and the "make check" script now returns the expected values (i.e.
non-zero doubles). Moreover, Gnucash 2.4.2 which now looks the error and
had been exiting with a pop-up error message now continues to run and
displays non-zero values for its numbers. Thanks to all for your assistance
in sorting this problem out.

On Tue, Feb 15, 2011 at 9:29 PM, Launchpad Bug Tracker <
<email address hidden>> wrote:

> ** Branch linked: lp:ubuntu/maverick-proposed/libdbi
>
> --
> You received this bug notification because you are a direct subscriber
> of the bug.
> https://bugs.launchpad.net/bugs/673307
>
> Title:
> libdbi / libdbd-sqlite3 returns incorrect values for several data
> types
>

Martin Pitt (pitti) on 2011-02-16
tags: added: verification-done
removed: verification-needed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package libdbi - 0.8.3-0ubuntu1.1

---------------
libdbi (0.8.3-0ubuntu1.1) maverick-proposed; urgency=low

  * configure.in, configure: patched CFLAGS to pass -fno-fast-math to
    prevent floating point problems on i386 (LP: #673307)
 -- Clint Byrum <email address hidden> Sat, 12 Feb 2011 15:51:51 +0100

Changed in libdbi (Ubuntu Maverick):
status: Fix Committed → Fix Released
Changed in libdbi (Debian):
status: New → Fix Released
tags: added: testcase
Changed in libdbi-drivers (Fedora):
importance: Unknown → Medium
status: Unknown → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.