Comment 15 for bug 408694

Revision history for this message
Ross M. (rwmcfa1) wrote : Re: [Bug 408694] Re: main.func_math fails due to precision error

On Tue, Aug 4, 2009 at 9:26 PM, Ross McFarland<email address hidden> wrote:
> On Tue, Aug 4, 2009 at 7:50 PM, Ross McFarland<email address hidden> wrote:

> problem is in drizzle mystrings/dtoa.cc, dtoa

found it:

 res= dtoa(x, 4, type == MY_GCVT_ARG_DOUBLE ?
            min(width, DBL_DIG) : width,
            &decpt, &sign, &end);

should be:

 res= dtoa(x, 4, type == MY_GCVT_ARG_DOUBLE ?
            min(width, DBL_DIG) : min(width, FLT_DIG),
            &decpt, &sign, &end);

it was passing in width (the size of the buffer, 341) for ndigits
which is supposed to be the (max) number of digits to include in the
string (buffer should be this size or larger.) when 341 is passed it
tries to convert to a string that is higher precision than the double
it's copying from can reliably be used for (> DBL_DIG digits) and thus
gets noise. this just happens to work on linux since it gives out the
double value of exactly 2 in the first place.

good news is that fixes the log(3, 9) func_math test. bad news is that
others are now failing. looking at the failures i'm not 100% sure yet,
but i think the expected values may be wrong. digging yet more...

--
-rm