Numeric inputs have the wrong precision

Bug #607218 reported by Tristan Seligmann
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Methanal
Triaged
Low
Unassigned

Bug Description

The internal value for IntegerInput, DecimalInput, and other derived inputs, is stored as a JavaScript number. In most (all?) implementations, numbers are stored as double-precision binary floating point values. However, integer values are stored Python-side as Python ints (or longs), which have exact integer precision, and infinite range (although Axiom has range limits. Also, decimal values are stored as Decimal values on the Python side, which have prec=28 in the default context. Thus, values lose precision when sent from the server to the client; and values entered by the user also lose precision compared with the server-side capability.

Changed in methanal:
status: New → Triaged
importance: Undecided → Low
milestone: none → 0.3.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.