Numeric inputs have the wrong precision
Bug #607218 reported by
Tristan Seligmann
This bug affects 1 person
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Methanal |
Triaged
|
Low
|
Unassigned |
Bug Description
The internal value for IntegerInput, DecimalInput, and other derived inputs, is stored as a JavaScript number. In most (all?) implementations, numbers are stored as double-precision binary floating point values. However, integer values are stored Python-side as Python ints (or longs), which have exact integer precision, and infinite range (although Axiom has range limits. Also, decimal values are stored as Decimal values on the Python side, which have prec=28 in the default context. Thus, values lose precision when sent from the server to the client; and values entered by the user also lose precision compared with the server-side capability.
Changed in methanal: | |
status: | New → Triaged |
importance: | Undecided → Low |
milestone: | none → 0.3.0 |
To post a comment you must log in.