Your Digital Media Has Never Looked So Good

 
User avatar
Komag
Topic Author
Posts: 746
Joined: Fri Aug 22, 2014 3:42 am

Issue with math: float calculation, ToStr()

Mon Oct 29, 2018 1:46 am

Try this:
? 1.1* 9
console prints 9.9

But try this:
? str(1.1 * 9)
console prints 9.900001

I assume this is some sort of floating point rounding error having to do with binary math and how computers work, etc etc. But it's a problem for me, because in my game when the player is holding 9 spider abdomens in his or her inventory, on the list the weight is showing up at 9.900001, when everything else is just one decimal point, not six!

Any way to force a float to only be one decimal point accurate?

Is my only recourse to always check the final string and count characters after the "." ?
 
User avatar
Komag
Topic Author
Posts: 746
Joined: Fri Aug 22, 2014 3:42 am

Re: Issue with math: float calculation, ToStr()

Mon Oct 29, 2018 2:25 am

I just implemented this to brute force solve it:

FUNCTION checkStrFloat(Str AS STRING) AS STRING ' trig by getItemDscr(1), c2Inventory(1)   ' Search for a . in the string, then check if the string is longer than one character past the . and if so, trim off the rest
   StrLen = Str.Len()
   FOR i = 1 TO StrLen
      chr = Str.Mid(i -1, 1)                        ' String.Mid() is 0 based, not 1 based, so need to -1
      IF chr = "."                                                            ' Ex: "9.900001", i will be 2
         IF StrLen > i +1                                                      ' If there is more than just one character after the decimal point
            StrNEW = Left(Str, i+1)                  ' Left() is 1 based      ' Ex: i +1 will be 3, thus first three characters of "9.900001" will be "9.9", one character past the "."
            ? "checkStrFloat() found long float Str, orig " Str ", shortened " StrNew
            RETURN StrNEW
         END IF
      END IF
   END FOR
   RETURN Str
END FUNCTION
 
belltown
Posts: 1461
Joined: Thu Dec 09, 2010 1:43 pm
Contact:

Re: Issue with math: float calculation, ToStr()

Mon Oct 29, 2018 4:58 pm

Due to the way floating point numbers are implemented, your number could be "9.899999", in which case your function will return "9.8" when it should return "9.9".

I'd do something like this (assuming your numbers are stored internally as Doubles, but you want to print them to one significant decimal place):

function convert(f as double) as string
  a = CInt(f * 10)
  n = a \ 10
  m = abs(a) mod 10
  return n.toStr() + "." + m.toStr()
end function
https://github.com/belltown/
 
User avatar
RokuKC
Posts: 305
Joined: Wed Sep 10, 2014 10:44 am
Location: Roku HQ

Re: Issue with math: float calculation, ToStr()

Mon Oct 29, 2018 5:07 pm

Komag wrote:
Try this:
? 1.1* 9
console prints 9.9

But try this:
? str(1.1 * 9)
console prints 9.900001


A couple of options:
x=1.1 * 9

' 1) use Fix to try to force 1 digit of precision.  may or may not work?
? Str(Fix(x * 10)/10)

' 2) use printf-style conversion
? x.ToStr("%3.1f")

' Note that Str prefixes a blank on non-negative numbers, the above ToStr does not.
' if you want that, use x.ToStr("% 3.1f")
 
renojim
** Valued Community Member **
Posts: 3360
Joined: Mon Feb 15, 2010 1:35 pm

Re: Issue with math: float calculation, ToStr()

Mon Oct 29, 2018 8:57 pm

My advice is to only use integer math unless there's a compelling reason not to (which there never is). If you only want one decimal point of precision then multiply everything by ten (9 spiders at 11 tenths per abdomen is 99 tenths). There's any number of ways to display 99 tenths as 9.9.

-JT
 
User avatar
squirreltown
Posts: 829
Joined: Sun Apr 21, 2013 2:20 pm

Re: Issue with math: float calculation, ToStr()

Tue Oct 30, 2018 8:20 am

renojim wrote:
My advice is to only use integer math unless there's a compelling reason not to (which there never is).  

Animation might be an exception.
Kinetics Screensavers
 
User avatar
Komag
Topic Author
Posts: 746
Joined: Fri Aug 22, 2014 3:42 am

Re: Issue with math: float calculation, ToStr()

Tue Oct 30, 2018 10:00 am

For some reason I had the impression that x.ToStr() was only for integers, not floats. Because that actually prints correctly:
? str(1.1*9)
 9.900001
? (1.1*9).tostr()
9.9

Thanks for the suggestions. I like the idea of always doing all math with integers, but at this point I don't want to go back and rework everything, so I'm going to go with "laundering" all my floats before converting to strings, with:


FUNCTION washFloat(dirtyFloat)
 Int10 = Cint(dirtyFloat *10)
 cleanFloat = Int10 / 10
 RETURN cleanFloat
END FUNCTION

And then I'll use float.ToStr() for double measure (unless there are Roku models or firmware on which that won't work?)
 
User avatar
RokuKC
Posts: 305
Joined: Wed Sep 10, 2014 10:44 am
Location: Roku HQ

Re: Issue with math: float calculation, ToStr()

Tue Oct 30, 2018 6:18 pm

Komag wrote:
And then I'll use float.ToStr() for double measure (unless there are Roku models or firmware on which that won't work?)


Default ToStr() on intrinsic types (numeric, Boolean, and string) goes back to firmware 7.0.
ToStr and string Format with printf format specifiers goes back to firmware 7.6.

Who is online

Users browsing this forum: No registered users and 3 guests