If you run the following expression 102 * 0.0254 you will get 2.59079999999998 but the answer is 2.5908 But yet, multiple 102 * 0.0253 or 102*0.0255 or 102*0.0256 and you get the right answer!! Just with this number. This fails on both the emulator for eclipse and on my actual htc phone. Help! Can anyone help me solve this problem?
Welcome to the world of Java and double precision numbers. Same thing happens in Windows and everywhere else. Its not a "bug" but the way Java double precision numbers are implemented.
I'm basically trying to make a simple calculator and found this number totally at random. The calculator on my phone calculates this properly so why can't my app? I guess i'm trying to get is what are the workarounds for this as I take it there must be for the calculator software to work properly.
even BigDecimal isn't perfect, but it works much better. We typically don't worry about the rounding error too much since we usually round to a particular level of precision when we display a number to the user, but that doesn't work so well with a calculator app. I've written write code that'll take a number like 2.59079999999998 and figure out its really 2.5908 for the cases where we don't know the level of precision beforehand.