Page 1 of 1
precision problem?
Posted: 25 Mar 2015, 16:14
by partyboarder
Hi
i found this precision problem when using a float expression to divide 0.01 by 10
i know that mathematically 0.0009999999 is considered 0.001 but when you use the result in a textnode it is rather annoying to get rid of the error
partyboarder
Re: precision problem?
Posted: 25 Mar 2015, 16:46
by lerou
You can't guarantee something like that in float. Depending on your usecase, you should round your results or limit the decimals.
cheers,
rou
Re: precision problem?
Posted: 26 Mar 2015, 09:40
by Dierk Ohlerich
You can use a ToText node to convert one or more floating point values into a more human-friendly form.
This node is controlled by a "Format" string that is a bit cryptic. In your case, the format string "{0:0.0000}" will convert 0.00099999 to 0.0010. The zeros after the ':' are placeholders for digits. the zero before the colon specifies that the Value0 should be used.
A detailed description of how this works can be found at
https://msdn.microsoft.com/en-us/librar ... 10%29.aspx and
https://msdn.microsoft.com/en-us/librar ... 10%29.aspx.
Re: precision problem?
Posted: 26 Mar 2015, 10:40
by partyboarder
Hi
Thank you dierk.
I know .... the problem is that it will also add trailing zeros when using an input value of 0.1 .... result 0.0100 ... which is not wanted.
okay ... i will use that and add a script that will remove trailing zeroes ....
partyboarder
Re: precision problem?
Posted: 26 Mar 2015, 12:22
by lerou
Hey,
you should have a look at the C# string formatting documentation - this is not Ventuz specific. You don't need a script! Try:
{0:0.####}
cheers,
rou
Re: precision problem?
Posted: 26 Mar 2015, 12:33
by partyboarder
that worked .... thx
pb