Hi
i found this precision problem when using a float expression to divide 0.01 by 10
i know that mathematically 0.0009999999 is considered 0.001 but when you use the result in a textnode it is rather annoying to get rid of the error
partyboarder
Hello Ventuzians!
THE FORUMS ARE CLOSED!
Please join our discord server HERE!! << click me
We are shutting our Ventuz Forum, but don't worry, it will all be archived for you to search in if you have a query. From now on, please add all your comments, questions or observations into our Discord Server
Thanks for the great time - see you on discord!!
Dee, Karol, Daniel and the whoooole Product and Support team!
THE FORUMS ARE CLOSED!
Please join our discord server HERE!! << click me
We are shutting our Ventuz Forum, but don't worry, it will all be archived for you to search in if you have a query. From now on, please add all your comments, questions or observations into our Discord Server
Thanks for the great time - see you on discord!!
Dee, Karol, Daniel and the whoooole Product and Support team!
precision problem?
Moderator: Support
Re: precision problem?
You can't guarantee something like that in float. Depending on your usecase, you should round your results or limit the decimals.
cheers,
rou
cheers,
rou
- Dierk Ohlerich
- Posts: 74
- Joined: 10 Jul 2012, 09:01
- Location: Hamburg, Germany
Re: precision problem?
You can use a ToText node to convert one or more floating point values into a more human-friendly form.
This node is controlled by a "Format" string that is a bit cryptic. In your case, the format string "{0:0.0000}" will convert 0.00099999 to 0.0010. The zeros after the ':' are placeholders for digits. the zero before the colon specifies that the Value0 should be used.
A detailed description of how this works can be found at https://msdn.microsoft.com/en-us/librar ... 10%29.aspx and https://msdn.microsoft.com/en-us/librar ... 10%29.aspx.
This node is controlled by a "Format" string that is a bit cryptic. In your case, the format string "{0:0.0000}" will convert 0.00099999 to 0.0010. The zeros after the ':' are placeholders for digits. the zero before the colon specifies that the Value0 should be used.
A detailed description of how this works can be found at https://msdn.microsoft.com/en-us/librar ... 10%29.aspx and https://msdn.microsoft.com/en-us/librar ... 10%29.aspx.
-- Dierk Ohlerich
-
- Posts: 13
- Joined: 20 Jun 2013, 20:58
Re: precision problem?
Hi
Thank you dierk.
I know .... the problem is that it will also add trailing zeros when using an input value of 0.1 .... result 0.0100 ... which is not wanted.
okay ... i will use that and add a script that will remove trailing zeroes ....
partyboarder
Thank you dierk.
I know .... the problem is that it will also add trailing zeros when using an input value of 0.1 .... result 0.0100 ... which is not wanted.
okay ... i will use that and add a script that will remove trailing zeroes ....
partyboarder
Re: precision problem?
Hey,
you should have a look at the C# string formatting documentation - this is not Ventuz specific. You don't need a script! Try:
{0:0.####}
cheers,
rou
you should have a look at the C# string formatting documentation - this is not Ventuz specific. You don't need a script! Try:
{0:0.####}
cheers,
rou
-
- Posts: 13
- Joined: 20 Jun 2013, 20:58
Re: precision problem?
that worked .... thx
pb
pb