Hello Ventuzians!
THE FORUMS ARE CLOSED!

Please join our discord server HERE!! << click me :D

We are shutting our Ventuz Forum, but don't worry, it will all be archived for you to search in if you have a query. From now on, please add all your comments, questions or observations into our Discord Server

Thanks for the great time - see you on discord!!
Dee, Karol, Daniel and the whoooole Product and Support team!

precision problem?

Please report bugs in this group. We will maybe populate a list of known bugs with a possibly modified description at a later time.

Moderator: Support

Post Reply
partyboarder
Posts: 13
Joined: 20 Jun 2013, 20:58

precision problem?

Post by partyboarder » 25 Mar 2015, 16:14

Hi

i found this precision problem when using a float expression to divide 0.01 by 10
rounderror.PNG
i know that mathematically 0.0009999999 is considered 0.001 but when you use the result in a textnode it is rather annoying to get rid of the error

partyboarder

User avatar
lerou
Posts: 345
Joined: 06 Sep 2013, 07:14
Location: Hamburg, Germany

Re: precision problem?

Post by lerou » 25 Mar 2015, 16:46

You can't guarantee something like that in float. Depending on your usecase, you should round your results or limit the decimals.

cheers,
rou

User avatar
Dierk Ohlerich
Posts: 74
Joined: 10 Jul 2012, 09:01
Location: Hamburg, Germany

Re: precision problem?

Post by Dierk Ohlerich » 26 Mar 2015, 09:40

You can use a ToText node to convert one or more floating point values into a more human-friendly form.

This node is controlled by a "Format" string that is a bit cryptic. In your case, the format string "{0:0.0000}" will convert 0.00099999 to 0.0010. The zeros after the ':' are placeholders for digits. the zero before the colon specifies that the Value0 should be used.

A detailed description of how this works can be found at https://msdn.microsoft.com/en-us/librar ... 10%29.aspx and https://msdn.microsoft.com/en-us/librar ... 10%29.aspx.
-- Dierk Ohlerich

partyboarder
Posts: 13
Joined: 20 Jun 2013, 20:58

Re: precision problem?

Post by partyboarder » 26 Mar 2015, 10:40

Hi

Thank you dierk.

I know .... the problem is that it will also add trailing zeros when using an input value of 0.1 .... result 0.0100 ... which is not wanted.

okay ... i will use that and add a script that will remove trailing zeroes ....

partyboarder

User avatar
lerou
Posts: 345
Joined: 06 Sep 2013, 07:14
Location: Hamburg, Germany

Re: precision problem?

Post by lerou » 26 Mar 2015, 12:22

Hey,

you should have a look at the C# string formatting documentation - this is not Ventuz specific. You don't need a script! Try:

{0:0.####}

cheers,
rou

partyboarder
Posts: 13
Joined: 20 Jun 2013, 20:58

Re: precision problem?

Post by partyboarder » 26 Mar 2015, 12:33

that worked .... thx

pb

Post Reply