Cluster Sync Delay

All other topics about Ventuz 5 here.

Cluster Sync Delay

Postby ejgazz » 15 Mar 2017, 15:16

Hello, I'm working on a permanent installation that uses 5 Ventuz V5 runtime machines in a cluster. I've noticed 4 frames of delay between machines in the cluster. I have set up a scientific test with a simple scene using a counter node to generate time in milliseconds. So far I've checked:

-the machines are all in the same cluster
-we have a simple LAN with minimal network traffic
-the delay is the same between machines of the same resolution and machines of different resolutions
-I've tried a few different test scenes, and the delay seems consistent
-Currently we have no sync cards with the GPU, but we plan to add them soon

System Info:
Ventuz 5.2.2
Quadro P5000
Windows 10
Eric Gazzillo
Display Designer
Quince Imaging
ejgazz
 
Posts: 7
Joined: 15 Mar 2017, 14:48
Location: Washington, D.C.

Re: Cluster Sync Delay

Postby Karol » 16 Mar 2017, 13:20

Hi Eric!

4 Frames delay are pretty strange!
Actually there shouldn't be any delay besides the rendering delay caused by the not sync'ed graphics boards.
Did you go through the checklist: http://www.ventuz.com/support/help/late ... #CheckList
Did you start the build-in Default presentation on your Cluster?

I attached a test scene (actually for 3x3 display Cluster). Please adapt it to your setup and test it.

Best Regards
Karol
Attachments
ClusterSyncTest.vzs
(13.56 KiB) Downloaded 54 times
User avatar
Karol
 
Posts: 458
Joined: 10 Jan 2012, 12:07

Re: Cluster Sync Delay

Postby ejgazz » 16 Mar 2017, 14:40

This test scene is brilliant! If I understand correctly, this just updates the cluster clock display every few seconds to easily see that the clocks match across the cluster? My servers are in a truck, but I will test this and get back to you.

We did go through the checklist, the only one I'm not sure on is the firewall, but I will check that as well.
Eric Gazzillo
Display Designer
Quince Imaging
ejgazz
 
Posts: 7
Joined: 15 Mar 2017, 14:48
Location: Washington, D.C.

Re: Cluster Sync Delay

Postby Karol » 16 Mar 2017, 16:19

Exactly! :)
And please check the Cluster ID states as described here: http://www.ventuz.com/support/help/late ... dClusterID

Cheers
Karol
User avatar
Karol
 
Posts: 458
Joined: 10 Jan 2012, 12:07

Re: Cluster Sync Delay

Postby ejgazz » 20 Mar 2017, 16:14

Okay so the test scene ended up working well. The cluster clock matched up fine, but the black-white flash was out of sync. Other questions that come to mind:

-Are there any special considerations for a cluster when mixing and matching resolutions? (3840x2160@60 vs 1920x1200@60)
-Because the Nvidia P5000 is so new, do you think there's a chance this could be related to GPU? This uses a completely new sync board.
-Everything in our signal path claims to be "zero" latency, but I am confirming those specs direct with the manufacturer
-When making a test scene, does a scene event rely on the cluster clock? Would an absolute mover be more accurate for generating milliseconds
Eric Gazzillo
Display Designer
Quince Imaging
ejgazz
 
Posts: 7
Joined: 15 Mar 2017, 14:48
Location: Washington, D.C.

Re: Cluster Sync Delay

Postby lerou » 20 Mar 2017, 16:25

the scene event is fired after the scene is loaded. It is not sync'd over your cluster. You need to start everything once all machines are up. Either use OSC to start your stuff manually or build it depending on an absolute mover or build a sync mechanism where the scene is restartet via OSC multicast after each machine starts -> that way after each of the cluster's machines is ready, it will restart the whole cluster. Optimally you implement a 10s timer before sending that OSC.

cheers,
rou
User avatar
lerou
 
Posts: 237
Joined: 06 Sep 2013, 07:14
Location: Hamburg, Germany

Re: Cluster Sync Delay

Postby ejgazz » 22 Mar 2017, 05:32

As an FYI to anyone reading this later: It turns out the AJA HA5-4k adds approximately 4 frames of delay to an output. I'm still working with AJA tech support, but they admitted they did not test this product for latency when it was developed.
Eric Gazzillo
Display Designer
Quince Imaging
ejgazz
 
Posts: 7
Joined: 15 Mar 2017, 14:48
Location: Washington, D.C.


Return to General Discussion