Hi @Jez,
That latency looks a little strange. Can you post the code to your app?
Latency with uploading video data to a texture often isn't due to OF, but problems with CPU -> GPU syncing.
Whenever a buffer or texture is updated (copying from system RAM to GPU memory, for example), the data is uploaded and this stalls the GPU until it has finished uploading so that the texture is safe to use.
If you only have 1 texture that the video data is uploaded to each frame, then what happens is that as soon as you use that texture in a draw, everything stalls until the upload is finished so that the updated texture can be used.
Stalling the GPU is something that you want to avoid whenever possible. A lot of engine optimizations revolve around reducing GPU stalls and CPU->GPU synchronization points. One way around this is to use 2 or 3 buffers (you may have heard the term multi-buffered before).
If you have 2 textures that you are uploading the video to, you can do something like this:
Frame 1:
Upload video to Texture 1
Draw Texture 2
Frame 2:
Upload video to Texture 2
Draw Texture 1
Frame 3:
Upload video to Texture 1
Draw Texture 2
And so on.... You can profile this and see if cycling through 2 textures is enough. I've found that I needed 3 in one project to avoid stalls. You'll always be drawing a video data frame that's 1 or 2 frames old, but that's ok. In fact, all modern game engines do this for rendering - UE4 for example is always 1 or 2 frames behind in rendering compared to the game world state.
Arturo has a good example of Pixel Buffer Objects (PBO) used to download a texture (you'll see he has 2 buffers front and back buffer). You can do something similar for updating a texture on the GPU vs. downloading. Here's his example:
The general idea with rendering engines is to keep the CPU busy preparing the current frame while the GPU is busy rendering the previous frame. If all is designed well, they line up 
Cheers,
James