With h264 it runs ok on my laptop, but didn't really think of the impact it would have once running on the RasPI, good point.
I was thinking of using something like keyframe every frame, but I'll give some different encodings a run to see which performs better for frame jumping.