I've had similar experience, but not a full second unless I was running something like CoreImage face detection or some heave OpenCV stuff on the video.
I was able to swap out a MacMini with an i5 processor to a model with an i7 and I saw a big difference. I've always been able to get at least 30 fps with my 2 year old Macbook Pro - with both the C920 and C930e.
USB2 could be the culprit. Are you able to try USB3 instead?