Without some of the tips Zach said, it's pretty hard to know what's going on.
Regardless of that, the first thing I thought was the serial data. Drivers, connections, bus sharing, etc. can cause serious bottlenecks. So, this would be the first I would start to debug.
If serial is the culprit, maybe you can try ofxSerial, since it uses it's own thread.