What IR camera are you using? I tried Kinect1 IR image with ofxFaceTracker, but the resolution is bad (not the pixel resolution but wide-angle lens makes everything appear small, so you have to be close enough to the camera).
Btw, ofxFaceTracker seems not using color image anyways: