ofxCv contourFinder.draw()

Usually, the value of a scale variable multiply the size of the drawn objects. For example:
scale = 2 for double size.
scale = 0.5 for half size.
That's why I wrote:

ofScale( scale, scale );

Doing that, my modified mousePressed method works for me:

void testApp::mousePressed(int x, int y, int button) {
    targetColor = cam.getPixelsRef().getColor(x / scale, y / scale);

If you want to stick to your scale interpretation, I think you have to invert the operations in both the draw and the mousePressed methods:

// draw
ofScale( 1 / scale, 1 / scale );

// mousePressed
targetColor = cam.getPixelsRef().getColor(x * scale, y * scale);

Avoid using grabScreen(), which is a slow method (I read that many times, and I believe that even for just one pixel, it's better not to use it).