ofImage resize is so inefficient,any ideas?

If you only want it for rendering, just use OpenGL to scale it. If you need to store it back to RAM you can also render it smaller and then use ofTexture::readToPixels but that might not be that much faster.