Are you utilizing the ofxMidiConnectionListener callback functions on iOS? They allow you to know when new midi ports are created or released in CoreAudio.
I bet your crash is coming because you open a port that was created by another app and when you close that app, you don't received the message that it's midi port has been closed, so that pointer is no longer valid. See the ofxMidi exampleIOS which shows how the ofxMidiConnectionListener class is used in ofApp.