Use frame_num and reference instead of poc for matching frames for direct mode MBs, this seems to work better with field/frame mixes. POC of both can be the same and can be different that makes its use tricky.