Make sure that when the parser is feeded with frame==packet that the packets are passed through and wont be marked as static which would require them to be copied by av_dup_packet().