Don't return 0 if buffer setup failed. That signals the RTSP demuxer that the packet was filled in, leading to virtually random behaviour in the decoder later on. Instead, return a negative value.