Actually return with an error condition if we're being asked to deal with too many reference frames. Also check max num ref frames against our internal ref buffer sizes. Part of fix for roundup issue 281