annotate DOCS/tech/mpcf.txt @ 17615:363d23eb55d3

fix index, first keyframe cannot be written with pts=0

========================================
NUT Open Container Format DRAFT 20060207
========================================



Intro:
======

Features / goals:
    (supported by the format, not necessarily by a specific implementation)

Simple
    use the same encoding for nearly all fields
    simple decoding, so slow CPUs (and embedded systems) can handle it

Extendible
    no limit for the possible values of all fields (using universal vlc)
    allow adding of new headers in the future
    allow adding more fields at the end of headers

Compact
    ~0.2% overhead, for normal bitrates
    index is <100kb per hour
    a usual header for a file is about 100 bytes (audio + video headers together)
    a packet header is about ~1-5 bytes

Error resistant
    seeking / playback without an index
    headers & index can be repeated
    damaged files can be played back with minimal data loss and fast
    resync times



Definitions:
============

MUST    the specific part must be done to conform to this standard
SHOULD  it is recommended to be done that way, but not strictly required



Syntax:
=======

Since NUT heavily uses variable length fields, the simplest way to describe it
is using a pseudocode approach.



Conventions:
============

The data types have a name, used in the bitstream syntax description, a short
text description and a pseudocode (functional) definition, optional notes may
follow:

name    (text description)
    functional definition
    [Optional notes]

The bitstream syntax elements have a tagname and a functional definition, they
are presented in a bottom up approach, again optional notes may follow and
are reproduced in the tag description:

name:    (optional note)
    functional definition
    [Optional notes]

The in-depth tag description follows the bitstream syntax.
The functional definition has a C-like syntax.



Type definitions:
=================

f(n)    (n fixed bits in big-endian order)
u(n)    (unsigned number encoded in n bits in MSB-first order)

v   (variable length value, unsigned)
    value=0
    do{
        more_data                       u(1)
        data                            u(7)
        value= 128*value + data
    }while(more_data)

s   (variable length value, signed)
    temp                                v
    temp++
    if(temp&1) value= -(temp>>1)
    else       value=  (temp>>1)

b   (binary data or string, to be use in vb, see below)
    for(i=0; i<length; i++){
        data[i]                         u(8)
    }
    [Note: strings MUST be encoded in UTF-8]

vb  (variable length binary data or string)
    length                              v
    value                               b



Bitstream syntax:
=================

Common elements:
----------------

packet header:
    forward ptr                         v

reserved_bytes:
    for(i=0; i<forward_ptr - length_of_non_reserved; i++)
        reserved                        u(8)
    [a demuxer MUST ignore any reserved bytes
    a muxer MUST NOT write any reserved bytes, as this would make it
    impossible to add new fields at the end of packets in the future
    in a compatible way]

        Headers:

main header:
    main_startcode                      f(64)
    packet header
    version                             v
    stream_count                        v
    max_distance                        v
    tmp_pts=0
    tmp_mul=1
    tmp_stream=0
    for(i=0; i<256; ){
        tmp_flag                        v
        tmp_fields                      v
        if(tmp_fields>0) tmp_sflag      v
        else tmp_sflag=0
        if(tmp_fields>1) tmp_pts        s
        if(tmp_fields>2) tmp_mul        v
        if(tmp_fields>3) tmp_stream     v
        if(tmp_fields>4) tmp_size       v
        else tmp_size=0
        if(tmp_fields>5) tmp_res        v
        else tmp_res=0
        if(tmp_fields>6) count          v
        else count= tmp_mul - tmp_size
        for(j=7; j<tmp_fields; j++){
            tmp_reserved[i]             v
        }
        for(j=0; j<count && i<256; j++, i++){
            if (i == 'N') {
                flags[i]= 4;
                j--;
                continue;
            }
            flags[i]= tmp_flag;
            stream_flags[i]= tmp_sflag;
            stream_id_plus1[i]= tmp_stream;
            data_size_mul[i]= tmp_mul;
            data_size_lsb[i]= tmp_size + j;
            pts_delta[i]= tmp_pts;
            reserved_count[i]= tmp_res;
        }
    }
    reserved_bytes
    checksum                            u(32)

stream_header:
    stream_startcode                    f(64)
    packet_header
    stream_id                           v
    stream_class                        v
    fourcc                              vb
    time_base_nom                       v
    time_base_denom                     v
    msb_pts_shift                       v
    max_pts_distance                    v
    decode_delay                        v
    fixed_fps                           u(1)
    reserved                            u(7)
    codec_specific_data                 vb

video_stream_header:
    stream_header
    width                               v
    height                              v
    sample_width                        v
    sample_height                       v
    colorspace_type                     v
    reserved_bytes
    checksum                            u(32)

audio_stream_header:
    stream_header
    samplerate_nom                      v
    samplerate_denom                    v
    channel_count                       v
    reserved_bytes
    checksum                            u(32)

other_stream_header:
    stream_header
    reserved_bytes
    checksum                            u(32)

        Basic Packets:

frame:
    frame_code                          f(8)
    if(stream_id_plus1[frame_code]==0){
        stream_id                       v
    }
    if(pts_delta[frame_code]==0){
        coded_pts                       v
    }
    if(flags[frame_code]&1){
        data_size_msb                   v
    }
    if(flags[frame_code]&2){
        coded_stream_flags              v
    }
    for(i=0; i<reserved_count[frame_code]; i++)
        reserved                        v
    if (after_syncpoint){
        syncpoint_checksum              u(32)
    }
    data

index:
    index_startcode                     f(64)
    packet header
    max_pts                             v
    syncpoints                          v
    for(i=0; i<syncpoints; i++){
        syncpoint_pos_div8              v
    }
    for(i=0; i<stream_count; i++){
        last_pts= -1
        for(j=0; j<syncpoints; ){
            x                           v
            type= x & 1
            x>>=1
            n=j
            if(type){
                flag= x & 1
                x>>=1
                while(x--)
                    has_keyframe[n++][i]=flag
                has_keyframe[n++][i]=!flag;
            }else{
                while(x != 1){
                    has_keyframe[n++][i]=x&1;
                    x>>=1;
                }
            }
            for(; j<n && j<syncpoints; j++){
                if (!has_keyframe[j][i]) continue
                A                           v
                if(!A){
                    A                       v
                    B                       v
                    eor_pts[j][i] = last_pts + A + B
                }else
                    B=0
                keyframe_pts[j][i] = last_pts + A
                last_pts += A + B
            }
        }
    }
    reserved_bytes
    index_ptr                           u(64)
    checksum                            u(32)

info_frame: (optional)
    for(;;){
        id                              v
        if(id==0) break
        name= info_table[id][0]
        type= info_table[id][1]
        if(type==NULL)
            type                        vb
        if(name==NULL)
            name                        vb
        if(type=="v")
            value                       v
        else if(type=="s")
            value                       s
        else
            value                       vb
    }
    reserved_bytes
    checksum                            u(32)

info_packet: (optional)
    info_startcode                      f(64)
    packet header
    info_frame

syncpoint:
    syncpoint_startcode                 f(64)
    coded_pts                           v
    stream = coded_pts % stream_count
    global_key_pts = coded_pts/stream_count
    back_ptr_div8                       v

            Complete definition:

file:
    file_id_string
    while(!eof){
        main_header
        for(i=0; i<stream_count; i++){
            if(next_packet==video_stream_header)
                video_stream_header
            else if(next_packet==audio_stream_header)
                audio_stream_header
            else
                other_stream_header
        }
        while(next_code == info_startcode){
            info_packet
        }
        if(next_code == index_startcode){
            index
        }
        if (!eof) while(next_code != main_startcode){
            if(next_code == syncpoint_startcode)
                syncpoint
            frame
        }
    }


Tag description:
----------------

file_id_string
    "nut/multimedia container\0"

*_startcode
        all startcodes start with 'N'

main_startcode
    0x7A561F5F04ADULL + (((uint64_t)('N'<<8) + 'M')<<48)

stream_starcode
    0x11405BF2F9DBULL + (((uint64_t)('N'<<8) + 'S')<<48)

syncpoint_startcode
    0xE4ADEECA4569ULL + (((uint64_t)('N'<<8) + 'K')<<48)

index_startcode
    0xDD672F23E64EULL + (((uint64_t)('N'<<8) + 'X')<<48)

info_startcode
    0xAB68B596BA78ULL + (((uint64_t)('N'<<8) + 'I')<<48)

version
    NUT version. The current value is 2.

forward_ptr
    size of the packet data (exactly the distance from the first byte
    after the forward_ptr to the first byte of the next packet)

max_distance
    max distance of syncpoints, the distance may only be larger if
    there is no more than a single frame between the two syncpoints. This can
    be used by the demuxer to detect damaged frame headers if the damage
    results in too long of a chain

    syncpoints SHOULD be placed immediately before a keyframe if the
    previous frame of the same stream was a non-keyframe, unless such
    non-keyframe - keyframe transitions are very frequent

    SHOULD be set to <=32768 or at least <=65536 unless there is a very
    good reason to set it higher, otherwise reasonable error recovery will
    be impossible

max_pts_distance
    max absoloute difference of pts of new frame from last_pts in the
    timebase of the stream, without a syncpoint immediately before the
    frame. Note that last_pts is not necessarily the pts of the last frame
    on the same stream, as it is altered by syncpoint timestamps.

stream_id
    Stream identifier
    stream_id MUST be < stream_count

stream_class
    0    video
    1    audio
    2    subtiles
    3    metadata
    4    userdata
    in metadata streams each frame contains exactly one info frame
    Note: the remaining values are reserved and MUST NOT be used
          a demuxer MUST ignore streams with reserved classes

fourcc
    identification for the codec
    example: "H264"
    MUST contain 2 or 4 bytes, note, this might be increased in the future
    if needed

time_base_nom / time_base_denom = time_base
    the length of a timer tick in seconds, this MUST be equal to the 1/fps
    if fixed_fps is 1
    time_base_nom and time_base_denom MUST NOT be 0
    time_base_nom and time_base_denom MUST be relatively prime
    time_base_denom MUST be < 2^31
    examples:
        fps       time_base_nom    time_base_denom
        30        1                30
        29.97     1001             30000
        23.976    1001             24000

convert_ts
    To switch from 2 different timebases, the following calculation is
    defined:

    ln        = from_time_base_nom*to_time_base_denom
    sn        = from_timestamp
    d1        = from_time_base_denom
    d2        = to_time_base_nom
    timestamp = (ln/d1*sn + ln%d1*sn/d1)/d2
    Note: this calculation MUST be done with unsigned 64 bit integers, and
    is equivalent to (ln*sn)/(d1*d2) but this would require a 96bit integer

compare_ts
    Compares timestamps from 2 different timebases,
    if a is before b then compare_ts(a, b) = -1
    if a is after  b then compare_ts(a, b) =  1
    else                  compare_ts(a, b) =  0

    Care must be taken that this is done exactly with no rounding errors,
    simply casting to float or double and doing the obvious
    a*timebase > b*timebase is not compliant or correct, neither is the
    same with integers, and
    a*a_timebase.num*b_timebase.den > b*b_timebase.num*a_timebase.den
    will overflow. One possible implementation which shouldn't overflow
    within the range of legal timestamps and timebases is:

    if (convert_ts(a, a_timebase, b_timebase) < b) return -1;
    if (convert_ts(b, b_timebase, a_timebase) < a) return  1;
    return 0;

msb_pts_shift
    amount of bits in lsb_pts
    MUST be <16

decode_delay
    maximum time between input and output for a codec, used to generate
    dts from pts
    is set to 0 for streams without B-frames, and set to 1 for streams with
    B-frames, may be larger for future codecs

fixed_fps
    1 indicates that the fps is fixed

codec_specific_data
    private global data for a codec (could be huffman tables or ...)

frame_code
    the meaning of this byte is stored in the main header
    the value 78 ('N') is forbidden to ensure that the byte is always
    different from the first byte of any startcode

flags[frame_code]
    Bit  Name             Description
      1  data_size_msb    if set, data_size_msb is at frame header,
                          otherwise data_size_msb is 0
      2  more_flags       if set, stream control flags are at frame header.
      4  invalid          if set, frame_code is invalid.

stream_flags
    stream_flags is "stream_flags[frame_code] ^ coded_stream_flags"

    Bit  Name               Description
      1  is_key             if set, frame is keyframe
      2  end_of_relevance   if set, stream has no relevance on
                            presentation. (EOR)

    EOR frames MUST be zero-length and must be set keyframe.
    All streams SHOULD end with EOR, where the pts of the EOR indicates the
    end presentation time of the final frame.
    An EOR set stream is unset by the first content frames.
    EOR can only be unset in streams with zero decode_delay .

stream_id_plus1[frame_code]
    must be <250
    if it is 0, then the stream_id is coded in the frame

data_size_mul[frame_code]
    must be <16384

data_size_lsb[frame_code]
    must be <16384

pts_delta[frame_code]
    must be <16384 and >-16384

data_size
    data_size= data_size_lsb + data_size_msb*data_size_mul;

coded_pts
    if coded_pts < (1<<msb_pts_shift) then it is an lsb
    pts, otherwise it is a full pts + (1<<msb_pts_shift)
    lsb pts is converted to a full pts by:
    mask  = (1<<msb_pts_shift)-1;
    delta = last_pts - mask/2
    pts   = ((pts_lsb-delta)&mask) + delta

lsb_pts
    least significant bits of the pts in time_base precision
        Example: IBBP display order
        keyframe pts=0                       -> pts=0
        frame                    lsb_pts=3   -> pts=3
        frame                    lsb_pts=1   -> pts=1
        frame                    lsb_pts=2   -> pts=2
        ...
        keyframe msb_pts=257                 -> pts=257
        frame                    lsb_pts=255 -> pts=255
        frame                    lsb_pts=0   -> pts=256
        frame                    lsb_pts=4   -> pts=260
        frame                    lsb_pts=2   -> pts=258
        frame                    lsb_pts=3   -> pts=259
    all pts's of keyframes of a single stream MUST be monotone

dts
    dts is calculated by using a decode_delay+1 sized buffer for each
    stream, into which the current pts is inserted and the element with
    the smallest value is removed, this is then the current dts
    this buffer is initalized with decode_delay -1 elements

    Pts of all frames in all streams MUST be bigger or equal to dts of all
    previous frames in all streams, compared in common timebase. (EOR
    frames are NOT exempt from this rule)

width/height
    MUST be set to the coded width/height

sample_width/sample_height (aspect ratio)
    sample_width is the horizontal distance between samples
    sample_width and sample_height MUST be relatively prime if not zero
    MUST be 0 if unknown

colorspace_type
     0    unknown
     1    ITU Rec 624 / ITU Rec 601 Y range: 16..235 Cb/Cr range: 16..240
     2    ITU Rec 709               Y range: 16..235 Cb/Cr range: 16..240
    17    ITU Rec 624 / ITU Rec 601 Y range:  0..255 Cb/Cr range:  0..255
    18    ITU Rec 709               Y range:  0..255 Cb/Cr range:  0..255

samplerate_nom / samplerate_denom = samplerate
    the number of samples per second

checksum
    crc32 checksum
    checksum is calculated for the area pointed to by forward_ptr not
    including the checksum itself (from first byte after the
    forward_ptr until last byte before the checksum).

syncpoint_checksum
    crc32 checksum
    checksum covers from first byte after syncpoint startcode until last 
    byte before the syncpoint_checksum.

back_ptr_div8
    back_ptr = back_ptr_div8 * 8 + 7
    back_ptr must point to a position within 8 bytes of a syncpoint
    startcode. This syncpoint MUST be the closest syncpoint such that at
    least one keyframe with a pts lower or equal to the original syncpoint's
    global_key_pts for all streams lies between it and the current syncpoint.

    A stream where EOR is set is to be ignored for back_ptr.

global_key_pts
    After a syncpoint, last_pts of each stream is to be set to:
    last_pts[i] = convert_ts(global_key_pts, timebase[stream], timebase[i])

    global_key_pts MUST be bigger or equal to dts of all past frames across
    all streams, and smaller or equal to pts of all future frames.

max_pts
    s = max_pts % stream_count
    pts = max_pts / stream_count
    The highest pts in the entire file in the timebase of stream 's' .

syncpoint_pos_div8
    offset from begginning of file to up to 7 bytes before the syncpoint
    referred to in this index entry. Relative to position of last
    syncpoint.

has_keyframe
    indicates whether this stream has a keyframe between this syncpoint and
    the last syncpoint.

keyframe_pts
    The pts of the first keyframe for this stream in the region between the
    2 syncpoints, in the stream's timebase. (EOR frames are also keyframes)

eor_pts
    Coded only if EOR is set at the position of the syncpoint. The pts of
    that EOR. EOR is unset by the first keyframe after it.

index_ptr
    Length in bytes of the entire index, from the first byte of the
    startcode until the last byte of the checksum.
    Note: A demuxer can use this to find the index when it is written at
    EOF, as index_ptr will always be 12 bytes before the end of file if
    there is an index at all.

id
    the ID of the type/name pair, so it is more compact
    0 means end

type
    for example: "UTF8" -> string or "JPEG" -> JPEG image
    Note: nonstandard fields should be prefixed by "X-"
    Note: MUST be less than 6 byte long (might be increased to 64 later)

info packet types
    the name of the info entry, valid names are
    "StreamId"
        the stream(s) to which the info packet applies
    "Author"
    "Description"
    "Copyright"
    "Encoder"
        the name & version of the software used for encoding
    "Title"
    "Cover"
        image of the (CD, DVD, VHS, ..) cover (preferably PNG or JPEG)
    "Source"
        "DVD", "VCD", "CD", "MD", "FM radio", "VHS", "TV", "LD"
        Optional: appended PAL, NTSC, SECAM, ... in parentheses
    "CaptureDevice"
        "BT878", "BT848", "webcam", ... (more exact names are fine too)
    "CreationTime"
        "2003-01-20 20:13:15Z", ...
        (ISO 8601 format, see http://www.cl.cam.ac.uk/~mgk25/iso-time.html)
        Note: do not forget the timezone
    "Keywords"
    "TotalTime"
        total length of the stream in msecs
    "Language"
        ISO 639 and ISO 3166 for language/country code
        something like "eng" (US english), can be 0 if unknown
        and "multi" if several languages
        see http://www.loc.gov/standards/iso639-2/englangn.html
        and http://www.din.de/gremien/nas/nabd/iso3166ma/codlstp1/en_listp1.html
        the language code
    "Disposition"
        "original", "dub" (translated), "comment", "lyrics", "karaoke"
        Note: if someone needs some others, please tell us about them, so we
              can add them to the official standard (if they are sane)
        Note: nonstandard fields should be prefixed by "X-"
        Note: MUST be less than 64 bytes long

value
    value of this name/type pair

stuffing
    0x80 can be placed in front of any type v entry for stuffing purposes

info_table[][2]={
    {NULL            ,  NULL }, // end
    {NULL            ,  NULL },
    {NULL            , "UTF8"},
    {NULL            , "v"},
    {NULL            , "s"},
    {"StreamId"      , "v"},
    {"Author"        , "UTF8"},
    {"Title"         , "UTF8"},
    {"Language"      , "UTF8"},
    {"Description"   , "UTF8"},
    {"Copyright"     , "UTF8"},
    {"Encoder"       , "UTF8"},
    {"Keyword"       , "UTF8"},
    {"Cover"         , "JPEG"},
    {"Cover"         , "PNG"},
    {"Disposition"   , "UTF8"},
};


Structure:
----------

the headers MUST be in exactly the following order (to simplify demuxer design)
main header
stream_header (id=0)
stream_header (id=1)
...
stream_header (id=n)

headers may be repeated, but if they are, then they MUST all be repeated
together and repeated headers MUST be identical

Each set of repeated headers not at the beginning or end of the file SHOULD
be stored at the earliest possible position after 2^x where x is
an integer and the file end, so the headers may be repeated at 4102 if that is
the closest position after 2^12=4096 at which the headers can be placed

Note: this allows an implementation reading the file to locate backup
headers in O(log filesize) time as opposed to O(filesize)

headers MUST be placed at least at the start of the file and immediately before
the index or at the file end if there is no index
headers MUST be repeated at least twice (so they exist three times in a file)

there MUST be a sync point immediately before the first frame after any headers


Index:
------

Note: with realtime streaming, there is no end, so no index there either
Index MAY only be repeated after main headers.
If an index is written anywhere in the file, it MUST be written at end of
file as well.


Info frames:
------------

Info frames can be used to describe the file or some part of it (chapters)


Unknown packets:
----------------

MUST be ignored by the demuxer


demuxer (non-normative):
------------------------

in the absence of a valid header at the beginning, players SHOULD search for
backup headers starting at offset 2^x; for each x players SHOULD end their
search at a particular offset when any startcode is found (including syncpoint)



Semantic requirements:
======================

If more than one stream of a given stream class is present, each one SHOULD
have info tags specifying disposition, and if applicable, language.
It often highly improves usability and is therefore strongly encouraged.

A demuxer MUST NOT demux a stream which contains more than one stream, or which
is wrapped in a structure to facilitate more than one stream or otherwise
duplicate the role of a container. any such file is to be considered invalid.



Sample code (Public Domain, & untested):
========================================

typedef BufferContext{
    uint8_t *buf;
    uint8_t *buf_ptr;
}BufferContext;

static inline uint64_t get_bytes(BufferContext *bc, int count){
    uint64_t val=0;

    assert(count>0 && count<9);

    for(i=0; i<count; i++){
        val <<=8;
        val += *(bc->buf_ptr++);
    }

    return val;
}

static inline void put_bytes(BufferContext *bc, int count, uint64_t val){
    uint64_t val=0;

    assert(count>0 && count<9);

    for(i=count-1; i>=0; i--){
        *(bc->buf_ptr++)= val >> (8*i);
    }

    return val;
}

static inline uint64_t get_v(BufferContext *bc){
    uint64_t val= 0;

    for(; space_left(bc) > 0; ){
        int tmp= *(bc->buf_ptr++);
        if(tmp&0x80)
            val= (val<<7) + tmp - 0x80;
        else
            return (val<<7) + tmp;
    }

    return -1;
}

static inline int put_v(BufferContext *bc, uint64_t val){
    int i;

    if(space_left(bc) < 9) return -1;

    val &= 0x7FFFFFFFFFFFFFFFULL; // FIXME can only encode upto 63 bits currently
    for(i=7; ; i+=7){
        if(val>>i == 0) break;
    }

    for(i-=7; i>0; i-=7){
        *(bc->buf_ptr++)= 0x80 | (val>>i);
    }
    *(bc->buf_ptr++)= val&0x7F;

    return 0;
}

static int64_t get_dts(int64_t pts, int64_t *pts_cache, int delay, int reset){
    if(reset) memset(pts_cache, -1, delay*sizeof(int64_t));

    while(delay--){
        int64_t t= pts_cache[delay];
        if(t < pts){
            pts_cache[delay]= pts;
            pts= t;
        }
    }

    return pts;
}



Authors:
========

Folks from the MPlayer developers mailing list (http://www.mplayerhq.hu/).
Authors in alphabetical order: (FIXME! Tell us if we left you out)
    Beregszaszi, Alex        ([e-mail])
    Bunkus, Moritz           ([e-mail])
    Diedrich, Tobias         ([e-mail])
    Felker, Rich             ([e-mail])
    Franz, Fabian            ([e-mail])
    Gereoffy, Arpad          ([e-mail])
    Hess, Andreas            ([e-mail])
    Niedermayer, Michael     ([e-mail])
    Shimon, Oded             ([e-mail])