Multimedia encoding

The three layers: content, codec and container

These terms are often mixed together, but they describe separate jobs.

The file extension usually identifies the container, not the codecs inside it. Two .mp4 files can contain different video or audio codecs, and a player may support one but not the other.

Why multimedia is compressed

Raw media is enormous. Uncompressed 1920×1080 video at 30 frames per second, with 24 bits per pixel, is about 1.49 gigabits per second before any overhead. One minute would consume roughly 11 GB. Stereo CD-quality PCM audio is about 1.41 megabits per second.

Compression removes two kinds of redundancy:

• Statistical redundancy: predictable patterns can be written more efficiently without losing information.
• Perceptual redundancy: some detail is less noticeable to human sight or hearing and can be discarded.

Lossless compression reconstructs the exact original samples. Lossy compression reconstructs a close approximation and achieves much smaller files. Re-encoding lossy media loses more information each time, so editing workflows normally keep a high-quality master and encode delivery copies from that.

How video codecs shrink moving pictures

A video codec takes advantage of similarity within one frame and between neighbouring frames. Although codecs differ, many use the same broad ideas.

1. Describe colour efficiently. Video is commonly converted from RGB to luma (brightness) and chroma (colour difference). Chroma may be stored at lower resolution because vision notices fine brightness detail more readily than fine colour detail.
2. Predict. Intra prediction estimates a block from nearby pixels in the same frame. Inter prediction finds a similar block in another frame and records a motion vector.
3. Encode the difference. The prediction error is transformed into frequency-like coefficients.
4. Quantise. Coefficients are rounded. Stronger rounding saves bits but removes detail; this is the main lossy step.
5. Entropy-code. Frequent symbols and patterns receive shorter representations without further loss.

I, P and B pictures

• I-frame: decoded without another picture. It provides a random-access point but costs many bits.
• P-frame: predicts from an earlier reference picture.
• B-frame: can predict using pictures on both sides in display order, improving compression but adding dependency and sometimes delay.

A sequence organised around reference pictures is a group of pictures (GOP). Short GOPs seek and recover quickly; long GOPs compress more efficiently. Production codecs often encode every frame independently because that makes editing responsive, even though the files are larger.

Common video codecs

H.264 is also called AVC or MPEG-4 Part 10. It is not the same as the older MPEG-4 Part 2 used by DivX and Xvid. H.265 is the same codec family as HEVC.

A codec name alone is not a complete compatibility promise. Profiles group coding features, and levels limit demands such as resolution, frame rate and bitrate. A device may decode H.264 Main Profile at 1080p but reject a high-bit-depth profile or a 4K level.

How digital audio and audio codecs work

A microphone signal is measured at regular intervals. The sample rate is how many measurements are taken per second; 48 kHz means 48,000 samples per second per channel. Bit depth sets the available numerical precision and dynamic range. Channels describe separate signals such as left, right, centre or surround.

PCM stores those sample values directly. Lossless codecs predict sample patterns and compress the prediction error so every sample can be restored exactly. Lossy perceptual codecs divide sound into frequency and time regions, model what listeners are less likely to hear, quantise that information and encode the result compactly.

At low bitrates, lossy codecs may produce watery high frequencies, smeared transients or unstable stereo. The encoder implementation and settings matter as well as the codec name.

Common audio codecs

A bitstream may carry compressed multi-channel audio directly to a television or receiver for decoding. Alternatively, the source device can decode it to multi-channel PCM first. Both routes can reproduce the same underlying programme when the formats and processing match.

Common container formats

Compatibility is the intersection of the container, every codec inside it, codec profile and level, media properties, and player. Renaming movie.mkv to movie.mp4 changes none of those structures and does not convert the file.

Remuxing copies encoded streams into a different container without re-encoding, so it is fast and lossless. Transcoding decodes and re-encodes a stream, changing codec or media properties; it takes longer and can lose quality.

What controls quality and file size?

• Resolution: the pixel dimensions of each frame. More pixels need more bits, all else equal.
• Frame rate: frames per second. Higher rates represent motion more smoothly but create more pictures to encode.
• Bitrate: bits used per second. More bitrate usually permits a closer reconstruction, but returns diminish.
• Encoder effort: a slower preset searches harder for an efficient representation. It usually improves quality at the same bitrate without changing decoder requirements.
• Bit depth: commonly 8 or 10 bits per component. More precision reduces banding and supports HDR workflows, but needs end-to-end support.
• Chroma subsampling: 4:4:4 keeps full colour resolution; 4:2:2 and consumer-standard 4:2:0 reduce it to save data.
• Content: grain, rain, confetti and rapid detail are difficult to compress; a static presentation slide is easy.

Rate-control modes

“1080p” says resolution, not quality. A carefully encoded 1080p file can look better than starved 4K. Likewise, bitrate comparisons only make sense with similar codecs, content, settings and quality targets.

Streaming: playlists, segments and adaptation

Streaming services normally prepare several encodes of the same programme at different resolutions and bitrates. The player measures conditions and switches between them at aligned segment boundaries. This is adaptive bitrate streaming.

• HLS uses an .m3u8 playlist describing media playlists and segments. Segments commonly use fragmented MP4 or MPEG-TS.
• MPEG-DASH uses an MPD manifest describing representations and segmented media, commonly fragmented MP4 or WebM.
• Fragmented MP4 divides media into independently deliverable fragments while retaining MP4 structures and timing.

HLS and DASH are streaming delivery systems, not video codecs. A manifest tells a player where the encoded pieces are and how they relate. The codecs inside still determine whether the device can decode them.

Segments generally begin around a suitable random-access picture so switching is clean. Shorter segments reduce latency and allow faster adaptation but increase request and packaging overhead.

Choosing a format—and finding failures

A practical troubleshooting order

1. Identify the container from the file structure, not only its extension.
2. List every stream: video codec, audio codec, subtitles and attachments.
3. Check properties: profile, level, bit depth, chroma format, dimensions, frame rate, audio channel layout and sample rate.
4. Check the playback path: application, operating system, hardware decoder, display and audio device may each impose limits.
5. Decide whether to remux or transcode. Remux when the codecs work but the container does not; transcode only the incompatible stream when possible.

Tools such as MediaInfo and ffprobe reveal what is actually inside a file. If sound works but the picture does not, the container was opened successfully—the likely problem is the video codec or one of its properties.