What is the difference between media containers and codecs

We receive a lot of similar questions in our technical support, and many of them concern the choice of the optimal format for video or audio. Users often choose MP4 for audio because the number is higher than that of MP3. Or they choose between ALAC and M4A. I am tired of explaining every time that these are completely incomparable parameters and that an MP4 file may well contain audio compressed by the MP3 codec. This is how this article appeared.

I will try, without going too deep into technical details, to explain the differences between a codec and a container. Which of them is responsible for the quality of the video, and which for the presence of subtitles and switching audio tracks.

1 Why do people confuse containers and codecs

One of the most common mistakes is to think that the file extension (for example, .mp4) tells us what is inside. It simply allows us to identify the format of the file itself, and not always. I have encountered many MP3 files that had a WAV extension. But even if the extension is specified correctly, MP4 does not necessarily contain video. It is just a wrapper that can pack different types of data, including video, music or text.

Yes, sometimes you can find multimedia data in the so-called "raw" format, without using a container. For example, MP3 or GSM. But such files often cause problems, because without the correct header it is difficult to determine the correct sampling frequency or number of channels. Players usually rely on some "standard" values and do not always guess. A striking example of this is the incorrect determination of the number of channels in AAC files, which can be compressed with both a regular codec and HE-AAC.

2 What is a video codec?

A codec is an algorithm that is responsible for compressing and decompressing data so that you can fit as much data as possible into a smaller volume. In essence, it is a specialized archiver, similar to ZIP, but taking into account the features of video, audio or images. It is responsible for how frames, motion vectors, audio channels are stored, how they are encoded to save space, how they are restored during playback and what additional features are supported.

Many people mistakenly believe that a codec is software, but it can be implemented in hardware, without the ability to configure or update.

Main characteristics of a codec:

• Bitrate is the number of bits used per second of video. The higher the bitrate, the better the quality, but the larger the file size.
• Color space - the more colors are supported, the richer and more natural the image. To reduce the amount of data, color subsampling is used, for example, yuv422. Support for HDR and Dolby Vision is responsible for the dynamic range.
• Resolution is the maximum number of pixels that the codec can process. For example, H.264 works up to 4K, HEVC and AV1 - up to 8K and higher.
• Compression method — intercoding (I-frames), predictive (P-frames), bidirectional (B-frames). These technologies allow you to reduce the volume of video while maintaining high quality.
• Hardware acceleration — support by specialized chips (GPU, DSP) for fast decoding. For example, HEVC often requires hardware acceleration on mobile devices, otherwise the video slows down.

3 And what is a container?

Instead of writing dry technical documentation, I will simply give an analogy. Imagine that you have a bag in which you put not only a shirt, but also socks, a belt, keys. Well, the container is exactly this bag. It defines how the data inside the file is stored and organized: video, audio, subtitles, menus, thumbnails, timing.

The container stores information in the form of chunks or atoms - small blocks, each of which contains information about the type of data, its location and duration. For example, the MP4 container uses atoms such as moov (metadata), mdat (media data), trak (individual tracks).

These blocks are alternated in such a way as to ensure precise synchronization between video and audio. That is, when playing video and audio, the container must ensure that you hear the actor's voice at the very moment when he opens his mouth.

The purpose of any container is to organize data, although the internal structure and capabilities may differ.

Main characteristics of container:

• Hardware device support — MP4 is supported by almost all devices, including TVs, smartphones, and media players. MKV is loved by computer users, but not all TVs understand it.
• VFR (Variable Frame Rate) support — variable frame rate is important for some types of content, such as streaming video. MKV and MOV support VFR, while MP4 does not always.
• Chapters — the ability to divide a movie into chapters. Supported by MKV, MOV, but not all MP4 variants.
• Streaming — the container should allow you to start playback without downloading the entire file. MP4 and WebM are good for this, MKV is worse because metadata is usually located at the end of the file.
• Metadata (additional information) — track description, authorship, genre, cover, etc. MKV and MOV support rich metadata, MP4 — limited.

4 Popular Media Containers

We have already mentioned the AVI (Audio Video Interleaved) format introduced by Microsoft. AVI files can contain both audio and video data in a file container that allows synchronous audio and video playback. AVI supports multi-streaming audio and video, although these features are rarely used. Since AVI was designed for Windows, it lacks some of the features that newer containers such as MP4 have. With the adoption of mobile devices, this format has lost its relevance.

MP4 is a newer format developed by the Motion Pictures Expert Group. Unlike AVI, it allows streaming video over the Internet. It is the recommended format for online video, and services such as Vimeo and YouTube specify it as the preferred format. The MP4 container uses MPEG-4 or H.264 encoding, and uses AAC or AC3 for audio. It is widely supported by most consumer devices.

Finally, the most modern MKV format (also known as Matroska), which is designed with the future in mind. The container supports virtually any audio or video format, making it adaptable and efficient. It is considered one of the best ways to store audio and video files. Its unique ability to play damaged video files, support for multiple subtitle tracks, menus and DVD chapters allows it to quickly gain support in both software and hardware.

5 Popular Video Codecs

AVC (H.264) is the standard codec for all available platforms. It can use both lossy and lossless compression depending on the settings you choose when encoding, such as frame rate, quality, and target file size. H.264 almost always uses hardware acceleration to encode video, and audio is often encoded using AAC or MP3 audio codecs, depending on the size and quality you are targeting.

HEVC, also known as H.265, is a video compression standard designed as part of the MPEG-H project as a successor to the widely used AVC(H.264). In comparison to AVC, HEVC offers from 25% to 50% better data compression at the same level of video quality, or substantially improved video quality at the same bit rate. Unfortunately, high licensing fees prevent its widespread use.

AV1 is a video codec that was released four years ago, in March 2018. It was created to surpass the previous generation codecs - HEVC, VP9, H.264 and VP8. Due to new optimizations, AV1 compresses video 30-50% better than H.264 or VP8, and up to 30% better than HEVC. Widespread support among streaming services quickly resolved the low performance issues and made it usable in production.

6 Why are some codecs and containers incompatible?

Because containers have strict rules about what data can be stored inside. For example:

• MP4 officially supports H.264, HEVC, AAC, but does not always correctly accept VP9 or AV1.
• MKV is more flexible, it allows you to store almost any codec, but not all players can read it.
• MOV — tied to Apple and works well with ProRes, but can cause problems on Android.

Additionally, even if the container theoretically supports the codec, in practice everything depends on the implementation in a specific program or device. For example, an old TV may not play HEVC in MKV, because the manufacturer did not provide the necessary decoder.

7 Optimal combinations of codecs and containers

For streaming

If you upload videos to YouTube, Twitch or Vimeo, it is best to choose H.264 in an MP4 container. This format is compatible with almost all devices and platforms, buffers quickly and copes well with compression. If you have a modern channel and your audience has access to good Internet, you can try VP9 or AV1 to save on traffic.

For storing movies

If you collect movies, you should choose HEVC (H.265) in an MKV container. This allows you to maintain maximum quality with a smaller file size. Plus, MKV allows you to store several versions of subtitles and audio tracks - for example, English and Russian versions, or the original voiceover with DTS sound.

For editing

Professionals prefer ProRes in a MOV container or DNxHD in an MXF container because these formats are easier to handle in programs like Adobe Premiere, DaVinci Resolve or Final Cut Pro. They are less compressed, which reduces the load on the processor during editing and allows for more precise work with color and effects.

8 Interesting facts that few people know

AVI and AV1 are not the same thing. Apparently the developers of the AV1 codec thought it was funny, but not everyone can immediately distinguish between 1 and I.

The AVI format is one of the oldest, but it is still alive. However, it has file size limitations (maximum 2 GB), which makes it unsuitable for modern projects.

MKV is an open format created by the community. Not all TVs support it, but many media players, such as VLC, love it infinitely.

MP4 - does not always contain H.264. There may be other codecs inside, for example, MPEG-4 Part 2 or even HEVC.

AV1 is the first truly competitive open codec that can replace H.264 and VP9. But its implementation is not yet widespread due to the complexity of hardware decoding.

MOV is an Apple format that was created back in 1991. It is still actively used in the film industry.

9 Conclusion

As you can see, everything is quite simple: the container is responsible for the correct placement of heterogeneous data within one file, and the codec is responsible for the compression of the data itself. If the data is homogeneous, then you can do without a container.

Remember that there is no single "best" format for all tasks. It all depends on your goals - some formats are easier to edit video with, while others are optimized for streaming.

But the most important thing is that now you will definitely not get confused between MP3 and MP4.

---

Related Articles