What is a Codec? Your beginner’s guide to codecs and how they work

Today, technology advances quickly. As video streaming continues to take over the internet, it should be no surprise that the way we process, store, and transmit videos has evolved tremendously as well. At the heart of this evolution to ever-more efficient processes, like video encoding and the use of codecs. In this article, you’ll learn about the types of codecs available today and their significance in interactive live streaming. Let’s dive in.

What is video encoding?

Before we dive into how video gets encoded, let’s touch on what video encoding is and why you’d do it.

Put simply, video encoding is the process of compressing and converting raw video files into a more usable digital format. This is crucial for reducing the size of the video without significantly compromising its quality, enabling smoother transmission and storage.

What is a codec?

A codec, short for “coder – decoder,” is software (usually an algorithm) used to compress (encode) and decompress (decode) digital data. The idea is to shrink large video files for transmission and then expand them again for playback.

This means that a codec is the tool responsible for video encoding. It also decodes on the other end, meaning it uncompresses the file back into (more or less) its original state once it arrives. If you have ever zipped or unzipped a file to make sharing/emailing it easier, you are already familiar with the concept.

A brief history of video codecs

As we touched on at the beginning of this article, video codecs have continually evolved.

Initially, videos were stored in analog formats like VHS tapes. With the digital revolution, codecs like MPEG-2 emerged, paving the way for alternate storage formats, like DVDs.

Later, advancements like H.264 (which we’ll dive into in more detail a bit later) enabled high-quality streaming on platforms like YouTube. Today, we have even more advanced codecs like H.265 and AV1. As new iterations emerge, each builds on the last to better refine video quality and compression ratios.

What are the types of codecs?

There are a lot of forms of media – still images, audio, video, and general data. These file types have different needs for both compression and decompression, so there are different codecs designed to handle them.

Lossless vs. lossy

There are two types of codecs: lossless and lossy.

• Lossless codecs
  Lossless codecs compress data in a way that is, well, without loss (lossless). When using a lossless codec, in other words, the file is still completely intact for a perfect reconstruction of the original data when it’s eventually decompressed.By identifying and eliminating redundant information, a lossless codec can reduce the file size a touch without losing quality. Lossless codecs are used most often when you need to preserve the highest possible quality, like for video storage or future video editing.
• Lossy codecs
  Lossy codecs, on the other hand, achieve higher compression ratios (meaning they can make the file much smaller) by discarding bits of data deemed less significant. The scrapped information is typically based on an algorithm that has taken the limitations of human perception into account.

For example, the human brain can only process so many frames per second, so deleting extra frames reduces the file size for a relatively low amount of detectable loss. Lossy codecs are used in scenarios where the primary concerns revolve around storage or bandwidth limitations. This is the case for most streaming services or video-sharing platforms.

How do codecs work?

As we alluded to, lossless codecs allow you to retain what is essentially the original file.

There are two styles of compression that work behind the scenes to compress the file(s): intra-frame and inter-frame. Most lossy codecs use a combination of the two to achieve the end compression result. Here’s a bit about the two types and how they work for streaming compression specifically:

Intra-frame compression (AKA, spatial compression):

Intra-frame compression compresses each individual frame of a video independently without taking the surrounding frames into consideration, so compression is applied directly within that frame.

This technique is great for compressing relatively static or less complex video sections. Intra-frame compression retains high detail within a single frame, making it suitable for preserving image quality.

Lossless codecs sometimes employ intra-frame compression to maintain pixel-perfect accuracy. However, this technique results in much more choppiness in complicated or high-movement video scenes.

Inter-frame compression (AKA, temporal compression):

Inter-frame compression takes advantage of the temporal redundancy between consecutive frames of a video. Instead of encoding each frame independently, inter-frame compression identifies and encodes only the differences (or motion vectors) between frames, known as prediction or delta frames.

In other words, it stores the changes between frames rather than the entire content of each frame. This approach significantly reduces the amount of data required to represent a video. By referencing previously encoded frames, inter-frame compression achieves higher compression ratios, particularly for videos with repetitive or similar motion patterns. However, it introduces dependencies between frames, meaning that the decompression process requires access to previous frames for accurate reconstruction. Lossy codecs often employ inter-frame compression to achieve higher compression efficiency while maintaining perceptual quality.

What’s the best type of codec for streaming?

Long story short, lossy codecs are best for video streaming. Let’s get into why:

While some codecs, like the Apple Animation codec, offer lossless compression for video, they are not well suited for streaming due to the still-too-large file size. Lossless codecs excel in preserving the highest quality, but that makes them less practical for situations that have bandwidth or storage limitations. As such, streaming platforms tend to rely on lossy codecs to reach higher compression ratios to make file sharing and streaming more seamless.

Best video codecs for streaming

Now that you understand the general types of codecs, let’s get into the details. What is the best codec for streaming today? Different applications have different codec requirements.

For streaming, the most notable codecs are:

• H.264/AVC. At the time of writing, H.264/AVC dominates the streaming world due to its balance between compression efficiency and decompression quality integrity.
• H.265/HEVC. H.265/HEVC is a next-generation codec that offers up to 50% better compression than H.264. This makes it ideal for 4K streaming.
• AV1. AV1 is a royalty-free codec, emerging as a competitor to H.265 with similar or better compression rates.
• VP9. Developed by Google, it’s a royalty-free alternative to H.265 and is extensively used on YouTube.
• H.266/VVC. The up-and-coming successor to H.265 as it promises even better compression, especially for 8K content.

Encoding best practices

When trying to determine the codec that’s right for your streaming use case, here are some encoding best practices to consider:

• Bitrate selection. Choosing the right bitrate is crucial for balancing quality and file size. We have an article here that dives into the concept of bitrates and why they matter for your stream if you’d like a refresher.
• Resolution matching. Next, make sure the resolution matches the intended display device.
• Multi-pass encoding. Finally, consider multi-pass encoding. Doing multiple passes can enhance quality by analyzing the video more thoroughly.

Multi-codec video delivery

The multi-codec video delivery approach involves encoding videos in multiple codecs. When a user requests a video, the server then chooses the best codec for that user’s device and network conditions, ensuring optimal playback.

What’s a video container format?

A video container format, sometimes called a wrapper, houses both video and audio streams, metadata, and (if included), subtitles. Examples include MP4, AVI, and MKV.

The container keeps all these individual components in one file, making it easier to manage, share, and stream the content in one piece. A codec compresses (encodes) the pieces, after which they get wrapped in a single container for easy transmission.

What’s the best codec for device compatibility?

When livestreaming audio and video to audiences around the world, it’s important to be cognizant of the devices your viewers use to connect to you. Is there a “most compatible” codec that will work on the highest number of devices?

The answer to that question is yes.

There are a lot of codecs out there, but for streaming, the most widely used for its efficient balance between lossy compression and device compatibility is H.264 Advanced Video Coding (AVC).

H.264 is one of the most widely supported video codecs and, as such, is commonly used for streaming, video conferencing, and video playback on various devices. It’s compatible with a wide range of smartphones, tablets, smart TVs, and web browsers.

It’s the codec used in Livery’s Cloud Encoder technology for that reason.

Video codecs vs. containers: What’s the difference?

While the terms “video codec” and “video container” often get used interchangeably, codecs and containers each serve different purposes:

• Codec. A codec focuses on compressing and decompressing video and audio data. H.264 is the most widely used codec for video streaming.
• Container. A container, on the other hand, wraps the compressed data, along with other elements like metadata, into a single file making it easy to transmit. MP4 is an example of a video container (a single file that holds all audio, video, and related data).

Simply put, a codec determines how the video is compressed, while a container determines how that compressed video is stored and transmitted.

Hardware vs. software codecs

So you now understand what a codec is, what it does, and how to find the best one, but should you invest in a hardware or software codec? Let’s talk about it.

Pros of hardware codecs:

When purchasing a hardware codec, you’re buying a physical piece of equipment (usually a chip or embedded component) designed for this specific purpose. Here are some advantages to hardware codecs:

• Function. You’re buying a device optimized for this purpose, so it tends to work efficiently. This will be helpful in the event your streaming device is running low on battery, for example.
• Device compatibility. Similarly, you’re buying a specific piece of equipment designed for your machine, which should make it function on that machine seamlessly.

Pros of software codecs:

When using a software-based codec, you’re using a program that’s either installed on your device or hosted in the cloud to perform this function. Here are some advantages to software-based codecs:

• Flexibility. You can use a software-based codec on a wide range of devices (if you want to stream from your phone, tablet, or laptop, for example. They’re not tied to a specific configuration like hardware codecs.
• Updates. With software, you’re likely to receive compatibility updates, improvements, bug fixes, and more as time goes on, keeping it more compatible for longer.

For these reasons, Livery uses a cloud software encoder.

Encoding vs. transcoding

You may have also come across the term “transcoding” in your research. What is it and how to does it relate to encoding?

Encoding is the initial compression of raw video. Transcoding, on the other hand, is the process of converting an already encoded video into a different encoded format. It’s often used for improved compatibility or further file compression.

Closing thoughts: Codecs and video encoding

Anyone in the digital media space should take the time to understand video encoding and the available codecs that make it happen. As technology advances and user demands shift over time, keeping up-to-date on these topics ensures you’ll stay ahead of the curve of high-quality, efficient video delivery.

Get a Demo

If you’re interested in livestreaming, Livery offers a best-in-class solution with out-of-the-box interactivity tools that make your streams come to life.

Give Livery a try for yourself through a free demo. Click here to schedule yours.