How Video Streaming Actually Works: A Deep Dive into Codecs, Transcoding, and Buffering

You're watching your favorite show when suddenly—spinning buffer wheel. Or maybe you've wondered how Netflix can stream 4K video to millions of people simultaneously without the internet collapsing. The technology behind video streaming is actually fascinating, and understanding it can help you troubleshoot problems, optimize your own media server, or just appreciate the engineering marvel happening every time you click play.

Let's break down the three core technologies that make streaming work: codecs, transcoding, and buffering.

What Are Codecs and Why Do We Need Them?

A codec (short for "coder-decoder") is software that compresses and decompresses video files. Without codecs, streaming would be nearly impossible.

Here's why: A single minute of uncompressed 1080p video at 30fps takes up about 3.6 GB of storage. That's roughly 216 GB per hour. Even with today's fast internet connections, streaming that much data in real-time just isn't feasible for most people.

Codecs solve this by using clever compression algorithms to reduce file sizes by 95% or more while maintaining visual quality. The most common codecs you'll encounter are:

H.264 (AVC): The industry standard for years, supported by virtually every device

H.265 (HEVC): Offers 50% better compression than H.264 but requires more processing power

VP9: Google's open-source alternative, used heavily by YouTube

AV1: The newest codec, offering even better compression than HEVC with no licensing fees

How Compression Actually Works

Codecs use two types of compression:

Spatial compression reduces redundancy within a single frame. If your video shows a blue sky, the codec doesn't store the color value for every single pixel—it basically says "make the top half of the frame blue" in a much more efficient way.

Temporal compression is even more powerful. It takes advantage of the fact that most video frames are very similar to the frames before and after them. Instead of storing every frame completely, the codec stores "keyframes" (complete images) every few seconds, then only records what changed between keyframes.

This is why seeking in video files can sometimes be slow—your player has to decode from the last keyframe to reconstruct the frame you're trying to jump to.

The Art and Science of Transcoding

Transcoding is the process of converting video from one codec or format to another. When you upload a video to YouTube, it doesn't just store your original file—it transcodes it into multiple versions.

Why Multiple Versions?

Here's a typical transcoding setup for a streaming service:

This approach is called adaptive bitrate streaming (ABR). Your player automatically switches between these versions based on your current network conditions. When your connection slows down, it drops to a lower quality to prevent buffering. When bandwidth improves, it switches back up.

The Transcoding Process

When you run a media server like Plex or Jellyfin, transcoding happens in real-time when a client requests a stream. The process looks like this:

bash
Example ffmpeg transcoding command

ffmpeg -i input.mp4 \
  -c:v libx264 \
  -preset medium \
  -crf 23 \
  -c:a aac \
  -b:a 128k \
  output.mp4

This command:

Takes input.mp4 as the source

Encodes video using H.264 (libx264)

Uses a "medium" preset balancing speed and quality

Sets quality level to 23 (where 0 is lossless and 51 is terrible)

Encodes audio as AAC at 128 kbps

The preset parameter is crucial. Faster presets encode quickly but create larger files. Slower presets take more time but achieve better compression. For live transcoding, you need faster presets. For preparing files ahead of time, slower presets give better results.

Direct Play vs. Transcoding

The best streaming experience happens with direct play—when the client can play the original file without any transcoding. This:

Reduces server CPU load

Eliminates quality loss from re-encoding

Starts playback faster

Saves electricity and server resources

This is why choosing the right codec when storing your media matters. If you encode everything in H.264, most devices can direct play. If you use HEVC, older devices will require transcoding.

Buffering: Your Streaming Safety Net

Buffering gets a bad rap, but it's actually a crucial technology that makes streaming work at all.

How Buffering Works

When you start playing a video, your player doesn't just download and display the current second—it downloads ahead. Typically, players buffer 5-30 seconds of video before playback begins, then continuously download ahead as you watch.

This buffer is stored in RAM on your device. It acts as a safety cushion against network fluctuations. If your internet speed drops for a few seconds, you keep watching from the buffer while your player downloads more data.

Why Videos Buffer (and How to Fix It)

The dreaded spinning wheel appears when your buffer runs out—when the player can't download data fast enough to keep up with playback. Common causes:

1. Network congestion: Your internet speed temporarily drops below the video bitrate

2. Server limitations: The streaming server can't deliver data fast enough (common with overloaded media servers)

3. Transcoding bottleneck: If your media server is transcoding in real-time, a slow CPU can't encode fast enough

4. Poor bitrate selection: The player chose a quality too high for your connection

Solutions depend on the cause:

For network issues, manually select a lower quality

For server problems, consider upgrading your media server hardware

For transcoding bottlenecks, enable hardware acceleration (GPU encoding) or pre-transcode your files

Some players let you increase buffer size, though this uses more RAM

Advanced: HLS and DASH Protocols

Modern streaming uses protocols like HLS (HTTP Live Streaming) or DASH (Dynamic Adaptive Streaming over HTTP). These break videos into small chunks (typically 2-10 seconds each) and serve them over regular HTTP.

Each chunk exists in multiple quality levels. Your player constantly measures network performance and requests the appropriate quality chunk. This is why you can see quality shift mid-stream—each chunk can be a different bitrate.

bash
HLS playlist example (m3u8 file)

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-STREAM-INF:BANDWIDTH=2000000,RESOLUTION=1920x1080
1080p.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1000000,RESOLUTION=1280x720
720p.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=500000,RESOLUTION=854x480
480p.m3u8

Putting It All Together

Here's what happens in the first few seconds when you click play on a streaming video:

Your player requests the video manifest (playlist of available qualities)

Based on measured bandwidth, it selects an appropriate starting quality

It downloads the first several chunks (building the buffer)

Playback begins while continuing to download ahead

The player monitors network performance and switches qualities as needed

If the source codec/format isn't compatible, the server transcodes on-the-fly

This entire process happens seamlessly, usually in under 2-3 seconds.

Bringing It Home

Understanding codecs, transcoding, and buffering helps you make better decisions about your media setup. Choose H.264 for maximum compatibility or HEVC for better compression. Pre-transcode files if your server struggles with real-time encoding. Adjust buffer sizes if you have a fast but unstable connection.

If you're running your own media server or seedbox and want to ensure smooth streaming, having adequate storage and processing power matters. Services like SonicBit make this easy by providing high-performance seedboxes with enough horsepower to handle transcoding, plenty of storage for your media files, and the bandwidth to stream without buffering. You can even use Remote Upload to sync your media to cloud storage services like Google Drive or OneDrive for redundancy.

Sign up free at SonicBit.net and get 4GB storage. Download our app on Android and iOS to access your seedbox on the go.