Media Converter Service
Case overview
This is a cloud-based media conversion service that takes video and audio files and turns them into HLS streaming format with adaptive bitrate support. The system listens to an SQS queue for new files, spins up a Kubernetes job for each one, and produces a full set of m3u8 playlists and .ts segments ready for any modern video player.
It supports multiple resolutions for video, multiple bitrate tiers for audio, and handles videos with several audio tracks — including different languages or commentary. The service works for multiple projects simultaneously and organises outputs into separate folders on S3 based on the project or publisher.
Goal: Build a fully automated service that converts any video or audio file into HLS format without manual intervention - working across multiple projects, handling diverse content types, and scaling to zero when idle.
Key project info
Industries
Media & Entertainment, Video Streaming Platforms, OTT & Broadcasting Services, Digital Publishing, Content Delivery Networks — any business that needs to deliver video or audio content to end users across different devices and varying network conditions.
Services
SQS Queue Processing with long polling and visibility timeout management, FFmpeg & FFprobe media transcoding and stream analysis, Kubernetes job orchestration with ephemeral volumes and node affinity, Parallel S3 upload pipeline with pre-upload cleanup.
Solutions
Multi-resolution HLS video — 1080p, 720p, 480p, 360p, 240p with adaptive bitrate, Multi-bitrate audio HLS — 32, 64, 96, 128, 192 kbps tiers per audio track, Automatic detection of all audio streams with language metadata, Thumbnails at 2-second mark, low-res MP4 preview, MP3 and WAV copies.
Technologies
AWS SQS, S3, Kubernetes, FFmpeg, FFprobe, H.264, AAC/HE-AAC, HLS/m3u8, NVIDIA NVENC, Intel Quick Sync, EC2 Auto Scaling, Docker, Helm
The challenges
Many Output Formats from One Input
A single video file can produce hundreds of files — five video resolutions, five audio bitrates per audio track, plus thumbnails and previews. We had to organize all of that cleanly and upload it fast.
Multiple Audio Streams per Video File
Some videos come with several audio tracks in different languages. The system had to detect them all automatically, convert each one separately, and build a master playlist that references every track with the right language tag.
Queue Reliability and Retry Logic
With SQS you need to handle duplicates, visibility timeouts, and failed jobs properly. If a job fails, the message should come back to the queue for retry. If the same file gets queued twice, we should not process it again.
Management on Kubernetes
Each conversion job uses a lot of CPU and memory. We had to set up ephemeral volumes, node affinity rules, and proper cleanup so jobs do not pile up or block each other.
Multi-Project Isolation on S3
The service runs for several projects at once. Each one needs its own output path on S3, and we had to support both publisher-based and series-based folder structures without breaking anything.
Automatic Codec Selection
Not every machine has GPU acceleration. The system needed to detect what codecs are available — NVIDIA, Intel Quick Sync, AMD, or just software — and pick the best one automatically.
The process
The entire pipeline runs without human involvement — from the moment a file lands on S3 to the final HLS output being ready for playback. Each step hands off cleanly to the next, with built-in safeguards for failures, duplicates, and resource contention. Below is how the system moves a file through the full conversion cycle.
Queue Polling
The job manager runs on Kubernetes and polls the SQS queue with long polling. When a message arrives, it parses the S3 bucket and object key to identify the source file.
Duplicate Prevention
Before creating a new job, the manager checks for an existing Kubernetes job for the same file. If found, the message is re-queued with a delay to prevent double processing.
Kubernetes Job Creation
A new pod is created from a template with correct resource limits, node affinity for ARM-based instances, and ephemeral volumes for temp files, input, output, and logs.
Video & Audio Conversion
FFmpeg converts video into five HLS resolutions (1080p→240p) with 4‑second segments. Each audio track is detected and converted into five bitrate tiers — 32 to 192 kbps.
Master Playlist Generation
A single m3u8 master file references all video resolutions and audio tracks with language metadata — enabling adaptive streaming and language switching in any modern player.
Upload, Cleanup & Autoscaling
All files upload to S3 in parallel via a 500-thread pool. Old HLS segments are cleaned before the new set lands. The autoscaler scales nodes to zero once the queue empties.
Solutions
The key features of solution
Adaptive Bitrate Video - 5 Tiers —240p (300 kbps) → 1080p (5000 kbps) in one playlist. Auto quality selection based on connection speed.
Multi-Track Audio with Language Tags — Detects and converts all audio tracks, including commentary and descriptions, while preserving language metadata.
Automatic Codec Detection — Uses NVIDIA / Intel / AMD acceleration, or software fallback if no GPU.
Zero Idle Cost Autoscaling — EC2 nodes scale down to zero when the queue is empty. Active jobs are protected from premature termination.
Multi-Project S3 Isolation — Each project has its own S3 output path. Supports both publisher- and series-based folder structures.
Results in numbers
1080p
Five adaptive bitrate tiers from 240p at 300 kbps up to full HD 1080p at 5000 kbps — the player switches quality automatically based on connection speed.
500x
Hundreds of output files — segments, playlists, thumbnails, previews — are uploaded to S3 simultaneously via a thread pool, keeping delivery time minimal.
4sec
4-second HLS segments balance adaptive switching speed with buffering efficiency — the player reacts to network changes within a single segment boundary.
5x
Every audio track — including commentary and description — gets up to five quality levels from 32 kbps (HE-AAC) to 192 kbps (LC-AAC) with loudness normalisation.