NVENC (NVIDIA GPU) Hardware Encoding — Fast GPU Conversion with FFmpeg

Target OS: Windows / Linux (a CUDA driver is required)

What Is NVENC

NVENC is the video encoder built into NVIDIA GPUs. It is much faster than encoding on the CPU and uses very little CPU. In exchange, it tends to compress a little less well than a CPU encoder.

Prerequisites

You need the following:

1. An NVIDIA GPU with NVENC, from the Maxwell generation (GeForce GTX 750, the 900 series and so on) or newer. NVIDIA’s support matrix lists which models support which codecs.
2. An NVIDIA driver new enough for your FFmpeg build. The gyan.dev build of FFmpeg 8.1 needs 570.0 or later.
3. An FFmpeg build with NVENC. Run ffmpeg -encoders | grep nvenc and check that h264_nvenc and hevc_nvenc are listed. In the Windows Command Prompt, use findstr nvenc instead of grep.

The list only shows what the build contains. It does not prove that your GPU and driver can use NVENC. To check that, encode a one-second test clip. If it finishes without an error, NVENC works:

※ This command requires an NVIDIA GPU environment
ffmpeg -f lavfi -i testsrc2=duration=1 -c:v h264_nvenc -f null -

Basic h264_nvenc Command

The simplest command encodes the video with h264_nvenc and copies the audio as is:

※ This command requires an NVIDIA GPU environment
ffmpeg -i input.mp4 -c:v h264_nvenc -c:a copy output_nvenc.mp4

To set the quality, use -cq. The value runs from 1 to 51, and lower means better quality (0 means automatic):

※ This command requires an NVIDIA GPU environment
ffmpeg -i input.mp4 -c:v h264_nvenc -rc vbr -cq 23 -c:a aac -b:a 128k output_nvenc.mp4

hevc_nvenc (H.265)

H.265 gives a smaller file than H.264 at about the same quality:

※ This command requires an NVIDIA GPU environment
ffmpeg -i input.mp4 -c:v hevc_nvenc -rc vbr -cq 28 -c:a aac -b:a 128k output_hevc_nvenc.mp4

Presets

-preset chooses between speed and quality:

※ This command requires an NVIDIA GPU environment
ffmpeg -i input.mp4 -c:v h264_nvenc -preset p4 -rc vbr -cq 23 output.mp4

There are seven presets, from -preset p1 (fastest) to -preset p7 (best quality). fast, medium, and slow also work. They are other names for p1, p4, and p7, and slow also turns on two-pass encoding.

Combining GPU Decoding + NVENC Encoding

If the GPU decodes as well, the CPU load drops further:

※ This command requires an NVIDIA GPU environment
ffmpeg -hwaccel cuda -hwaccel_output_format cuda -i input.mp4 -c:v h264_nvenc -preset p4 -rc vbr -cq 23 output.mp4

-hwaccel_output_format cuda keeps the decoded frames in GPU memory and hands them straight to NVENC.

Bitrate-Targeted Encoding

To encode at a set bitrate, use -rc cbr, which keeps the bitrate constant:

※ This command requires an NVIDIA GPU environment
ffmpeg -i input.mp4 -c:v h264_nvenc -rc cbr -b:v 4M -maxrate 4M -bufsize 8M output.mp4

Common Errors and Fixes

No capable devices found

FFmpeg can’t find an NVIDIA GPU that can run NVENC. It may print No CUDA capable devices found instead. Run nvidia-smi to check that the GPU is detected. If the driver is too old, FFmpeg names the version it needs, for example The minimum required Nvidia driver for nvenc is 570.0 or newer.

OpenEncodeSessionEx failed: out of memory (10)

Other processes are already using all the NVENC sessions allowed. On GeForce cards, NVIDIA limits how many NVENC sessions can run at once in the whole system, counting all GeForce cards in it together. The limit depends on the driver and the product line, so look up your card in NVIDIA’s support matrix.

Cannot load libcuda.so.1 (Linux)

FFmpeg can’t find the CUDA library. Look for it with ldconfig -v | grep cuda, or check whether it is in /usr/lib/x86_64-linux-gnu/.

Choosing Between Software and Hardware Encoding

Speed and Quality Ballparks

Speed depends heavily on the GPU generation, the input resolution, and the preset.

• Encode speed: On a 1080p source, the basic h264_nvenc command and the -cq 23 example above ran at about 260–270 fps (about 9× realtime). CPU libx264 -preset medium -crf 23 ran at 54 fps (about 1.8×), so NVENC was about 5× faster. Test setup: RTX 4090, Core i9-14900KF, FFmpeg 8.1, Big Buck Bunny (© Blender Foundation, CC BY 3.0) at 1080p, 30 fps, 120 s; median of 3 runs each.
• Quality: To reach the same VMAF score (a measure of picture quality), NVENC usually needs a higher bitrate than libx264. The gap depends on the GPU generation and the content, and it is smaller on newer generations. Ada Lovelace (RTX 40) is reported to close much of it.
• When to use it: NVENC is several times faster, so it is the usual choice for streaming, large batch jobs, and work on a deadline. For an archive you encode once and keep for a long time, libx265 compresses better and is worth the extra time.

Common Pitfalls

-preset slow has no effect / presets seem ignored

• Symptom: -preset slow doesn’t give the speed and quality you expected.
• Cause: fast, medium, and slow are other names for p1, p4, and p7, and they do work. But slow also turns on two-pass encoding (-multipass fullres), so it is even slower than p7.
• Fix: Use p1–p7 so you know exactly what you get, for example -preset p7 -tune hq. Use p6–p7 for high quality, p4 for a balance, and p1–p2 for low latency.

B-frames added but compression doesn’t improve

• Symptom: Adding -bf 3 hardly makes the file smaller, or seems to be ignored.
• Cause: B-frames are frames that refer to the frames before and after them, which saves space. NVENC supports them for HEVC only from the Turing generation (RTX 20), so older GPUs may not use HEVC B-frames. H.264 B-frames work on most generations.
• Fix: Check your GPU generation and use -b_ref_mode middle -bf 3. If HEVC B-frames don’t work on an older GPU, raise the bitrate to keep the quality, or use H.264.

Encoding two or more streams at once fails with out of memory (10)

• Symptom: A batch runs several NVENC processes at once, and the ones over the session limit fail with OpenEncodeSessionEx failed.
• Cause: You hit NVIDIA’s limit on concurrent NVENC sessions. On GeForce cards it is currently 12 for the whole system, counting all GeForce cards in it together, so a second GeForce card does not raise it. Professional cards that NVIDIA lists as “Unrestricted” have no fixed number; the encoder’s capacity and memory set the limit.
• Fix: Look up your GPU’s limit in NVIDIA’s support matrix and run no more encodes at once than that, or update the driver. Writing several outputs from one process can also be more stable.

CPU-decode → GPU-encode isn’t as fast as expected

• Symptom: You switched to NVENC, but CPU usage stays high and the speed doesn’t go up.
• Cause: The CPU still does the decoding. Decoding on the CPU and copying the frames from the CPU to the GPU slow everything down.
• Fix: Put -hwaccel cuda -hwaccel_output_format cuda before -i. The GPU then decodes too, and the frames go to NVENC without leaving GPU memory. This alone can leave the decoder threads using the CPU. In the test setup above, also adding -threads 1 before -i took the run from about 270 fps and 56 s of CPU time to about 460 fps and 2 s.

Related Articles

• VAAPI (Linux) Hardware Encoding
• VideoToolbox (macOS) Hardware Encoding
• Compress Video — CRF and Target Bitrate

FAQ

How does NVENC compare to libx264 in image quality?

At the same bitrate, NVENC looks slightly worse. The newest NVENC (Ada Lovelace and later) comes close to libx264 at -preset medium, and given its speed it is good enough in practice.

NVENC isn’t working

Check that the GPU is an NVIDIA card, the driver is up to date, and your FFmpeg was built with --enable-nvenc. ffmpeg -encoders | grep nvenc shows whether the build has NVENC.

Should I use CRF or CQ?

Use -cq with NVENC (-crf is for libx264). The value runs from 1 to 51 (0 means automatic). As with libx264’s CRF, lower means better quality. Values of 19–23 are common.

Should I use B-frames?

Yes. Turning on B-frames with -b_ref_mode middle -bf 3 compresses much better. HEVC B-frames need a Turing-generation GPU or newer, so they may not work on older cards.

Can I do streaming delivery with NVENC?

Yes, it works well for low-latency streaming. For the lowest latency, combine -tune ll or -tune ull (ultra-low latency) with -preset p1.