Target OS: Windows / Linux (a CUDA driver is required)

What Is NVENC

NVENC is the video encoder built into NVIDIA GPUs. It is much faster than encoding on the CPU and uses very little CPU. In exchange, it tends to compress a little less well than a CPU encoder.

Item NVENC Hardware libx264 Software
Speed About 2–3× faster (measured at 1080p; see below) Baseline
CPU usage Low (GPU processing) High
Compression efficiency h264_nvenc: 2–18% larger at the same quality (measured against libx264 medium; see below) Baseline
Supported codecs H.264, H.265, AV1 (RTX 40 and later) H.264

Prerequisites

You need the following:

  1. An NVIDIA GPU with NVENC, from the Maxwell generation (GeForce GTX 750, the 900 series and so on) or newer. NVIDIA’s support matrix lists which models support which codecs.
  2. An NVIDIA driver new enough for your FFmpeg build. The gyan.dev build of FFmpeg 8.1 needs 570.0 or later.
  3. An FFmpeg build with NVENC. Run ffmpeg -encoders | grep nvenc and check that h264_nvenc and hevc_nvenc are listed. In the Windows Command Prompt, use findstr nvenc instead of grep.

The list only shows what the build contains. It does not prove that your GPU and driver can use NVENC. To check that, encode a one-second test clip. If it finishes without an error, NVENC works:

※ This command requires an NVIDIA GPU environment
ffmpeg -f lavfi -i testsrc2=duration=1 -c:v h264_nvenc -f null -

Basic h264_nvenc Command

The simplest command encodes the video with h264_nvenc and copies the audio as is:

※ This command requires an NVIDIA GPU environment
ffmpeg -i input.mp4 -c:v h264_nvenc -c:a copy output_nvenc.mp4

To set the quality, use -cq. The value runs from 1 to 51, and lower means better quality (0 means automatic):

※ This command requires an NVIDIA GPU environment
ffmpeg -i input.mp4 -c:v h264_nvenc -rc vbr -cq 23 -c:a aac -b:a 128k output_nvenc.mp4

hevc_nvenc (H.265)

H.265 gives a smaller file than H.264 at about the same quality:

※ This command requires an NVIDIA GPU environment
ffmpeg -i input.mp4 -c:v hevc_nvenc -rc vbr -cq 28 -c:a aac -b:a 128k output_hevc_nvenc.mp4

Presets

-preset chooses between speed and quality:

※ This command requires an NVIDIA GPU environment
ffmpeg -i input.mp4 -c:v h264_nvenc -preset p4 -rc vbr -cq 23 output.mp4

There are seven presets, from -preset p1 (fastest) to -preset p7 (best quality). fast, medium, and slow also work. They are other names for p1, p4, and p7, and slow also turns on two-pass encoding.

Preset Speed Quality
p1 / fast Fastest Lower
p4 / medium Balanced Standard
p7 / slow Slow High

Combining GPU Decoding + NVENC Encoding

If the GPU decodes as well, the CPU load drops further:

※ This command requires an NVIDIA GPU environment
ffmpeg -hwaccel cuda -hwaccel_output_format cuda -i input.mp4 -c:v h264_nvenc -preset p4 -rc vbr -cq 23 output.mp4

-hwaccel_output_format cuda keeps the decoded frames in GPU memory and hands them straight to NVENC.

Bitrate-Targeted Encoding

To encode at a set bitrate, use -rc cbr, which keeps the bitrate constant:

※ This command requires an NVIDIA GPU environment
ffmpeg -i input.mp4 -c:v h264_nvenc -rc cbr -b:v 4M -maxrate 4M -bufsize 8M output.mp4

Common Errors and Fixes

No capable devices found

FFmpeg can’t find an NVIDIA GPU that can run NVENC. It may print No CUDA capable devices found instead. Run nvidia-smi to check that the GPU is detected. If the driver is too old, FFmpeg names the version it needs, for example The minimum required Nvidia driver for nvenc is 570.0 or newer.

OpenEncodeSessionEx failed: out of memory (10)

Other processes are already using all the NVENC sessions allowed. On GeForce cards, NVIDIA limits how many NVENC sessions can run at once in the whole system, counting all GeForce cards in it together. The limit depends on the driver and the product line, so look up your card in NVIDIA’s support matrix.

Cannot load libcuda.so.1 (Linux)

FFmpeg can’t find the CUDA library. Look for it with ldconfig -v | grep cuda, or check whether it is in /usr/lib/x86_64-linux-gnu/.

Choosing Between Software and Hardware Encoding

Use Case Recommendation
Archiving / high quality libx264 / libx265 (software)
Real-time / live streaming NVENC
Batch conversion / time-critical NVENC
Environment without a GPU Software encoding

Speed and Quality Ballparks

Speed depends heavily on the GPU generation, the input resolution, and the preset.

  • Encode speed: On 60 seconds of live-action 1080p at 24 fps, h264_nvenc (-preset p4) ran at 434 fps and -preset p7 at 279 fps; CPU libx264 (-preset medium) ran at 156 fps, so p4 was about 2.8× faster. For H.265 the gap widens to about 6×: hevc_nvenc (p4) 417 fps against libx265 (medium) 66 fps. Test setup: RTX 4090, Core i9-14900KF, FFmpeg 8.1, a 10-second Tears of Steel clip (© Blender Foundation, CC BY 3.0) joined into 60 seconds; median of 3 runs each.
  • Quality: At the same VMAF score (a measure of picture quality), h264_nvenc files were 2–5% larger than libx264 on live action and 11–18% larger on animation (Big Buck Bunny). hevc_nvenc against libx265 was −2% to +4% on live action and 7–14% on animation (VMAF 85–93; each setting run at 5–6 CRF/CQ values and compared at equal VMAF). The gap depends on the content; on these two clips it was wider on animation.
  • When to use it: NVENC is 2–6× faster, so it suits streaming, large batch jobs, and work on a deadline. hevc_nvenc at p7 (166 fps) ran about as fast as libx264 (156 fps) and gave files 22–30% smaller at the same quality, which is worth considering when the recipient can play H.265. For an archive you encode once and keep, spending the time on libx265 or SVT-AV1 is also worth considering. The gap depends on the content, quality and preset: on the live-action clip, hevc_nvenc p7 came out about the same size as libx265 medium. The measurements are published on the data page.

Common Pitfalls

-preset slow has no effect / presets seem ignored

  • Symptom: -preset slow doesn’t give the speed and quality you expected.
  • Cause: fast, medium, and slow are other names for p1, p4, and p7, and they do work. But slow also turns on two-pass encoding (-multipass fullres), so it is even slower than p7.
  • Fix: Use p1–p7 so you know exactly what you get, for example -preset p7 -tune hq. Use p6–p7 for high quality, p4 for a balance, and p1–p2 for low latency.

B-frames added but compression doesn’t improve

  • Symptom: Adding -bf 3 hardly makes the file smaller, or seems to be ignored.
  • Cause: B-frames are frames that refer to the frames before and after them, which saves space. NVENC supports them for HEVC only from the Turing generation (RTX 20), so older GPUs may not use HEVC B-frames. H.264 B-frames work on most generations.
  • Fix: Check your GPU generation and use -b_ref_mode middle -bf 3. If HEVC B-frames don’t work on an older GPU, raise the bitrate to keep the quality, or use H.264.

Encoding two or more streams at once fails with out of memory (10)

  • Symptom: A batch runs several NVENC processes at once, and the ones over the session limit fail with OpenEncodeSessionEx failed.
  • Cause: You hit NVIDIA’s limit on concurrent NVENC sessions. On GeForce cards it is currently 12 for the whole system, counting all GeForce cards in it together, so a second GeForce card does not raise it. Professional cards that NVIDIA lists as “Unrestricted” have no fixed number; the encoder’s capacity and memory set the limit.
  • Fix: Look up your GPU’s limit in NVIDIA’s support matrix and run no more encodes at once than that, or update the driver. Writing several outputs from one process can also be more stable.

CPU-decode → GPU-encode isn’t as fast as expected

  • Symptom: You switched to NVENC, but CPU usage stays high and the speed doesn’t go up.
  • Cause: The CPU still does the decoding. Decoding on the CPU and copying the frames from the CPU to the GPU slow everything down.
  • Fix: Put -hwaccel cuda -hwaccel_output_format cuda before -i. The GPU then decodes too, and the frames go to NVENC without leaving GPU memory. This alone can leave the decoder threads using the CPU. Encoding 120 seconds of Big Buck Bunny at 1080p and 30 fps with h264_nvenc (p4, CQ 23): CPU decoding ran at 438 fps with 48 s of CPU time; -hwaccel cuda -hwaccel_output_format cuda alone gave 435 fps and 47 s, almost no change; adding -threads 1 before -i as well gave 451 fps and 2.7 s (median of 3 runs each). On this PC the speed stayed about the same while CPU use dropped sharply.

FAQ

How does NVENC compare to libx264 in image quality?

At the same bitrate, NVENC looks slightly worse. The newest NVENC (Ada Lovelace and later) comes close to libx264 at -preset medium, and given its speed it is good enough in practice.

NVENC isn’t working

Check that the GPU is an NVIDIA card, the driver is up to date, and your FFmpeg was built with --enable-nvenc. ffmpeg -encoders | grep nvenc shows whether the build has NVENC.

Should I use CRF or CQ?

Use -cq with NVENC (-crf is for libx264). The value runs from 1 to 51 (0 means automatic). As with libx264’s CRF, lower means better quality. Values of 19–23 are common.

Should I use B-frames?

Yes. Turning on B-frames with -b_ref_mode middle -bf 3 compresses much better. HEVC B-frames need a Turing-generation GPU or newer, so they may not work on older cards.

Can I do streaming delivery with NVENC?

Yes, it works well for low-latency streaming. For the lowest latency, combine -tune ll or -tune ull (ultra-low latency) with -preset p1.