How come the quality of HEVC with alpha from my ffmpeg with videotoolbox is not ok

Question

I'm encoding ProRes 4444 with alpha channel video's to transparent video for web in both VP9 (webm) and HEVC (mp4). I'm using a great free tool by Rotato to encode to both formats at once.

However, because I would like to have more control over e.g. the dimensions and the bitrate, I'm looking into ffmpeg. I can encode to both formats with the following params.

# vp9
ffmpeg -i in.mov -vf scale=-1:720 -c:v libvpx-vp9 -auto-alt-ref 0 -pix_fmt yuva420p -acodec libvorbis -cpu-used 3 -deadline good -crf 40 out.webm 

# hevc
ffmpeg -i in.mov -vf scale=-1:720 -c:v hevc_videotoolbox -allow_sw 1 -alpha_quality 0.9 -tag:v hvc1 -q:v 65 out.mp4

The problem: the videotoolbox encoder generates a weird white garbage around the black pencil animations and fade-in's from transparent to black pencil become fade-in's from white pencil to black pencil!?

Here are screenshots of the two compared (top one is my ffmpeg output):

Anyone have good settings for ffmpeg I cano use to encode good quality HEVC with alpha? Hardware: MacBook Air M1 and Win11 + GTX 1060 (I should be able to use NVEnc, but don't know how)

Answer 1

Before the options I would check the system preferences to retain the alpha channel transparency more from Larry here, they are build natively - see the two options larry has on his site.

Check to ensure in System Preferences > Keyboard. The Services category on the left, then select enable Encode Selected Video Files on the right. From any video application, export a project that contains transparency directly using the ProRes 4444 codec. Then, in the Finder, right-click the resulting .mov and select Services -> Encode Selected Video Files. Wait a few seconds for the next menu to appear.

---

Seems like you are looking for the encoder and bitrate options, you can list them out fully as below

Option 1: H.264 encoder

// Help you understand the ***H.264 encoder's options.
ffmpeg -hide_banner -h encoder=h264_nvenc | xclip -sel clip

[![enter image description here][1]][1]

Option 1: HEVC/H.265 encoder

// Help you understand the **H.265 encoder's options.
ffmpeg -hide_banner -h encoder=h264_nvenc | xclip -sel clip

Now for bitrates, setup values using -b:v, -maxrate:v and -bufsize:v options

And since you have Mac Hardware - if you type this it will tell your configuration.

 ~ ffmpeg -h encoder=hevc_videotoolbox

Once you have this you can choose to tweak it in the configuration, rebuild and yourself.. with different options look here for build on VM + Apple M1 and here for the encoding Support for Apple Silicon M1 encoders