The spec for nal_ref_idc doesn't indicate that it should be used for
priority like x264 seems to be using it for. NVENC seems to pass
different values, so let's not rely on it.
The new scheme might be naive, but we can iterate on it, and apply
evenly to all H.264 encoders.