tree b7d6f927a2a8ecfea65602818d48d35cccd9ed5e
parent e0607ac99db5a4b0a66a9a97124821f3466f72a5
author Rachel Barker <rachelbarker@google.com> 1679937337 +0000
committer Urvang Joshi <urvang@google.com> 1679981752 +0000

Fix fullpel search limits

(Ported from libaom commit fa59a65)

Motion search is divided into a full-pixel search followed by
a sub-pixel refinement step. These two steps have separate sets
of limits for which motion vectors they will consider, set by
the functions:

fullpel: av1_set_mv_search_range()
subpel: av1_set_subpel_mv_search_range()

The subpel search limits are very simple - we allow
ref_mv +/- 1023 pixels, with some appropriate clamping.

For fullpel search, it appears that the intended logic was to allow
any full-pixel MV which lies within the subpel limits. Put another
way, the fullpel limits should be the same as the subpel limits, but
pushed inwards to full-pixel values.

The code in av1_set_mv_search_range almost does this, but the
rounding isn't quite right. This sometimes causes the limits to be
off by 1 pixel. That allows the fullpel search to select a motion
vector which is outside of the subpel limits. That MV can then be
carried through the subpixel search.

This does not truly break anything, since the range of MV diffs
that we can encode is larger than what we search. But it's definitely
not intended, and in rare cases could trip an assertion in
av1_refine_warped_mv, as seen in the linked bug report.

By fixing the rounding, we can remove this discrepancy and prevent
the assertion failure. It also allows one extra row/column of fullpel
motion vectors which are inside the subpel limits, but which were
not being searched.

Also add assertions in many places to check that this issue does
not reoccur.

STATS_CHANGED

Bug: aomedia:3384