tree 15e0994d445e344479f6c7bfa6b71363a31da07e
parent 90f39bb849cfb6c3864512d05b2b430a256c82b4
author Rachel Barker <rachelbarker@google.com> 1680207883 +0000
committer Wan-Teh Chang <wtc@google.com> 1683332723 +0000

Fix MV clamping in av1_int_pro_motion_estimation

Before this commit, av1_int_pro_motion_estimation does a full-pixel
motion search, but then converts its result to a normal MV and clamps
against the subpel MV search limits. This can lead to issues if clamping
occurs, because the result may end up not being pixel-aligned any more.

In particular, search_new_mv takes the result of
av1_int_pro_motion_estimation and converts it back to a full-pixel MV.
If the MV was clamped and now has non-pixel-aligned components, then
the resulting fullpel MV can be pushed outside of the motion search limits,
leading to an assertion failure a few lines later.

The correct solution is for av1_int_pro_motion_estimation to clamp its
result against the fullpel motion search limits. This ensures that the
output motion vector will always be pixel-aligned. Further, the fullpel
motion search limits are computed with the correct rounding to ensure
that they properly lie within the subpel search limits.

STATS_CHANGED

Bug: aomedia:3418
Change-Id: Icb1c87a95b287c6f217b38de6dd7d0d74d657839
(cherry picked from commit 61fa302827b1673de5c3a63cb391682c7abb414b)