Fix MV clamping in av1_int_pro_motion_estimation Before this commit, av1_int_pro_motion_estimation does a full-pixel motion search, but then converts its result to a normal MV and clamps against the subpel MV search limits. This can lead to issues if clamping occurs, because the result may end up not being pixel-aligned any more. In particular, search_new_mv takes the result of av1_int_pro_motion_estimation and converts it back to a full-pixel MV. If the MV was clamped and now has non-pixel-aligned components, then the resulting fullpel MV can be pushed outside of the motion search limits, leading to an assertion failure a few lines later. The correct solution is for av1_int_pro_motion_estimation to clamp its result against the fullpel motion search limits. This ensures that the output motion vector will always be pixel-aligned. Further, the fullpel motion search limits are computed with the correct rounding to ensure that they properly lie within the subpel search limits. STATS_CHANGED Bug: aomedia:3418 Change-Id: Icb1c87a95b287c6f217b38de6dd7d0d74d657839 (cherry picked from commit 61fa302827b1673de5c3a63cb391682c7abb414b)
diff --git a/av1/encoder/mcomp.c b/av1/encoder/mcomp.c index b455d23..8fd1ab1 100644 --- a/av1/encoder/mcomp.c +++ b/av1/encoder/mcomp.c
@@ -2094,11 +2094,11 @@ best_sad = tmp_sad; } - convert_fullmv_to_mv(best_int_mv); + FullMvLimits mv_limits = x->mv_limits; + av1_set_mv_search_range(&mv_limits, ref_mv); + clamp_fullmv(&best_int_mv->as_fullmv, &mv_limits); - SubpelMvLimits subpel_mv_limits; - av1_set_subpel_mv_search_range(&subpel_mv_limits, &x->mv_limits, ref_mv); - clamp_mv(&best_int_mv->as_mv, &subpel_mv_limits); + convert_fullmv_to_mv(best_int_mv); if (scaled_ref_frame) { int i;