Fix MV clamping in av1_int_pro_motion_estimation
Before this commit, av1_int_pro_motion_estimation does a full-pixel
motion search, but then converts its result to a normal MV and clamps
against the subpel MV search limits. This can lead to issues if clamping
occurs, because the result may end up not being pixel-aligned any more.
In particular, search_new_mv takes the result of
av1_int_pro_motion_estimation and converts it back to a full-pixel MV.
If the MV was clamped and now has non-pixel-aligned components, then
the resulting fullpel MV can be pushed outside of the motion search limits,
leading to an assertion failure a few lines later.
The correct solution is for av1_int_pro_motion_estimation to clamp its
result against the fullpel motion search limits. This ensures that the
output motion vector will always be pixel-aligned. Further, the fullpel
motion search limits are computed with the correct rounding to ensure
that they properly lie within the subpel search limits.
STATS_CHANGED
Bug: aomedia:3418
Change-Id: Icb1c87a95b287c6f217b38de6dd7d0d74d657839
(cherry picked from commit 61fa302827b1673de5c3a63cb391682c7abb414b)
diff --git a/av1/encoder/mcomp.c b/av1/encoder/mcomp.c
index b455d23..8fd1ab1 100644
--- a/av1/encoder/mcomp.c
+++ b/av1/encoder/mcomp.c
@@ -2094,11 +2094,11 @@
best_sad = tmp_sad;
}
- convert_fullmv_to_mv(best_int_mv);
+ FullMvLimits mv_limits = x->mv_limits;
+ av1_set_mv_search_range(&mv_limits, ref_mv);
+ clamp_fullmv(&best_int_mv->as_fullmv, &mv_limits);
- SubpelMvLimits subpel_mv_limits;
- av1_set_subpel_mv_search_range(&subpel_mv_limits, &x->mv_limits, ref_mv);
- clamp_mv(&best_int_mv->as_mv, &subpel_mv_limits);
+ convert_fullmv_to_mv(best_int_mv);
if (scaled_ref_frame) {
int i;
