Fix bug in global motion model refinement
When refining a global motion model, we change one model parameter at a
time and re-calculate the prediction. For TRANSLATION and AFFINE type
models, we can just change one matrix element at a time, but for ROTZOOM
models we need to change two at a time because the constraints that
wmmat[4] = -wmmat[3] and wmmat[5] = wmmat[2].
This constraint was being applied in warp_plane()/highbd_warp_plane(),
but unfortunately the shear parameters (alpha, beta, gamma, delta),
which are used internally by the warp filter, were being calculated
before the constraint was applied.
This meant that, for ROTZOOM type models, the predictions used during
global motion refinement search were just slightly wrong, leading to
suboptimal decisions.
This patch fixes the issue by applying the ROTZOOM constraints in
av1_warp_error(), just before the shear parameters are calculated.
Causes noise-level changes to both BDRATE and encode time
STATS_CHANGED
Change-Id: Ib1b2b9cea50b0df83d4ee111786c5cb6e4cc6ebb
diff --git a/av1/encoder/global_motion.c b/av1/encoder/global_motion.c
index 9e84e53..9c1c0d2 100644
--- a/av1/encoder/global_motion.c
+++ b/av1/encoder/global_motion.c
@@ -206,6 +206,7 @@
int p_height, int p_stride, int subsampling_x,
int subsampling_y, int64_t best_error,
uint8_t *segment_map, int segment_map_stride) {
+ force_wmtype(wm, wm->wmtype);
if (wm->wmtype <= AFFINE)
if (!av1_get_shear_params(wm)) return INT64_MAX;
#if CONFIG_AV1_HIGHBITDEPTH
