Fix daala-dist, rd tx search
Previously, for block >=8x8, and tx < 8x8,
we skipped setting the early-exit flag in block_rd_txfm() because
distortion for sub8x8 tx block is from MSE but reference (best)
is from daala-dist.
However, not setting early-exit flag turned out to be the reason
for a regression in MSE probe mode of daala-dist because
it loses the chance to set rd_stats properly.
On the other hand, there is still a small regression, say 0.05% psnr bd-rate,
which seems to occur in the case that a tx block in a partition has chosen
the skipped rd_cost since it is smaller than non-skip rd_cost and
set the early-exit flag to 0 (so, not exit), but the daala-dist applied
to the whole partition cannot access the same info but can choose from
two kinds of rd_costs:
1) all tx blocks are skipped (even if a tx block has non-zero coeff) and 0 bits
2) sum of final distortion of all tx blocks (i.e. non-zero coeff decoded)
and bits to encode coeffs.
Change-Id: I2ec69972aa1f22d465293cb9e8d5e18ef2c6f7f3
diff --git a/av1/encoder/rdopt.c b/av1/encoder/rdopt.c
index e7f1683..fffd2d0 100644
--- a/av1/encoder/rdopt.c
+++ b/av1/encoder/rdopt.c
@@ -1757,6 +1757,8 @@
args->rd_stats.rdcost = rd;
args->this_rd = rd;
+
+ if (args->this_rd > args->best_rd) args->exit_early = 1;
}
#endif // CONFIG_DAALA_DIST
