Fix daala-dist, rd tx search Previously, for block >=8x8, and tx < 8x8, we skipped setting the early-exit flag in block_rd_txfm() because distortion for sub8x8 tx block is from MSE but reference (best) is from daala-dist. However, not setting early-exit flag turned out to be the reason for a regression in MSE probe mode of daala-dist because it loses the chance to set rd_stats properly. On the other hand, there is still a small regression, say 0.05% psnr bd-rate, which seems to occur in the case that a tx block in a partition has chosen the skipped rd_cost since it is smaller than non-skip rd_cost and set the early-exit flag to 0 (so, not exit), but the daala-dist applied to the whole partition cannot access the same info but can choose from two kinds of rd_costs: 1) all tx blocks are skipped (even if a tx block has non-zero coeff) and 0 bits 2) sum of final distortion of all tx blocks (i.e. non-zero coeff decoded) and bits to encode coeffs. Change-Id: I2ec69972aa1f22d465293cb9e8d5e18ef2c6f7f3
diff --git a/av1/encoder/rdopt.c b/av1/encoder/rdopt.c index e7f1683..fffd2d0 100644 --- a/av1/encoder/rdopt.c +++ b/av1/encoder/rdopt.c
@@ -1757,6 +1757,8 @@ args->rd_stats.rdcost = rd; args->this_rd = rd; + + if (args->this_rd > args->best_rd) args->exit_early = 1; } #endif // CONFIG_DAALA_DIST