commit 2615d6eaac3194b74ca61b95620f911047d7ad4b
author Yunqing Wang <yunqingwang@google.com> Fri Dec 16 21:23:50 2016 -0800
committer Yunqing Wang <yunqingwang@google.com> Wed Jan 18 19:13:35 2017 +0000
tree 2ef9dfc042d4a3090f3544f904026217e7e802a0
parent feb925fe847d76c03545dcaf9efe7acddfaca7ca

Fix the transform size/type search condition

While encoding a key frame with quantizer = 0 and aq-mode = 1,
for some segment_ids, the quantizer got modified and could be
> 0, and lossless[segment_id] might be 0 or 1 depending on the
segment_id. Namely, blocks with lossless[segment_id] = 0 were
allowed to choose transform sizes other than 4x4. This conflicted
with tx_mode which was a frame-level decision. In this patch,
the transform search condition was modified so that the transform
choice was consistent with tx_mode of that frame.

BUG=aomedia:104

Change-Id: Ia39127b5dee129283a133cf5e4000da62d9e0f1c

av1/encoder/rdopt.c

diff --git a/av1/encoder/rdopt.c b/av1/encoder/rdopt.c
index 5369e2b..d20b3ad 100644
--- a/av1/encoder/rdopt.c
+++ b/av1/encoder/rdopt.c
@@ -1849,7 +1849,7 @@
 
   assert(bs == xd->mi[0]->mbmi.sb_type);
 
-  if (xd->lossless[xd->mi[0]->mbmi.segment_id]) {
+  if (xd->lossless[0]) {
     choose_smallest_tx_size(cpi, x, rd_stats, ref_best_rd, bs);
   } else if (cpi->sf.tx_size_search_method == USE_LARGESTALL) {
     choose_largest_tx_size(cpi, x, rd_stats, ref_best_rd, bs);