commit 48f84dbd1c57ac475d4d7c13cc6b7cbd25290455
author Yushin Cho <ycho@mozilla.com> Mon Nov 07 21:20:17 2016 -0800
committer Yaowu Xu <yaowu@google.com> Tue Nov 08 08:15:57 2016 -0800
tree f1c357a43ae5e8c2baeaf7e20aa9000b3ff94cc0
parent f0481a590f3c90390fed8048a469d5fad1addc73

Fix the bug that PVQ commit broke dering

Since PVQ's max block size equals to the max transform size,
daala's definition of OD_BSIZE_MAX was changed from 5 down to 4 to
use AV1's max trasform size 32x32. However, dering also uses
OD_BSIZE_MAX and assumes its value is 5, which caused dering
not working.

Change-Id: I9d82bb24adc7d57552a8e0a8a7e798e77d96fd4b

av1/encoder/bitstream.c

diff --git a/av1/encoder/bitstream.c b/av1/encoder/bitstream.c
index d94f863..43452a8 100644
--- a/av1/encoder/bitstream.c
+++ b/av1/encoder/bitstream.c
@@ -2004,7 +2004,7 @@
                     &w->ec, pvq->qg[i], pvq->theta[i], pvq->max_theta[i],
                     pvq->y + pvq->off[i], pvq->size[i], pvq->k[i], model, adapt,
                     exg + i, ext + i, robust || is_keyframe,
-                    (plane != 0) * OD_NBSIZES * PVQ_MAX_PARTITIONS +
+                    (plane != 0) * OD_TXSIZES * PVQ_MAX_PARTITIONS +
                         pvq->bs * PVQ_MAX_PARTITIONS + i,
                     is_keyframe, i == 0 && (i < pvq->nb_bands - 1),
                     pvq->skip_rest, encode_flip, flip);