Miscellaneous fixes for var-tx
Lots of small bug fixes, mainly around the transform size coding:
* The loop filter was accidentally using the non-subsampled
block size for the V plane, due to comparing a plane index
(0, 1, or 2) against PLANE_TYPE_UV (== 1)
* We allowed an initial update of the transform partition probabilities
even on frames where we know they will never be used
(because tx_mode != TX_MODE_SELECT).
Further, these probabilities would not be reverted at the end
of the frame, leading to the probability delta persisting across frames.
Change this to behave more like the non-var-tx transform size coding,
where probability deltas are only coded for frames with
tx_mode == TX_MODE_SELECT, and the deltas only apply for one frame.
* Fix decoder for the case where the video as a whole isn't lossless,
and we have tx_mode == TX_MODE_SELECT, but the current segment
*is* lossless.
Note that the encoder already does the right thing in this case.
* Don't allow the transform splitting to recurse "below" 4x4.
This is really just a refactor, but means we can increase the
maximum depth when subdividing rectangular transforms if we
want to, whereas the previous code would have needed special cases
for 4x8 and 8x4 transforms.
* Finally, when we hit the maximum splitting depth, don't update
the counts as if we had coded a 'no split' symbol.
Change-Id: Iaebdacc9de81d2e93d3c49241e719bbc02e32682
diff --git a/av1/encoder/encodeframe.c b/av1/encoder/encodeframe.c
index 062b3b3..db20aa3 100644
--- a/av1/encoder/encodeframe.c
+++ b/av1/encoder/encodeframe.c
@@ -5806,6 +5806,18 @@
const TX_SIZE plane_tx_size = mbmi->inter_tx_size[tx_row][tx_col];
if (blk_row >= max_blocks_high || blk_col >= max_blocks_wide) return;
+ assert(tx_size > TX_4X4);
+
+ if (depth == MAX_VARTX_DEPTH) {
+// Don't add to counts in this case
+#if CONFIG_RECT_TX_EXT
+ if (tx_size == plane_tx_size)
+#endif
+ mbmi->tx_size = tx_size;
+ txfm_partition_update(xd->above_txfm_context + blk_col,
+ xd->left_txfm_context + blk_row, tx_size, tx_size);
+ return;
+ }
#if CONFIG_RECT_TX_EXT
if (tx_size == plane_tx_size ||
@@ -5829,7 +5841,7 @@
++counts->txfm_partition[ctx][1];
++x->txb_split_count;
- if (tx_size == TX_8X8) {
+ if (sub_txs == TX_4X4) {
mbmi->inter_tx_size[tx_row][tx_col] = TX_4X4;
mbmi->tx_size = TX_4X4;
txfm_partition_update(xd->above_txfm_context + blk_col,
