Avoid using MRC_DCT when the mask produced is invalid
If the mask is invalid, do not allow the encoder to select MRC_DCT.
Currently the mask is invalid if it is all 1 or all 0, but these
criteria will likely expand in a future patch.
Change-Id: I77230ea8357bfdb2bf1e6338903d44bbf1db22d1
diff --git a/av1/encoder/bitstream.c b/av1/encoder/bitstream.c
index a7e1dfc..377d75f 100644
--- a/av1/encoder/bitstream.c
+++ b/av1/encoder/bitstream.c
@@ -1642,6 +1642,11 @@
!supertx_enabled &&
#endif // CONFIG_SUPERTX
!segfeature_active(&cm->seg, mbmi->segment_id, SEG_LVL_SKIP)) {
+#if CONFIG_MRC_TX
+ if (tx_type == MRC_DCT)
+ assert(mbmi->valid_mrc_mask && "Invalid MRC mask");
+#endif // CONFIG_MRC_TX
+
const int eset =
get_ext_tx_set(tx_size, bsize, is_inter, cm->reduced_tx_set_used);
// eset == 0 should correspond to a set with only DCT_DCT and there
