RDO reconstruction bugfix for Inter IST
- For signaling, secondary transform type is skipped for eob <= 3 -- and assumed to be disabled.
- However, during RDO, the reconstruction using inverse transform was still applying secondary transform on encoder side.
This led to reconstruction mismatch between encoder and decoder.
We fix this by skipping inverse secondary transform during RDO for eob <= 3, to match what the signaling does.
Fixes #496
diff --git a/av1/common/idct.c b/av1/common/idct.c
index 6b9a00a..a57555a 100644
--- a/av1/common/idct.c
+++ b/av1/common/idct.c
@@ -619,6 +619,10 @@
assert(eob <= av1_get_max_eob(tx_size));
+ TX_TYPE primary_tx_type = get_primary_tx_type(tx_type);
+ if (!block_signals_sec_tx_type(xd, tx_size, primary_tx_type, eob))
+ tx_type = primary_tx_type;
+
TxfmParam txfm_param;
init_txfm_param(xd, plane, tx_size, tx_type, eob, reduced_tx_set,
#if CONFIG_INTER_DDT
