Correction to costing rect_tx
When built with var_tx and ext_tx, select_tx_size_fix_type is used
to compute the cost for using a particular tx_type.
The code indexes the array inter_tx_type_costs at the wrong location
resulting in a zero cost for signalling tx_type for rect_tx blocks.
Change-Id: Iba38be3a0d822109f778f0600b242dfb40359766
diff --git a/av1/encoder/rdopt.c b/av1/encoder/rdopt.c
index b776548..1c0bb9f 100644
--- a/av1/encoder/rdopt.c
+++ b/av1/encoder/rdopt.c
@@ -3309,9 +3309,9 @@
int ext_tx_set = get_ext_tx_set(mbmi->tx_size, bsize, is_inter);
if (is_inter) {
if (ext_tx_set > 0)
- *rate += cpi->inter_tx_type_costs[ext_tx_set]
- [txsize_sqr_up_map[mbmi->tx_size]]
- [mbmi->tx_type];
+ *rate +=
+ cpi->inter_tx_type_costs[ext_tx_set][txsize_sqr_map[mbmi->tx_size]]
+ [mbmi->tx_type];
} else {
if (ext_tx_set > 0 && ALLOW_INTRA_EXT_TX)
*rate += cpi->intra_tx_type_costs[ext_tx_set][mbmi->tx_size][mbmi->mode]
