Define block_signals_txsize function
This returns true if a block signals tx_size in the stream and uses it
in the bitstream writing code and the decoder.
Note that we can't quite use it in pack_inter_mode_mvs when
CONFIG_VAR_TX && !CONFIG_RECT_TX but I've switched the code to using
it the rest of the time since rect-tx is adopted and eventually the
other code path should be deleted.
Also use the helper function in tx_size_cost in rdopt.c, where the
test was wrong and caused underestimates of block
costs. (Specifically, the code that subtracts tx_size_cost from
this_rate_tokenonly in rd_pick_intra_sby_mode ended up subtracting
zero for a 4x8 block).
The behaviour of the decoder should be unchanged. The only change in
the encoder's behaviour should be in tx_size_cost where it should now
match the rest of the code.
Change-Id: I97236c9ce444993afe01ac5c6f4a0bb9e5049217
diff --git a/av1/decoder/decodemv.c b/av1/decoder/decodemv.c
index 6005be5..351524c 100644
--- a/av1/decoder/decodemv.c
+++ b/av1/decoder/decodemv.c
@@ -520,12 +520,8 @@
const TX_MODE tx_mode = cm->tx_mode;
const BLOCK_SIZE bsize = xd->mi[0]->mbmi.sb_type;
if (xd->lossless[xd->mi[0]->mbmi.segment_id]) return TX_4X4;
-#if CONFIG_CB4X4 && (CONFIG_VAR_TX || CONFIG_EXT_TX) && CONFIG_RECT_TX
- if (bsize > BLOCK_4X4)
-#else
- if (bsize >= BLOCK_8X8)
-#endif // CONFIG_CB4X4 && CONFIG_VAR_TX
- {
+
+ if (block_signals_txsize(bsize)) {
if ((!is_inter || allow_select_inter) && tx_mode == TX_MODE_SELECT) {
const int32_t tx_size_cat = is_inter ? inter_tx_size_cat_lookup[bsize]
: intra_tx_size_cat_lookup[bsize];
