Don't trash memory in select_tx_type_yrd
This patch fixes a bug in select_tx_type_yrd. The function works by
looping over possible transform types to find the best option (calling
select_tx_size_fix_type for each). Whenever there's a new best
candidate, the code copies information about the transform from the
mbmi structure into stack-allocated "best candidate" structures. At
the end, it copies the "best candidate" data back to mbmi.
Before the patch, if ref_best_rd was small, each call to
select_tx_size_fix_type might return INT64_MAX (because they don't
find anything better than ref_best_rd) and so we'd never actually copy
anything to the "best candidate" structures. Then, at the end of the
function, we'd merrily overwrite mbmi with whatever happened to be on
the stack, causing general mayhem when something tried to read the
data from mbmi later.
This patch exits early if no candidates were found. It also adds an
assertion saying that if no candidates were found, ref_best_rd must
have been less than INT64_MAX. This should hopefully catch any bugs
where the continue keywords in the loop stop us ever actually calling
select_tx_size_fix_type.
Change-Id: I54b998148281dd80f98d1570f736964593dc753f
diff --git a/av1/encoder/rdopt.c b/av1/encoder/rdopt.c
index 63d34dc..11134b0 100644
--- a/av1/encoder/rdopt.c
+++ b/av1/encoder/rdopt.c
@@ -5321,6 +5321,8 @@
prune = prune_tx_types(cpi, bsize, x, xd, 0);
#endif // CONFIG_EXT_TX
+ int found = 0;
+
for (tx_type = txk_start; tx_type < txk_end; ++tx_type) {
RD_STATS this_rd_stats;
av1_init_rd_stats(&this_rd_stats);
@@ -5365,12 +5367,21 @@
best_tx = mbmi->tx_size;
best_min_tx_size = mbmi->min_tx_size;
memcpy(best_blk_skip, x->blk_skip[0], sizeof(best_blk_skip[0]) * n4);
+ found = 1;
for (idy = 0; idy < xd->n8_h; ++idy)
for (idx = 0; idx < xd->n8_w; ++idx)
best_tx_size[idy][idx] = mbmi->inter_tx_size[idy][idx];
}
}
+ // We should always find at least one candidate unless ref_best_rd is less
+ // than INT64_MAX (in which case, all the calls to select_tx_size_fix_type
+ // might have failed to find something better)
+ assert(IMPLIES(!found, ref_best_rd != INT64_MAX));
+ if (!found) return;
+
+ // We found a candidate transform to use. Copy our results from the "best"
+ // array into mbmi.
mbmi->tx_type = best_tx_type;
for (idy = 0; idy < xd->n8_h; ++idy)
for (idx = 0; idx < xd->n8_w; ++idx)
