Cost calculation for out of range probability
In some situations, the cdf probability can be out of range
[1, CDF_PROB_TOP - 1]. Then, corresponding cost calculation
would be wrong. This CL clamps the probabilty to resolve the
issue.
Ran Borg tests on lowres, midres, and hdres, and average PSNR
change was 0. Saw slight PSNR changes for following 4 files.
bridge_far_cif.y4m -0.003
into_tree_480p.y4m -0.006
station2_480p25.y4m 0.004
tears_of_steel2_480p.y4m 0.001
STATS_CHANGED
BUG=aomedia:2297
Change-Id: Ia725af7edda830c46f7cced4a8e3a1d49559cbbf
diff --git a/av1/encoder/cost.h b/av1/encoder/cost.h
index af5b098..be0241a 100644
--- a/av1/encoder/cost.h
+++ b/av1/encoder/cost.h
@@ -30,6 +30,10 @@
// Calculate the cost of a symbol with probability p15 / 2^15
static INLINE int av1_cost_symbol(aom_cdf_prob p15) {
+ // p15 can be out of range [1, CDF_PROB_TOP - 1]. Clamping it, so that the
+ // following cost calculation works correctly. Otherwise, if p15 =
+ // CDF_PROB_TOP, shift would be -1, and "p15 << shift" would be wrong.
+ p15 = (aom_cdf_prob)clamp(p15, 1, CDF_PROB_TOP - 1);
assert(0 < p15 && p15 < CDF_PROB_TOP);
const int shift = CDF_PROB_BITS - 1 - get_msb(p15);
const int prob = get_prob(p15 << shift, CDF_PROB_TOP);
