Cost calculation for out of range probability In some situations, the cdf probability can be out of range [1, CDF_PROB_TOP - 1]. Then, corresponding cost calculation would be wrong. This CL clamps the probabilty to resolve the issue. Ran Borg tests on lowres, midres, and hdres, and average PSNR change was 0. Saw slight PSNR changes for following 4 files. bridge_far_cif.y4m -0.003 into_tree_480p.y4m -0.006 station2_480p25.y4m 0.004 tears_of_steel2_480p.y4m 0.001 STATS_CHANGED BUG=aomedia:2297 Change-Id: Ia725af7edda830c46f7cced4a8e3a1d49559cbbf
diff --git a/av1/encoder/cost.h b/av1/encoder/cost.h index af5b098..be0241a 100644 --- a/av1/encoder/cost.h +++ b/av1/encoder/cost.h
@@ -30,6 +30,10 @@ // Calculate the cost of a symbol with probability p15 / 2^15 static INLINE int av1_cost_symbol(aom_cdf_prob p15) { + // p15 can be out of range [1, CDF_PROB_TOP - 1]. Clamping it, so that the + // following cost calculation works correctly. Otherwise, if p15 = + // CDF_PROB_TOP, shift would be -1, and "p15 << shift" would be wrong. + p15 = (aom_cdf_prob)clamp(p15, 1, CDF_PROB_TOP - 1); assert(0 < p15 && p15 < CDF_PROB_TOP); const int shift = CDF_PROB_BITS - 1 - get_msb(p15); const int prob = get_prob(p15 << shift, CDF_PROB_TOP);