Correct scaling of satd in tx search for hbd encoding
This CL scales the satd as per bit-depth in skip_trellis_opt_based_on_satd().
10-bit encoding shows quality improvement with some speed loss.
Instruction Count BD-Rate Impact(%)
cpu-used Reduction(%) avg.psnr ovr.psnr ssim
4 -1.345 -0.2467 -0.2549 -0.1986
5 -0.854 -0.2377 -0.2477 -0.1982
6 -0.534 -0.1708 -0.1741 -0.1438
STATS_CHANGED for hbd encoding.
Change-Id: I76ce5f95a634e3ceea861f82af334eda08f34f7d
diff --git a/av1/encoder/tx_search.c b/av1/encoder/tx_search.c
index 37da269..09b4b63 100644
--- a/av1/encoder/tx_search.c
+++ b/av1/encoder/tx_search.c
@@ -2147,6 +2147,7 @@
const int shift = (MAX_TX_SCALE - av1_get_tx_scale(tx_size));
int satd = (dc_only_blk) ? abs(coeff_ptr[0]) : aom_satd(coeff_ptr, n_coeffs);
satd = RIGHT_SIGNED_SHIFT(satd, shift);
+ satd >>= (x->e_mbd.bd - 8);
const int skip_block_trellis =
((uint64_t)satd >
diff --git a/test/horz_superres_test.cc b/test/horz_superres_test.cc
index 17a0ae6..31933b6 100644
--- a/test/horz_superres_test.cc
+++ b/test/horz_superres_test.cc
@@ -54,7 +54,7 @@
{ "park_joy_90p_8_420.y4m", AOM_IMG_FMT_I420, AOM_BITS_8, 0, 5, 0, 12.5 },
#if CONFIG_AV1_HIGHBITDEPTH
{ "park_joy_90p_10_444.y4m", AOM_IMG_FMT_I44416, AOM_BITS_10, 1, 5, 0,
- 27.74 },
+ 27.60 },
#endif
{ "screendata.y4m", AOM_IMG_FMT_I420, AOM_BITS_8, 0, 4, 1, 20.0 },
// Image coding (single frame).
