[NORMATIVE-DECODING, txmg]: Adjust IADST4
Coefficients used for the VP9 version of IADST4 satisfy the
mathematical relation sinpi[1]+sinpi[2]==sinpi[4] which is
a trigonometric property useful for optimizing the four point
transform. Unfortunately the change in shift used in the latest
code means that rounding errors are introduced so that this identity
no longer holds. We think the identity should be restored by
rounding 4964.563 to 4964 rather than 4965. Similarly the identity
has been fixed for other bit shifts in the table where it does not
hold (even though these are not currently used I think).
This change also fixes a bug with the range checking for IADST4.
We see little change on the statistics from a local ACWY run
on objective-1-fast I-frame only:
PSNR PSNR-HVS SSIM PSNR-Cr APSNR APSNR-Cr MS-SSIM PSRNR-YUV
Average +0.01 +0.01 -0.02 -0.15 +0.01 -0.15 +0.01 +0.01
BUG=aomedia:1360
Change-Id: Icc09a1c59929e58bbc922d4b8b73de4a14104a8e
diff --git a/av1/common/av1_inv_txfm1d.c b/av1/common/av1_inv_txfm1d.c
index b61c4c2..46cfc73 100644
--- a/av1/common/av1_inv_txfm1d.c
+++ b/av1/common/av1_inv_txfm1d.c
@@ -766,14 +766,16 @@
return;
}
+ assert(sinpi[1] + sinpi[2] == sinpi[4]);
+
// stage 1
- s0 = range_check_value(sinpi[1] * x0, stage_range[1]);
- s1 = range_check_value(sinpi[2] * x0, stage_range[1]);
- s2 = range_check_value(sinpi[3] * x1, stage_range[1]);
- s3 = range_check_value(sinpi[4] * x2, stage_range[1]);
- s4 = range_check_value(sinpi[1] * x2, stage_range[1]);
- s5 = range_check_value(sinpi[2] * x3, stage_range[1]);
- s6 = range_check_value(sinpi[4] * x3, stage_range[1]);
+ s0 = range_check_value(sinpi[1] * x0, stage_range[1] + bit);
+ s1 = range_check_value(sinpi[2] * x0, stage_range[1] + bit);
+ s2 = range_check_value(sinpi[3] * x1, stage_range[1] + bit);
+ s3 = range_check_value(sinpi[4] * x2, stage_range[1] + bit);
+ s4 = range_check_value(sinpi[1] * x2, stage_range[1] + bit);
+ s5 = range_check_value(sinpi[2] * x3, stage_range[1] + bit);
+ s6 = range_check_value(sinpi[4] * x3, stage_range[1] + bit);
s7 = clamp_value(x0 - x2, stage_range[1]);
// stage 2
diff --git a/av1/common/av1_txfm.h b/av1/common/av1_txfm.h
index 62cc435..0f9dff41 100644
--- a/av1/common/av1_txfm.h
+++ b/av1/common/av1_txfm.h
@@ -82,13 +82,13 @@
14359, 12785, 11204, 9616, 8022, 6424, 4821, 3216, 1608 }
};
-// sinpi_arr_data[i][j] = (int)round((sqrt(2) * sin(kPi/9) * 2 / 3) * (1 <<
-// (cos_bit_min + i)))
+// sinpi_arr_data[i][j] = (int)round((sqrt(2) * sin(j*Pi/9) * 2 / 3) * (1 <<
+// (cos_bit_min + i))) modified so that elements j=1,2 sum to element j=4.
static const int32_t sinpi_arr_data[7][5] = {
- { 0, 330, 621, 836, 951 }, { 0, 660, 1241, 1672, 1902 },
- { 0, 1321, 2482, 3344, 3803 }, { 0, 2642, 4965, 6689, 7606 },
- { 0, 5283, 9929, 13377, 15212 }, { 0, 10566, 19858, 26755, 30425 },
- { 0, 21133, 39717, 53510, 60849 }
+ { 0, 330, 621, 836, 951 }, { 0, 660, 1241, 1672, 1901 },
+ { 0, 1321, 2482, 3344, 3803 }, { 0, 2642, 4964, 6689, 7606 },
+ { 0, 5283, 9929, 13377, 15212 }, { 0, 10566, 19858, 26755, 30424 },
+ { 0, 21133, 39716, 53510, 60849 }
};
static INLINE const int32_t *cospi_arr(int n) {