Fix fixed-pt PVQ compand/expand outputs zero gain
For fixed-point version of PVQ, which is current default,
added MAXI(1, ) to limit the minimum companded or expanded gain to be one.
Previously, gain compand/expand function, which is invoked when
activity masking is enabled, sometimes outputs zero
then triggered the assert(gain != 0).
Metric change from floating-pt to fixed-pt PVQ is:
PSNR PSNR-HVS SSIM CIEDE-2000 PSNR Cb PSNR Cr MS-SSIM VMAF
0.02 0.10 0.08 0.11 0.01 0.02 0.13 -0.30
Change-Id: I64a60d1970d35a26af227841e4a5e50a89ddc44c
diff --git a/av1/common/pvq.c b/av1/common/pvq.c
index 327d8ed..b3b616d 100644
--- a/av1/common/pvq.c
+++ b/av1/common/pvq.c
@@ -698,7 +698,8 @@
tmp = cg0*q0*(int64_t)irt;
/*Expanded gain must be in Q(OD_COMPAND_SHIFT), thus OD_COMPAND_SHIFT is
not included here.*/
- return OD_VSHR_ROUND(tmp, OD_CGAIN_SHIFT + sqrt_outshift + sqrt_inshift);
+ return OD_MAXI(1,
+ OD_VSHR_ROUND(tmp, OD_CGAIN_SHIFT + sqrt_outshift + sqrt_inshift));
#endif
}
else {
@@ -715,7 +716,7 @@
expr = od_pow(cg, beta);
/*Expanded gain must be in Q(OD_COMPAND_SHIFT), hence the subtraction by
OD_COMPAND_SHIFT.*/
- return OD_SHR_ROUND(expr, OD_EXP2_OUTSHIFT - OD_COMPAND_SHIFT);
+ return OD_MAXI(1, OD_SHR_ROUND(expr, OD_EXP2_OUTSHIFT - OD_COMPAND_SHIFT));
#endif
}
}
