Reduce the NN output precision bits to 9 This patch modifies the precision bits from 11 to 9 in av1_nn_output_prec_reduce(). The input features to the NN module contains precision level difference across the OS, thereby causing the mismatch. This patch currently fixes the bitstream mismatch by reducing the precision bits. However this may not solve the problem entirely. Verified on below OS and compiler version, Windows - VS2019 Ubuntu - gcc 7.4.0, gcc 7.5.0 Centos - gcc 7.4.0 BUG=aomedia:2928 STATS_CHANGED Change-Id: I46284de1771500b492bbe6bc4020d6e6e1141701
diff --git a/av1/encoder/ml.c b/av1/encoder/ml.c index 57228ec..69a0cb4 100644 --- a/av1/encoder/ml.c +++ b/av1/encoder/ml.c
@@ -16,7 +16,7 @@ #include "av1/encoder/ml.h" void av1_nn_output_prec_reduce(float *const output, int num_output) { - const int prec_bits = 11; + const int prec_bits = 9; const int prec = 1 << prec_bits; const float inv_prec = (float)(1.0 / prec); for (int i = 0; i < num_output; i++) {