Reduce the NN output precision bits to 9
This patch modifies the precision bits from 11 to 9 in
av1_nn_output_prec_reduce(). The input features to the
NN module contains precision level difference across the
OS, thereby causing the mismatch. This patch currently
fixes the bitstream mismatch by reducing the precision
bits. However this may not solve the problem entirely.
Verified on below OS and compiler version,
Windows - VS2019
Ubuntu - gcc 7.4.0, gcc 7.5.0
Centos - gcc 7.4.0
BUG=aomedia:2928
STATS_CHANGED
Change-Id: I46284de1771500b492bbe6bc4020d6e6e1141701
diff --git a/av1/encoder/ml.c b/av1/encoder/ml.c
index 57228ec..69a0cb4 100644
--- a/av1/encoder/ml.c
+++ b/av1/encoder/ml.c
@@ -16,7 +16,7 @@
#include "av1/encoder/ml.h"
void av1_nn_output_prec_reduce(float *const output, int num_output) {
- const int prec_bits = 11;
+ const int prec_bits = 9;
const int prec = 1 << prec_bits;
const float inv_prec = (float)(1.0 / prec);
for (int i = 0; i < num_output; i++) {
