commit b844ee1b4b904794942cc81f09e72726f0726b3d
author David Barker <david.barker@argondesign.com> Thu Jan 25 13:14:42 2018 +0000
committer Urvang Joshi <urvang@google.com> Thu Jan 25 19:34:29 2018 +0000
tree 5830fc03269091d4b56b5887b0c97058949f7213
parent 8152737f4bad111c4fce6254bea5f3f45bee3924

[intra, bugfix] Prevent overflow in DC_PRED

Commit https://aomedia-review.googlesource.com/c/aom/+/40541 replaced
a division in the DC intra predictor by an approximate
multiply+shift sequence.

Unfortunately, this approximation is able to produce out-of-range
values. For example, consider 4x8 DC_PRED, with bit depth = 10.
If all of the context pixels are 0x3FF (the max value), then we get:

sum = 12 * 0x3FF
expected_dc = (sum * 0xAB) >> 11 = 1024 = 0x400

This means that we need to insert a clip_pixel(_highbd) operation
at the end of the DC prediction, to bring this value back in range.

BUG=aomedia:1272

Change-Id: I9beb9ac8a4b39803865f7e23932402ecd1d6f672

aom_dsp/intrapred.c

diff --git a/aom_dsp/intrapred.c b/aom_dsp/intrapred.c
index 9cf1113..b38ec65 100644
--- a/aom_dsp/intrapred.c
+++ b/aom_dsp/intrapred.c
@@ -385,6 +385,7 @@
   }
 
   expected_dc = (int)(((uint64_t)sum * multiplier) >> shift);
+  expected_dc = clip_pixel(expected_dc);
 
   for (r = 0; r < bh; r++) {
     memset(dst, expected_dc, bw);
@@ -1020,7 +1021,6 @@
                                             const uint16_t *left, int bd,
                                             uint32_t multiplier, int shift) {
   int i, r, expected_dc, sum = 0;
-  (void)bd;
 
   for (i = 0; i < bw; i++) {
     sum += above[i];
@@ -1030,6 +1030,7 @@
   }
 
   expected_dc = (int)(((uint64_t)sum * multiplier) >> shift);
+  expected_dc = clip_pixel_highbd(expected_dc, bd);
 
   for (r = 0; r < bh; r++) {
     aom_memset16(dst, expected_dc, bw);