Resolve -Wshorten-64-to-32 in highbd variance.
For 8-bit the subtrahend is small enough to fit into uint32_t.
cherry-picked from libvpx:
47b9a0912 Resolve -Wshorten-64-to-32 in highbd variance.
This is the same that was done for:
c0241664a Resolve -Wshorten-64-to-32 in variance.
For 10/12-bit apply:
63a37d16f Prevent negative variance
BUG=aomedia:445
Change-Id: Iab35e3f3f269035e17c711bd6cc01272c3137e1d
diff --git a/aom_dsp/variance.c b/aom_dsp/variance.c
index c5fb495..55fac22 100644
--- a/aom_dsp/variance.c
+++ b/aom_dsp/variance.c
@@ -366,7 +366,7 @@
uint32_t *sse) { \
int sum; \
highbd_8_variance(a, a_stride, b, b_stride, W, H, sse, &sum); \
- return *sse - (((int64_t)sum * sum) / (W * H)); \
+ return *sse - (uint32_t)(((int64_t)sum * sum) / (W * H)); \
} \
\
uint32_t aom_highbd_10_variance##W##x##H##_c(const uint8_t *a, int a_stride, \
