tree f774b1d4c20d0e199a733a53648af1925459545a
parent 04d6253b0c1689f49563a6da9ce047bd3e1b584f
author Wan-Teh Chang <wtc@google.com> 1712174896 +0000
committer Wan-Teh Chang <wtc@google.com> 1712183383 +0000

Fix integer overflows in calc of stride_in_bytes

Fix unsigned integer overflows in the calculation of stride_in_bytes in
img_alloc_helper() when d_w is huge.

Change the type of stride_in_bytes from unsigned int to int because it
will be assigned to img->stride[AOM_PLANE_Y], which is of the int type.

Test:
cmake ../aom -G Ninja -DCMAKE_C_COMPILER=clang \
  -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_BUILD_TYPE=Debug \
  -DSANITIZE=unsigned-integer-overflow
ninja
./test_libaom --gtest_filter=AomImageTest.AomImgAllocHugeWidth

Bug: chromium:332382766
Change-Id: Iaccb83bcd13ddc3ea5e6f01da91bb01215ddb461