tree 010a8845ba88a8d378fa16dc84d393defb6e0d7b
parent a727a1742abf33bf012ace88bca4e939c226eea8
author Yunqing Wang <yunqingwang@google.com> 1549396986 -0800
committer Yunqing Wang <yunqingwang@google.com> 1549588657 +0000

Fix an overflow issue related to the split partition cost

In partition search, the partition costs for blocks at the frame edges are
overwritten. As we have seen, icdfs for {PARTITION_VERT, PARTITION_SPLIT}
can be {0, 0}, which results in partition_cost[PARTITION_SPLIT] = INT_MAX.
Later, while trying to add other cost to this cost, overflow occurs.
16:03:19 [----------] 1 test from AV1/ResizeTest
16:03:19 [ RUN      ] AV1/ResizeTest.TestExternalResizeWorks/0
16:03:25 /var/jenkins/workspace/libaom__per_commit-compile_sanitizers/sanitizer/
integer/aom/av1/encoder/encodeframe.c:4270:22: runtime error: signed integer
overflow: 2147483647 + 6066 cannot be represented in type 'int'
16:03:25     #0 0x11769e9 in rd_pick_partition /var/jenkins/workspace/
libaom__per_commit-compile_sanitizers/sanitizer/integer/aom/av1/encoder/
encodeframe.c:4270:22
This CL fixed the issue.

Change-Id: I982420e6ce097dbf98f35469859d7574c3d87c3c