Fix an overflow issue related to the split partition cost
In partition search, the partition costs for blocks at the frame edges are
overwritten. As we have seen, icdfs for {PARTITION_VERT, PARTITION_SPLIT}
can be {0, 0}, which results in partition_cost[PARTITION_SPLIT] = INT_MAX.
Later, while trying to add other cost to this cost, overflow occurs.
16:03:19 [----------] 1 test from AV1/ResizeTest
16:03:19 [ RUN ] AV1/ResizeTest.TestExternalResizeWorks/0
16:03:25 /var/jenkins/workspace/libaom__per_commit-compile_sanitizers/sanitizer/
integer/aom/av1/encoder/encodeframe.c:4270:22: runtime error: signed integer
overflow: 2147483647 + 6066 cannot be represented in type 'int'
16:03:25 #0 0x11769e9 in rd_pick_partition /var/jenkins/workspace/
libaom__per_commit-compile_sanitizers/sanitizer/integer/aom/av1/encoder/
encodeframe.c:4270:22
This CL fixed the issue.
Change-Id: I982420e6ce097dbf98f35469859d7574c3d87c3c
diff --git a/av1/encoder/encodeframe.c b/av1/encoder/encodeframe.c
index 9774678..30a8672 100644
--- a/av1/encoder/encodeframe.c
+++ b/av1/encoder/encodeframe.c
@@ -4023,6 +4023,7 @@
}
partition_cost = tmp_partition_cost;
+ do_square_split &= partition_cost[PARTITION_SPLIT] != INT_MAX;
}
#ifndef NDEBUG
