Palette: Handle case of too few unique centroids.
Even though number of unique colors in a block may be at least
PALETTE_MIN_SIZE, sometimes the K-means can generate fewer than
PALETTE_MIN_SIZE unique centroids due to integer rounding.
This was resulting in "k - PALETTE_MIN_SIZE" being negative and using
that as an index into palette_y_size_cost array was causing a UBSan
error.
BUG=aomedia:515
Change-Id: Ia00b4bb2efebbe43d1fec7b619ebb837b1ae961c
diff --git a/av1/encoder/rdopt.c b/av1/encoder/rdopt.c
index 254d8f9..33512aa 100644
--- a/av1/encoder/rdopt.c
+++ b/av1/encoder/rdopt.c
@@ -2501,6 +2501,11 @@
centroids[i] = lb + (2 * i + 1) * (ub - lb) / n / 2;
av1_k_means(data, centroids, color_map, rows * cols, n, 1, max_itr);
k = av1_remove_duplicates(centroids, n);
+ if (k < PALETTE_MIN_SIZE) {
+ // Too few unique colors to create a palette. And DC_PRED will work well
+ // for that case anyway. So skip.
+ continue;
+ }
#if CONFIG_HIGHBITDEPTH
if (cpi->common.use_highbitdepth)
