CDEF: Do not filter chroma if subsampling_x != subsampling_y
Since CDEF looks uses the luma direction for chroma, CDEF would have
to change significantly to support formats like 4:2:2. The limited
use of such formats does not justify the complexity to support this,
so the simple solution is to mandate that the chroma planes aren't
filtered if subsampling_x != subsampling_y. Most of the visual gain
is in luma, anyway.
This also means that the chroma strengths and chroma skip condition
shall not be sent if subsampling_x != subsampling_y.
BUG=aomedia:720
Change-Id: I35c184a6fe0908ae0fee1e74494b6904fa9a3c82
diff --git a/av1/decoder/decodeframe.c b/av1/decoder/decodeframe.c
index dd18053..026866f 100644
--- a/av1/decoder/decodeframe.c
+++ b/av1/decoder/decodeframe.c
@@ -2857,7 +2857,9 @@
cm->nb_cdef_strengths = 1 << cm->cdef_bits;
for (i = 0; i < cm->nb_cdef_strengths; i++) {
cm->cdef_strengths[i] = aom_rb_read_literal(rb, CDEF_STRENGTH_BITS);
- cm->cdef_uv_strengths[i] = aom_rb_read_literal(rb, CDEF_STRENGTH_BITS);
+ cm->cdef_uv_strengths[i] = cm->subsampling_x == cm->subsampling_y
+ ? aom_rb_read_literal(rb, CDEF_STRENGTH_BITS)
+ : 0;
}
}
#endif // CONFIG_CDEF
