Round up subsampled frame size in av1_loop_restoration_corners_in_sb The previous code converted a frame_w (say) of 1 to zero for a plane where subsampling was enabled, causing a division by zero in av1_get_rest_ntiles. This doesn't match the spec, which says subsampling rounds up. The patch adds the rounding, and also adds an assertion to av1_get_rest_ntiles to help diagnose any other broken callsites. Change-Id: Ia6c249fa935c3a16d122ba6e7b450fe99f412fde
diff --git a/av1/common/restoration.c b/av1/common/restoration.c index f97cf5b..f1411db 100644 --- a/av1/common/restoration.c +++ b/av1/common/restoration.c
@@ -1463,8 +1463,11 @@ const int denom = 1; #endif // CONFIG_FRAME_SUPERRES - const int ss_frame_w = frame_w >> (plane > 0 && cm->subsampling_x != 0); - const int ss_frame_h = frame_h >> (plane > 0 && cm->subsampling_y != 0); + const int ss_x = plane > 0 && cm->subsampling_x != 0; + const int ss_y = plane > 0 && cm->subsampling_y != 0; + + const int ss_frame_w = (frame_w + ss_x) >> ss_x; + const int ss_frame_h = (frame_h + ss_y) >> ss_y; int rtile_w, rtile_h, nvtiles; av1_get_rest_ntiles(ss_frame_w, ss_frame_h,
diff --git a/av1/common/restoration.h b/av1/common/restoration.h index b3f9b69..9eee959 100644 --- a/av1/common/restoration.h +++ b/av1/common/restoration.h
@@ -194,6 +194,8 @@ int tile_width_, tile_height_; tile_width_ = (tilesize < 0) ? width : AOMMIN(tilesize, width); tile_height_ = (tilesize < 0) ? height : AOMMIN(tilesize, height); + assert(tile_width_ > 0 && tile_height_ > 0); + nhtiles_ = (width + (tile_width_ >> 1)) / tile_width_; nvtiles_ = (height + (tile_height_ >> 1)) / tile_height_; if (tile_width) *tile_width = tile_width_;