commit ab141115369e0d1792572c911b9ecc16f3479728
author Timothy B. Terriberry <tterriberry@mozilla.com> Sat Jun 10 12:25:52 2017 -0700
committer Timothy B. Terriberry <tterriberry@mozilla.com> Sat Jun 10 12:45:57 2017 -0700
tree cee2370e5d24d191fc7417985295fefb65477e1c
parent 1ce569d70eaa18887d311bc46021ec9decce79d1

var_tx: Fix distortion calc. in av1_tx_block_rd_b

This was hard-coding the assumption that the block size for the
smallest TX size was also the smallest block size. This is no
longer true since fe67ed6af26b landed.

As a result, for TX blocks that overlapped the frame edge, it was
only measuring distortion on the upper-left 2x2 part of each 4x4
sub-block, causing the encoder to prefer larger transforms which
cause such overlap and avoid transforms which do not, causing a
regression.

This patch uses the appropriate conversion table, which fixes the
regression.

BUG=aomedia:593

Change-Id: Id253cf0f3a5252378e3f340b8350120639ff5c88