tree c7d5c2584278053370e660d1a318664c5819bb23
parent e16b2665c29c4ba9e5da8b264135b6a40f2c1cb5
author David Barker <david.barker@argondesign.com> 1495626368 +0100
committer Urvang Joshi <urvang@google.com> 1495736947 +0000

Fix bug in av1_optimize_b when tx64x64 is enabled

In both versions of av1_optimize_b, the code to calculate the
dequantized coefficient from a modified quantized coefficient
was not compatible with tx64x64.

Specifically, the new dequantized coefficient was calculated as an
offset applied to the original coefficient. This had a fixup block
for 32x32 transforms, but not for 64x64 transforms.

For simplicity, instead of extending the fixup block, we now
recalculate the dequantized coefficient directly.

BUG=aomedia:537

Change-Id: I86c1c8820f324ad3f6afd6a720ff59f754384a19