| commit | 63bd6dc96bc8733b5782e0fffa9944a7786ec6fa | [log] [tgz] |
|---|---|---|
| author | Yi Luo <luoyi@google.com> | Thu Nov 17 09:13:39 2016 -0800 |
| committer | Yi Luo <luoyi@google.com> | Mon Nov 21 17:18:27 2016 +0000 |
| tree | 738e90d97c8cdedfe94545f31b1c4e7b5b95a99a | |
| parent | 125e72939c5e00e1f10041f325c57016173da0ff [diff] |
Fix rectangle transform computation overflow
- Add 16-bit saturation in fdct_round_shift().
- Add extreme value tests and round trip error tests.
- Fix inv 4x8 txfm calculation accuracy.
- Fix 4x8, 8x4, 8x16, 16x8, 16x32, 32x16 extreme value tests.
- BDRate: lowres: -0.034
midres: -0.036
hdres: -0.013
BUG=webm:1340
Change-Id: I48365c1e50a03a7b1aa69b8856b732b483299fb5