tree 497829472e73ff018b3238c13b165cef84f83e1e
parent 867fb6a2b35543d76821f33d5d009691f19ee4b2
author Rachel Barker <rachelbarker@google.com> 1691437819 +0000
committer Rachel Barker <rachelbarker@google.com> 1693396905 +0000

Remove abs() from highbd error metric

By thinking carefully about rounding, we can work directly with
negative error values in highbd_error_measure(), avoiding the need
to call abs() to make the error value positive.

This will simplify the upcoming SSE implementations of
av1_highbd_frame_error(), allowing them to use only SSE2 functions
(note that _mm_abs_epi16() requires SSSE3).

To support this, we need to extend error_measure_lut to add an
element at logical index -256. This means that all accesses to
the lookup table must be offset by +256 instead of +255.

There is no change to the result of any function, so there is no
stats change.

Change-Id: Ic686750cac899853070c8f716dc86eb429e29917