tree 16a37f3001a2292713ab4ddc566d6e4a790fe12a
parent 6f2b56c602d31bd154d510cafc1bbbf8c1b4b42f
author Rachel Barker <rachelbarker@google.com> 1702305144 +0000
committer Wan-Teh Chang <wtc@google.com> 1705592520 -0800

Fix a range check in disflow

In compute_flow_error(), we split a floating point value into integer
and floating point parts as follows:

    const int v_int = (int)floor(v);
    const double v_frac = v - floor(v);

This gets passed down to get_cubic_kernel_dbl(x = v_frac), which
asserts:

    assert(0 <= x && x < 1);

However, there is an edge case where v_frac can be equal to 1, if v
is very slightly less than an integer. For example, in the linked
bug report, v = -5.5...e-17, which leads to v_int = -1 and v_frac = 1
due to floating point rounding.

The current code actually handles this case perfectly fine, so all
that needs to be done is to change the range of the assertion to

    assert(0 <= x && x <= 1);

in all places where this code appears.

Bug: aomedia:3520
Change-Id: If3f89f36b5bad360c1459a2573fe02794b3ab2a7
(cherry picked from commit 15bcf20ce83af7549714bc20133a17b5348f00bd)