Fix a range check in disflow
In compute_flow_error(), we split a floating point value into integer
and floating point parts as follows:
const int v_int = (int)floor(v);
const double v_frac = v - floor(v);
This gets passed down to get_cubic_kernel_dbl(x = v_frac), which
asserts:
assert(0 <= x && x < 1);
However, there is an edge case where v_frac can be equal to 1, if v
is very slightly less than an integer. For example, in the linked
bug report, v = -5.5...e-17, which leads to v_int = -1 and v_frac = 1
due to floating point rounding.
The current code actually handles this case perfectly fine, so all
that needs to be done is to change the range of the assertion to
assert(0 <= x && x <= 1);
in all places where this code appears.
Bug: aomedia:3520
Change-Id: If3f89f36b5bad360c1459a2573fe02794b3ab2a7
(cherry picked from commit 15bcf20ce83af7549714bc20133a17b5348f00bd)
diff --git a/aom_dsp/flow_estimation/disflow.c b/aom_dsp/flow_estimation/disflow.c
index a8e7b06..ad8d659 100644
--- a/aom_dsp/flow_estimation/disflow.c
+++ b/aom_dsp/flow_estimation/disflow.c
@@ -53,7 +53,14 @@
#define UPSAMPLE_CENTER_OFFSET ((DOWNSAMPLE_FACTOR - 1) / 2)
static INLINE void get_cubic_kernel_dbl(double x, double *kernel) {
- assert(0 <= x && x < 1);
+ // Check that the fractional position is in range.
+ //
+ // Note: x is calculated from (eg.) `u_frac = u - floor(u)`.
+ // Mathematically, this implies that 0 <= x < 1. However, in practice it is
+ // possible to have x == 1 due to floating point rounding. This is fine,
+ // and we still interpolate correctly if we allow x = 1.
+ assert(0 <= x && x <= 1);
+
double x2 = x * x;
double x3 = x2 * x;
kernel[0] = -0.5 * x + x2 - 0.5 * x3;
diff --git a/aom_dsp/flow_estimation/x86/disflow_sse4.c b/aom_dsp/flow_estimation/x86/disflow_sse4.c
index a62e9a4..37170fe 100644
--- a/aom_dsp/flow_estimation/x86/disflow_sse4.c
+++ b/aom_dsp/flow_estimation/x86/disflow_sse4.c
@@ -28,7 +28,14 @@
// Note: Max sum(+ve coefficients) = 1.125 * scale
static INLINE void get_cubic_kernel_dbl(double x, double *kernel) {
- assert(0 <= x && x < 1);
+ // Check that the fractional position is in range.
+ //
+ // Note: x is calculated from (eg.) `u_frac = u - floor(u)`.
+ // Mathematically, this implies that 0 <= x < 1. However, in practice it is
+ // possible to have x == 1 due to floating point rounding. This is fine,
+ // and we still interpolate correctly if we allow x = 1.
+ assert(0 <= x && x <= 1);
+
double x2 = x * x;
double x3 = x2 * x;
kernel[0] = -0.5 * x + x2 - 0.5 * x3;
