blob: d88f5d2e222b0ffb761780879894db6715193e63 [file] [view] [edit]
---
name: libaom/clamp-floating-point-conversions-to-integer-limits
description: When casting floating-point expressions or 64-bit integers to narrower integer types, clamp values to target bounds using AOMMIN or saturate_cast_double_to_int to avoid overflow
rubric: correctness
severity: blocking
---
**Rule.** When casting floating-point calculations (such as values from ceil(),
floor(), round(), or rate-control formulas) or wider integer variables (such as
int64_t rate-distortion multipliers, frame targets, bit budgets, or
accumulators) to narrower integer types (like int or unsigned int), clamp values
to target integer limits using AOMMIN(..., INT_MAX) or
saturate_cast_double_to_int() before casting. Verify that scaled values or
multi-step accumulations cannot exceed INT_MAX (or [INT_MIN, INT_MAX]) and wrap
or trigger undefined behavior.
**Why.** In C and C++, converting a floating-point value that exceeds the target
integer type's representable range (greater than INT_MAX or less than INT_MIN)
results in undefined behavior. Similarly, casting out-of-range int64_t values
directly to signed 32-bit int causes signed integer overflow and undefined
behavior, leading to negative rate-distortion multipliers, corrupted bit
budgets, or runtime errors. Clamping or saturating prior to casting guarantees
well-defined, range-safe conversions.
## Examples
Bad:
```cpp
// Bad: Casting unbounded floating-point calculation directly to int
rc->max_consec_drop = (int)ceil(cpi->oxcf.rc_cfg.max_consec_drop_time *
cpi->framerate / 1000);
// Bad: Direct cast of unbounded wide 64-bit integer / rate calculation to int
int64_t rdmult = av1_compute_rd_mult_based_on_qindex(bit_depth, update_type,
qindex, tuning);
rdmult = (rdmult * rd_layer_depth_factor[layer_depth]) >> 7;
rdmult += ((rdmult * rd_boost_factor[boost_index]) >> 7);
return (int)rdmult;
```
Good:
```cpp
// Good: Clamping floating-point result or using saturation helper
rc->max_consec_drop = (int)AOMMIN(
ceil(cpi->oxcf.rc_cfg.max_consec_drop_ms * cpi->framerate / 1000),
INT_MAX);
rc->max_consec_drop = saturate_cast_double_to_int(
ceil(cpi->oxcf.rc_cfg.max_consec_drop_ms * cpi->framerate / 1000));
// Good: Clamping 64-bit integer to INT_MAX before casting to int
int64_t rdmult = av1_compute_rd_mult_based_on_qindex(bit_depth, update_type,
qindex, tuning);
rdmult = (rdmult * rd_layer_depth_factor[layer_depth]) >> 7;
rdmult += ((rdmult * rd_boost_factor[boost_index]) >> 7);
return rdmult > 0 ? (int)AOMMIN(rdmult, INT_MAX) : 1;
```