name: libaom/clamp-floating-point-conversions-to-integer-limits description: When casting floating-point expressions or 64-bit integers to narrower integer types, clamp values to target bounds using AOMMIN or saturate_cast_double_to_int to avoid overflow rubric: correctness severity: blocking

Rule. When casting floating-point calculations (such as values from ceil(), floor(), round(), or rate-control formulas) or wider integer variables (such as int64_t rate-distortion multipliers, frame targets, bit budgets, or accumulators) to narrower integer types (like int or unsigned int), clamp values to target integer limits using AOMMIN(..., INT_MAX) or saturate_cast_double_to_int() before casting. Verify that scaled values or multi-step accumulations cannot exceed INT_MAX (or [INT_MIN, INT_MAX]) and wrap or trigger undefined behavior.

Why. In C and C++, converting a floating-point value that exceeds the target integer type's representable range (greater than INT_MAX or less than INT_MIN) results in undefined behavior. Similarly, casting out-of-range int64_t values directly to signed 32-bit int causes signed integer overflow and undefined behavior, leading to negative rate-distortion multipliers, corrupted bit budgets, or runtime errors. Clamping or saturating prior to casting guarantees well-defined, range-safe conversions.

Examples

Bad:

// Bad: Casting unbounded floating-point calculation directly to int
rc->max_consec_drop = (int)ceil(cpi->oxcf.rc_cfg.max_consec_drop_time *
                                cpi->framerate / 1000);

// Bad: Direct cast of unbounded wide 64-bit integer / rate calculation to int
int64_t rdmult = av1_compute_rd_mult_based_on_qindex(bit_depth, update_type,
                                                     qindex, tuning);
rdmult = (rdmult * rd_layer_depth_factor[layer_depth]) >> 7;
rdmult += ((rdmult * rd_boost_factor[boost_index]) >> 7);
return (int)rdmult;

Good:

// Good: Clamping floating-point result or using saturation helper
rc->max_consec_drop = (int)AOMMIN(
    ceil(cpi->oxcf.rc_cfg.max_consec_drop_ms * cpi->framerate / 1000),
    INT_MAX);
rc->max_consec_drop = saturate_cast_double_to_int(
    ceil(cpi->oxcf.rc_cfg.max_consec_drop_ms * cpi->framerate / 1000));

// Good: Clamping 64-bit integer to INT_MAX before casting to int
int64_t rdmult = av1_compute_rd_mult_based_on_qindex(bit_depth, update_type,
                                                     qindex, tuning);
rdmult = (rdmult * rd_layer_depth_factor[layer_depth]) >> 7;
rdmult += ((rdmult * rd_boost_factor[boost_index]) >> 7);
return rdmult > 0 ? (int)AOMMIN(rdmult, INT_MAX) : 1;