Rule. When casting floating-point calculations (such as values from ceil(), floor(), round(), or rate-control formulas) or wider integer variables (such as int64_t rate-distortion multipliers, frame targets, bit budgets, or accumulators) to narrower integer types (like int or unsigned int), clamp values to target integer limits using AOMMIN(..., INT_MAX) or saturate_cast_double_to_int() before casting. Verify that scaled values or multi-step accumulations cannot exceed INT_MAX (or [INT_MIN, INT_MAX]) and wrap or trigger undefined behavior.
Why. In C and C++, converting a floating-point value that exceeds the target integer type's representable range (greater than INT_MAX or less than INT_MIN) results in undefined behavior. Similarly, casting out-of-range int64_t values directly to signed 32-bit int causes signed integer overflow and undefined behavior, leading to negative rate-distortion multipliers, corrupted bit budgets, or runtime errors. Clamping or saturating prior to casting guarantees well-defined, range-safe conversions.
Bad:
// Bad: Casting unbounded floating-point calculation directly to int rc->max_consec_drop = (int)ceil(cpi->oxcf.rc_cfg.max_consec_drop_time * cpi->framerate / 1000); // Bad: Direct cast of unbounded wide 64-bit integer / rate calculation to int int64_t rdmult = av1_compute_rd_mult_based_on_qindex(bit_depth, update_type, qindex, tuning); rdmult = (rdmult * rd_layer_depth_factor[layer_depth]) >> 7; rdmult += ((rdmult * rd_boost_factor[boost_index]) >> 7); return (int)rdmult;
Good:
// Good: Clamping floating-point result or using saturation helper rc->max_consec_drop = (int)AOMMIN( ceil(cpi->oxcf.rc_cfg.max_consec_drop_ms * cpi->framerate / 1000), INT_MAX); rc->max_consec_drop = saturate_cast_double_to_int( ceil(cpi->oxcf.rc_cfg.max_consec_drop_ms * cpi->framerate / 1000)); // Good: Clamping 64-bit integer to INT_MAX before casting to int int64_t rdmult = av1_compute_rd_mult_based_on_qindex(bit_depth, update_type, qindex, tuning); rdmult = (rdmult * rd_layer_depth_factor[layer_depth]) >> 7; rdmult += ((rdmult * rd_boost_factor[boost_index]) >> 7); return rdmult > 0 ? (int)AOMMIN(rdmult, INT_MAX) : 1;