tree 8b7d0baff6699de9af39723f48deed5e6bd4aba9
parent 5c0ded6b5668db1962dc649becf545a110afccd3
author James Zern <jzern@google.com> 1650574576 -0700
committer James Zern <jzern@google.com> 1651265987 +0000

pickrst,update_[ab]_sep_sym: clamp intermediates

this avoids integer 32 and 64-bit overflows due to large values returned
from linsolve_wiener().

av1/encoder/pickrst.c:1232:35: runtime error: signed integer overflow: 2
* -1901130238 cannot be represented in type 'int'

av1/encoder/pickrst.c:1150:39: runtime error: signed integer overflow:
-25909513532 * -1901130238 cannot be represented in type 'long'

no change in BDRATE in lowres, lowres2, midres, midres2 or hdres sets

Bug: b/229500034
Bug: b/229955363
Change-Id: I80db57c0ccaa05f0a1966d0de48017ac91fd4693