Refactor rate-distortion optimization of recursive transform partition

Support rectangular transform block in the rate-distortion cost
estimator.

Change-Id: I99201fcae797c1ed2f2184021a215867eac0288f
1 file changed