Fix chrome valgrind warning due to the use of mismatched bsize

This commit fixes a mismatched use case of block size in non-RD
intra prediction check. The residual SSE and variance should be
calculated per transform block size, instead of operating block
size, which caused chrome valgrind warning on conditional jump
based on uninitialized value (webm issue 823). This commit
resolves this issue.

Change-Id: I595c06599c7e0fd0e4a08736519ba68fc14bc79a