fdct8x8_test: fix unsigned overflow

the difference between src and dst will be signed, the error will be
unsigned.
quiets -fsanitize=integer:
unsigned integer overflow: 4294967295 * 4294967295

Change-Id: I580813093ee46284fde7954520dfcb1188f79268
1 file changed