fdct4x4_test: fix unsigned overflow

the difference between src and dst will be signed, the error will be
unsigned.
quiets -fsanitize=integer:
unsigned integer overflow: 4294967295 * 4294967295

Change-Id: I502fd707823c4faaa7f587c9cc0312f057e04904