Fix high bit depth mask and variance reference function

- Use arithmetic AND (&) instead of logical AND (&&) to
  generate correct testing input.
- Fix variance reference function to be consistent with
  our codebase implementation.
- Refer to the following issue:
https://bugs.chromium.org/p/webm/issues/detail?id=1166

Change-Id: I8c1ebb03e22dc9e1dcd96bdf935fc126cee71307
1 file changed