Temporarily revert some 4:2:2 code

As part of 9cf0c9cde7698754c0e07768f9290b5ff3c13341 the buffering was made
to better handle 4:2:2, but that causes regressions in the tests, so we're
backing out part of it for now.

Change-Id: I9ca4cfeb159aa65514613989e3dcbc30f86ec5b2