VP10: some changes to palette mode

Account for rounding in distortion calculation in k-means;
carry out rounding before duplicates removal of base colors;
replace numbers with macros;
use prefix increment.

Slight coding gain (<0.1%) on screen_content testset.

Change-Id: Ie8bd241266da6b82c7b2874befc3a0c72b4fcd8c
6 files changed