VP9: adjust trellis quant optimization RD parameters

Coding gain:
lowres  0.64%
midres  0.38%
hdres   0.58%

Change-Id: I233fa2a4b24bd1e15091a5f5ef6aff661f3f50ec
1 file changed