tree 2ca98f39d446381a25ff341535d152ad9009be6f
parent 0ddee40c6d8e407698055aa3654184311e02374d
author chiyotsai <chiyotsai@google.com> 1604368420 -0800
committer Chi Yo Tsai <chiyotsai@google.com> 1604441677 +0000

Skip palette mode when there is no valid mode

During interframe encoding, the current libaom encoder searches
palette mode when the following conditions are met:
  * Palette mode is enabled
  * Screen content tools are turned on
  * The current bsize is >= BLOCK_8X8 and the max linear dim is <= 64
  * The best mode found so far is an intra mode

The encoder checks the last condition by looking at
search_state.best_mbmode.mode in av1_rd_pick_inter_mode_sb. However,
because search_state.best_mode.mode is initialized to DC_PRED, this
behaviour is incorrect in the scenario when no valid prediction mode is
found for the current partition block. This CL fixes this by checking
the current rate is < INT_MAX to ensure that the encoder has found at
least one valid mode.

Performance on screen_contents, vbr mode, 35 frames:
 SPEED_SET | AVG_PSNR | OVR_PSNR |   SSIM  |  SPD
     1     |  -0.454% |  -0.450% | -0.408% | +0.8%
     3     |  +0.074% |  +0.105% | +0.148% | +1.9%
     5     |  +0.114% |  +0.141% | +0.070% | +2.1%

The performance is neutral on non-screen-contents.

Change-Id: I573dcc2c89e3689c650169e9cd362411abaad253