Fix the calculation of known_rd

In av1_handle_intra_y_mode(), known_rd is calculated based on header
rate and the same is compared with best_rd for early exit. The header
rate used mode_costs->skip_mode_cost[][]. As best_rd does not account
for mode_costs->skip_mode_cost[][], this should have been
mode_costs->skip_txfm_cost[][] and the same is corrected in this CL.

As mode_costs->skip_mode_cost[][] was not populated when
skip_mode_flag = 0, the same seems to contain stale data from previous
frames and was seen to cause mismatch between single-thread and
multi-thread encodes due to known_rd differences. Thus, usage of
mode_costs->skip_txfm_cost[][] should fix this issue as well.
mode_costs->skip_mode_cost[][] is also initialized appropriately
when skip_mode_flag = 0.

Bug: 496503793

Change-Id: I8de9c978c8cefadcf74506a9cce8c6de4c73b4ed

av1/encoder/intra_mode_search.c

diff --git a/av1/encoder/intra_mode_search.c b/av1/encoder/intra_mode_search.c
index 6f61a7c..4cefbab 100644
--- a/av1/encoder/intra_mode_search.c
+++ b/av1/encoder/intra_mode_search.c
@@ -1306,7 +1306,7 @@
   const int mode_cost =
       mode_costs->mbmode_cost[size_group_lookup[bsize]][mode] + ref_frame_cost;
   const int skip_ctx = av1_get_skip_txfm_context(xd);
-  int known_rate = mode_cost + mode_costs->skip_mode_cost[skip_ctx][0];
+  int known_rate = mode_cost + mode_costs->skip_txfm_cost[skip_ctx][0];
   const int64_t known_rd = RDCOST(x->rdmult, known_rate, 0);
   if (known_rd > best_rd) {
     intra_search_state->skip_intra_modes = 1;

av1/encoder/rd.c

diff --git a/av1/encoder/rd.c b/av1/encoder/rd.c
index 7005da4..e9f6e54 100644
--- a/av1/encoder/rd.c
+++ b/av1/encoder/rd.c
@@ -92,6 +92,8 @@
       av1_cost_tokens_from_cdf(mode_costs->skip_mode_cost[i],
                                fc->skip_mode_cdfs[i], NULL);
     }
+  } else {
+    av1_zero(mode_costs->skip_mode_cost);
   }
 
   for (i = 0; i < SKIP_CONTEXTS; ++i) {