rtc: Fix to non_last reference in nonrd_pickmode

For nonrd_pickmode (speed >= 7):
-avoid skip testing non_last reference if LAST
is not used as a reference
-only do the fast int_pro motion estimation for
non_last reference if LAST is used as a reference.

This improves the quality for LTR case, as in this issue
from av1_discuss:
https://groups.google.com/a/aomedia.org/g/av1-discuss/c/sqFad980SsA
Follow-up will be done to further improve quality.

No change in rtc test stats as LAST is always used as a
reference there.

Change-Id: Ic20d202b512cd903e9b47a389a905f75f933e8ec

av1/encoder/nonrd_pickmode.c

diff --git a/av1/encoder/nonrd_pickmode.c b/av1/encoder/nonrd_pickmode.c
index a39fab4..3f143bd 100644
--- a/av1/encoder/nonrd_pickmode.c
+++ b/av1/encoder/nonrd_pickmode.c
@@ -314,7 +314,7 @@
   int_mv *this_ref_frm_newmv = &frame_mv[NEWMV][ref_frame];
   unsigned int y_sad_zero;
   if (ref_frame > LAST_FRAME && cpi->oxcf.rc_cfg.mode == AOM_CBR &&
-      gf_temporal_ref) {
+      (cpi->ref_frame_flags & AOM_LAST_FLAG) && gf_temporal_ref) {
     int tmp_sad;
     int dis;
 
@@ -2466,6 +2466,11 @@
   // Skip the mode if use reference frame mask flag is not set.
   if (!search_state->use_ref_frame_mask[*ref_frame]) return true;
 
+  // Don't skip non_last references if LAST is not used a reference.
+  if (!(cpi->ref_frame_flags & AOM_LAST_FLAG) &&
+      (*ref_frame == GOLDEN_FRAME || *ref_frame == ALTREF_FRAME))
+    return false;
+
   // Skip mode for some modes and reference frames when
   // force_zeromv_skip_for_blk flag is true.
   if (x->force_zeromv_skip_for_blk &&